Skip to content

Documentation for rc0 tasks #2819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Aug 20, 2025
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/ISSUE_TEMPLATE/cut-release.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,9 @@ Help? Ring @release-managers on slack!
- [ ] Notify #release-management on Slack: <!-- Paste link to slack -->
- [ ] Update [`schedule.yaml` file](https://github.com/kubernetes/website/blob/main/data/releases/schedule.yaml) with the latest release using [`schedule-builder`](https://github.com/kubernetes/release/tree/master/cmd/schedule-builder) (_patch releases only_) <!-- Paste Pull Request URL here -->
- [ ] Collect metrics and add them to the `Release steps` table below through `krel history --branch release-1.xx --date-from yyyy-mm-dd` (current date)
<!-- ONLY FOR RC.0 RELEASES - [ ] Finish post-release branch creation tasks -->
<!-- ONLY FOR STABLE RELEASES - [ ] Code Thaw PR merged -->
<!-- ONLY FOR RC.0 RELEASES - [ ] Post-release branch creation tasks -->
<!-- ONLY FOR RC.1 RELEASES - [ ] Ensure that the new OBS project is set -->
<!-- ONLY FOR STABLE RELEASES - [ ] Code Thaw PR merged and mention @release-managers nudging them to perform the propedeutic tasks for the next alpha (e.g. setting up the new OBS project) -->

## Release Tools Version

Expand Down
79 changes: 79 additions & 0 deletions .github/ISSUE_TEMPLATE/post-release-branch-creation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
name: Post Release Branch Creation Tasks
about: Tasks to perform after the rc.0 is cut and the upcoming release branch is created
title: Post Release Branch Creation Tasks for v1.x.y-rc.0
labels: sig/release, area/release-eng
---

## As a follow up on: <!-- #<issue> typically the rc.0 cut issue (currently the branch is created by krel during that cut process) -->

## Tasks

<!--

Follow the docs here: https://github.com/kubernetes/sig-release/blob/master/release-engineering/handbooks/post-release-branch-creation.md

Help? Ring @release-managers on slack!

-->

- [ ] Create a thread in #release-management: <!-- Paste link to Slack thread -->
- [ ] Promote the subsequent minor release alpha.0 image
- [ ] (optional) Remove jobs for EOL versions, **only** if agreed upon with Release Managers and totally ok to delegate this to @release-managers
<!--
Branch Managers might not have a context on if it is "safe" to remove the EOL jobs. We try to be firm with the deadlines and stop cutting patches as soon as we reach the EOL date, but e.g. there might be a new patch needed because of some important security fix, in which case only Release Managers will know about that.

There might be a significant time/delay between the release reaching EOL and the new branch being created, leaving those jobs hanging for a while, which has an impact on the project infra costs.

The trigger for removing such jobs should be solely the EOL date but we shouldn't connect getting rid of EOL jobs and the new branch creation. Even if has been like that before, it shouldn't be longterm.
-->
- [ ] Update [`milestone_applier` rules](https://github.com/kubernetes/test-infra/blob/master/config/prow/plugins.yaml)
- [ ] Update [`kubekins-e2e-v2/variants.yaml`](https://github.com/kubernetes/test-infra/blob/master/images/kubekins-e2e-v2/variants.yaml) with the new version config
- [ ] Rotate configuration of release branch jobs in kubernetes/test-infra in particular `releng/test_config.yaml` for the upcoming release
- [ ] Run test generation script, configure the new release dashboards and send a PR with both tests and dashboards config
- [ ] Add a new variant for the `kube-cross` image (`kubernetes/release` repository) and ensure the image is built and pushed properly
- [ ] Add new variants for `k8s-cloud-builder` and `k8s-ci-builder` images (`kubernetes/release` repository) and ensure images are built and pushed properly
- [ ] Update references in `kubernetes/kubernetes` with the new kube-cross image (only after all images are pushed/promoted)
- [ ] Update publishing-bot rules to include the new release branch
- [ ] Ensure that a new [performance tests](https://github.com/kubernetes/perf-tests/) branch is created by SIG Scalability maintainers
- [ ] Inform stakeholders about the post branch creation tasks being completed
- [ ] Monitor the new release dashboard with the Release Signal Lead for at least 48 hours - mainly for missing or misconfigured jobs

## Action Items

<!--
During the post rc tasks, you may find a few things that require updates
(process changes, documentation updates, fixes to release tooling).

Please list them here.

It will be your responsibility to open issues/PRs to resolve these
issues/improvements. Keep this issue open until these action items
are complete.

- [ ] Item 1
- [ ] Item 2
- [ ] Item 3
-->

## Open Questions

<!--
During the post rc tasks, you may have a few questions that you can't
answer yourself or may require group discussion.

Please list them here.

Follow up with Branch Managers/Patch Release Team/Release Engineering
subproject owners to get these questions answered.

- [ ] Item 1
- [ ] Item 2
- [ ] Item 3
-->

/milestone <!-- v1.x e.g. v1.14 -->
/assign <!-- @ the Release or Branch Manager responsible for this release -->
/cc @kubernetes/release-managers @kubernetes/release-team-release-signal
/priority important-soon
/kind documentation
84 changes: 79 additions & 5 deletions release-engineering/handbooks/k8s-release-cut.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,38 @@
# Cutting a Kubernetes release

<!-- toc -->
- [Cutting a Kubernetes release](#cutting-a-kubernetes-release)
- [Prerequisites](#prerequisites)
- [Green Release Signal (pre-releases only)](#green-release-signal-pre-releases-only)
- [Access to GCP](#access-to-gcp)
- [Install latest software (every time)](#install-latest-software-every-time)
- [Download or update `Go` to the latest available stable version:](#download-or-update-go-to-the-latest-available-stable-version)
- [Download or update the `gcloud` CLI to interact with GCP.](#download-or-update-the-gcloud-cli-to-interact-with-gcp)
- [Download or update `krel`](#download-or-update-krel)
- [Download or update the latest `kpromo` tool](#download-or-update-the-latest-kpromo-tool)
- [Download schedule-builder](#download-schedule-builder)
- [Configure GitHub Personal Access Token](#configure-github-personal-access-token)
- [1. Release cut issue](#1-release-cut-issue)
- [2. Create a thread on the `#release-management` Slack channel](#2-create-a-thread-on-the-release-management-slack-channel)
- [3. Generate testgrid screenshots](#3-generate-testgrid-screenshots)
- [4. Check publishing-bot status](#4-check-publishing-bot-status)
- [5. Mock stage and Mock release](#5-mock-stage-and-mock-release)
- [6. No-mock stage](#6-no-mock-stage)
- [7. Image promotion](#7-image-promotion)
- [Merge promo PR](#merge-promo-pr)
- [Wait on image promotion job](#wait-on-image-promotion-job)
- [8. No-mock release](#8-no-mock-release)
- [9. Notify public dev Google group mailinglist](#9-notify-public-dev-google-group-mailinglist)
- [Manually create release HTML announcements](#manually-create-release-html-announcements)
- [Legacy Sendgrid method:](#legacy-sendgrid-method)
- [10. Post release tasks](#10-post-release-tasks)
- [\[RC.0 only\] Considerations and post branch creation release tasks](#rc0-only-considerations-and-post-branch-creation-release-tasks)
- [Next Release Branch Creation](#next-release-branch-creation)
- [Post branch creation release tasks](#post-branch-creation-release-tasks)
- [\[Stable only\] Code Thaw](#stable-only-code-thaw)
- [\[Patch only\] Update schedule on k/website](#patch-only-update-schedule-on-kwebsite)
- [Cleanup](#cleanup)

A step by step guide for cutting Kubernetes patch releases. At a high-level:

- Maintain GitHub release cut issue
Expand All @@ -15,13 +48,17 @@ A step by step guide for cutting Kubernetes patch releases. At a high-level:

## Prerequisites

### Green Release Signal
### Green Release Signal (pre-releases only)

On the same day of the release, a green signal must've been given in the #release-management Slack channel. If in doubt, double check with the current Release Signal Team Lead.
You can find the complete list of release signal team members at this link (substitute `1.xx` with the current release version):

`https://github.com/kubernetes/sig-release/blob/master/releases/release-1.xx/release-team.md`

> [!NOTE]
Ensure that there are no patch releases in progress, coordinating with @release-managers.
These are typically scheduled on different days of the week, so there is usually no need to plan around them, but since they can sometimes overlap with other release activities, it's good to double-check.

### Access to GCP

You must be a member of [k8s-infra-release-editors](https://github.com/kubernetes/k8s.io/blob/main/groups/sig-release/groups.yaml) on GitHub.
Expand Down Expand Up @@ -117,10 +154,18 @@ krel version

#### Download or update the latest `kpromo` tool

Run the following command ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#preparing-environment)):
Run the following command ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#preparing-environment)) to get the latest release of `kpromo`:

```
go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@main
```

or to build the latest version directly from a target branch:

```
go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@latest
git clone https://github.com/kubernetes-sigs/promo-tools
git pull origin <target-branch>
make kpromo
```

Validate with:
Expand Down Expand Up @@ -248,6 +293,9 @@ If the issue is open you must stop the release process and inform #release-manag

## 5. Mock stage and Mock release

> [!WARNING]
Before cutting `alpha.1` ideally some days before, ensure that @release-managers have performed the propedeutic tasks for the alpha cut (e.g. setting up the new OBS project)

Mock stages and mock releases are non-destructive and can be reran upon failure. To begin the mock stage, run the following `krel stage` command (replace the stage with the appropriate "type").

Run the following command from within the release repo directory.
Expand Down Expand Up @@ -477,12 +525,38 @@ krel history --branch release-1.33 --date-from 2025-04-23

## 10. Post release tasks

### [RC.0 only] Post rc.0 release tasks
### [RC.0 only] Considerations and post branch creation release tasks

Remember that before launching the `nomock release` command for an rc.0, you need to ensure that the image promo job for the next alpha.0 has been completed successfully.

See [here](post-rc0-release-tasks.md) for the complete list of post rc.0 release tasks.
#### Next Release Branch Creation

> [!IMPORTANT]
> The new release branch is created in the `nomock` staging phase and pushed to the repository during the `nomock` release phase of an rc.0 cut.

During a `rc.0` release our release tooling creates a new release branch named `release-X.Y`, where `X.Y.0` is the version of the upcoming release.
Additionally, the `rc.0` release automatically triggers an `alpha.0` build for the subsequent release (e.g. for `v1.34.0-rc.0`, `v1.35.0-alpha.0` is created automatically).

Behind the scenes `krel` is executing a `git branch create` command and `git push`.

At the same time Prow’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs every hour at 54 minutes past the hour and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `kubernetes/kubernetes` repo, including the newly created one.
No need to manually create the branch protection rule.

However, it is important to ensure that the branch is protected. We had cases where the branch was not protected and this was noticed very late.

> [!NOTE]
This means that the staging step will take about twice as long, as it will stage both versions `vX.Y.0-rc.0` and `vX.{Y+1}.0-alpha.0`.
The release step will also be extended, but not substantially longer in time.

#### Post branch creation release tasks

See [here](post-rc0-release-tasks.md) for the complete list of post branch creation release tasks.

Such list resides in a different document to mainain this one in a bite-sized SRE style format.

> [!WARNING]
You will not be able to cut an rc.1 or any other cut against the new branch until the post branch creation tasks (post rc.0) are complete.

### [Stable only] Code Thaw

Code thaw means you need to lift the code freeze, [here](https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md#code-thaw)
Expand Down
Loading