Imported from dydx/rke2-skills (skills/rke2-upgrades/SKILL.md). Install upstream with npx skills add dydx/rke2-skills --skill rke2-upgrades. Copyright stays with the author.

RKE2 Upgrades and Rollback

General Upgrade Rules

1. Upgrade server (control-plane) nodes first, one at a time. Only after ALL servers are upgraded should agent nodes be upgraded. This applies to both manual and automated upgrades.
2. Kubernetes version skew policy applies (https://kubernetes.io/releases/version-skew-policy/). Never skip intermediate minor versions (e.g. v1.31 -> v1.33 must go through v1.32). Nothing in the RKE2 upgrade process — including system-upgrade-controller — protects against unsupported version jumps. Upgrade one minor version at a time; patch versions within a minor can be applied directly.
3. Rancher-managed clusters: if the cluster was provisioned by or imported into Rancher, use the Rancher UI to manage upgrades. For imported/registered clusters, Rancher manages the system-upgrade-controller deployment and plans by default — do not manage SUC plans yourself unless version management is disabled in Rancher. For Rancher-provisioned clusters, Rancher uses the system agent for upgrades.
4. Take a datastore snapshot before upgrading. A snapshot taken on the old version is the only path back — rollback requires it (see Rollback section).
5. Read the release notes for the target version before upgrading. RKE2 release notes list a component version table per patch release (Kubernetes, etcd, containerd, runc, CoreDNS, ingress, CNI, etc.) plus upgrade warnings (e.g. Kubernetes "Urgent Upgrade Notes", default-component changes). Releases: https://github.com/rancher/rke2/releases

Release Channels

Upgrades via the install script or system-upgrade-controller can track a release channel:

Exhaustive, current channel list: https://update.rke2.io/v1-release/channels (channel server implementation: https://github.com/rancher/channelserver). A specific channel resolves at e.g. https://update.rke2.io/v1-release/channels/stable.

Version strings are of the form vX.Y.Z+rke2rN (Kubernetes version plus an RKE2 revision), e.g. v1.33.4+rke2r1.

Manual Upgrades

Three methods: re-run the install script, replace the binary, or RPM upgrade (for RPM-based installs). In all cases: servers first (one at a time), then agents, and restart the service after installing.

Method 1: Install script (tarball installs)

Re-run the installation script with the same flags used at install time:

# Upgrade to the most recent stable release (default)
curl -sfL https://get.rke2.io | sh -

# Agent nodes: specify the install type
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=agent sh -

# Track a specific channel
curl -sfL https://get.rke2.io | INSTALL_RKE2_CHANNEL=latest sh -

# Pin an exact version
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=vX.Y.Z+rke2rN sh -

Then restart the service (installing does not restart it):

# Server nodes
sudo systemctl restart rke2-server

# Agent nodes
sudo systemctl restart rke2-agent

Method 2: Binary replacement

1. Download the desired rke2 binary from https://github.com/rancher/rke2/releases
2. Copy it over the existing binary:
   • Tarball installs: /usr/local/bin/rke2
   • RPM installs: /usr/bin/rke2
3. Restart rke2-server or rke2-agent.

Method 3: RPM upgrade

For RPM installations, upgrade via the package manager. This pulls the latest package from the channel repo that the install script configured at initial installation:

# Servers
yum update rke2-server        # or: zypper update rke2-server

# Agents
yum update rke2-agent         # or: zypper update rke2-agent

# If rke2-selinux is enabled, upgrade it too
yum update rke2-selinux

Then restart rke2-server / rke2-agent as above.

Automated Upgrades (system-upgrade-controller)

Kubernetes-native upgrades using Rancher's system-upgrade-controller (SUC). SUC watches Plan custom resources (upgrade.cattle.io/v1) that declare which nodes to upgrade (via label selector) and to what version/channel, then runs privileged upgrade Jobs on selected nodes and labels each node when its Job completes.

Repos: https://github.com/rancher/system-upgrade-controller (controller + Plan spec docs) and https://github.com/rancher/rke2-upgrade (the upgrade image).

Do not use this on Rancher-managed clusters (see General Rules above).

1. Install the controller

Installs the Plan CRD, deployment, service account, cluster role binding, and configmap (namespace system-upgrade):

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml \
              -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml

The controller is configurable via its configmap; delete the controller pod for configmap changes to take effect.

2. Create Plans

Create at least two plans — one for servers, one for agents — so servers upgrade first. Plans must be created in the namespace where the controller runs (system-upgrade). These example plans continuously track the stable channel:

# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: server-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: In
      values:
      - "true"
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
  channel: https://update.rke2.io/v1-release/channels/stable
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: agent-plan
  namespace: system-upgrade
spec:
  concurrency: 1
  cordon: true
  nodeSelector:
    matchExpressions:
    - key: node-role.kubernetes.io/control-plane
      operator: DoesNotExist
  prepare:
    args:
    - prepare
    - server-plan
    image: rancher/rke2-upgrade
  serviceAccountName: system-upgrade
  upgrade:
    image: rancher/rke2-upgrade
  channel: https://update.rke2.io/v1-release/channels/stable

Key points about these plans:

• concurrency: how many nodes upgrade simultaneously. Keep servers at 1; agents may go higher.

• cordon: true: cordons the node before the upgrade Job runs. A drain block can be used instead of/alongside cordon for stricter workload eviction (see the Plan spec docs).

• Node targeting is by label selector: the server plan selects nodes with node-role.kubernetes.io/control-plane; the agent plan selects nodes without it.

• The agent plan's prepare step (args prepare server-plan) makes agent Jobs wait until the server plan completes. This ordering logic lives in the rancher/rke2-upgrade image, not in SUC itself.

• channel vs version: channel makes the controller poll the URL periodically and upgrade whenever it resolves to a new release (continuous auto-upgrade). Alternatively, omit channel and pin a one-shot target:

  apiVersion: upgrade.cattle.io/v1
  kind: Plan
  # ...
  spec:
    # ...
    version: v1.33.4+rke2r1

The upgrade starts as soon as the controller resolves a target version (from version or the channel). Modifying a plan triggers re-evaluation.

Monitoring

kubectl -n system-upgrade get plans -o wide
kubectl -n system-upgrade get jobs

Maintenance windows

Restrict when upgrade Jobs are created with window (same format as kured schedule options):

spec:
  window:
    days:
      - monday
      - tuesday
      - wednesday
      - thursday
      - friday
    startTime: 19:00
    endTime: 21:00
    timeZone: UTC

Jobs are not created outside the window, but Jobs already created may keep running after it closes.

Security note

Upgrade Jobs are highly privileged: host IPC/NET/PID namespaces, CAP_SYS_BOOT, and the host root filesystem mounted read-write at /host.

Rollback / Downgrade

Kubernetes does not support downgrading control-plane components. The rke2-upgrade image does not currently block a Plan from downgrading, but a bare binary downgrade is not a supported rollback. The supported rollback is: downgrade the RKE2 binary AND restore a datastore snapshot taken while running the older version. Without a restorable backup from the old minor version, rollback to a previous minor is impossible. (Rancher-provisioned clusters: perform the downgrade via Rancher's etcd snapshot restore instead.)

Cautions:

• Ensure a valid db/etcd snapshot from the old version exists before upgrading.
• rke2-killall.sh forcefully kills RKE2 processes and pods; possible data loss for apps not shut down cleanly.
• Verify RKE2/component versions before and after rollback.

Embedded etcd (default)

1. If the API is still up, drain all nodes: kubectl drain --ignore-daemonsets --delete-emptydir-data <node1> <node2> ...
2. On every node, stop RKE2 and all pod processes: rke2-killall.sh
3. On every node, downgrade the binary to the previous version:

   # Servers
   curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=vX.Y.Z+rke2rN sh -
   # Agents
   curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=vX.Y.Z+rke2rN INSTALL_RKE2_TYPE=agent sh -

   (Air-gapped: download artifacts and run the install script locally.)
4. On the first server node (the one without a server: entry in its config), restore the snapshot — this overwrites all etcd data; verify snapshot integrity first:

   rke2 server --cluster-reset --cluster-reset-restore-path=<PATH-TO-SNAPSHOT>

5. systemctl start rke2-server on that first server.
6. On the other server nodes: rm -rf /var/lib/rancher/rke2/server/db, then systemctl start rke2-server.
7. systemctl start rke2-agent on all agents.
8. Verify with systemctl status rke2-server / rke2-agent.

SQLite (single node)

Replace the .db file with the copy made when backing up, downgrade the binary, restart.

External database (PostgreSQL/MySQL)

1. Drain nodes, run rke2-killall.sh on each node (as above).
2. Restore the pre-upgrade DB backup, e.g. pg_restore -U <user> -d <db> <backup-file>, and verify integrity.
3. Downgrade the binary on every node (same install-script commands as above).
4. systemctl start rke2-server (or rke2-agent) on each node.

Verify after rollback

rke2 --version, kubectl get nodes, application functionality, and RKE2 logs.