tom@techcolumnist:~$

~/2026/08/26/talos-linux-upgrading-a-cluster-without-a-maintenance-window.md

Talos Linux: Upgrading a Cluster Without a Maintenance Window

---
author: 
date: 
updated: 
read: 7 min
in:   [engineering]
tags: [talos, kubernetes]
---

$ grep -n '^#' post.md

Asking for a maintenance window used to be the normal cost of doing an OS or Kubernetes upgrade. Talos Linux, combined with a cluster that's actually built to tolerate a node disappearing, changes that math enough that I no longer schedule downtime for routine version upgrades. I schedule a rolling operation that happens during business hours, with nobody outside the platform team aware it's happening. That took some deliberate groundwork, not just trusting the upgrade command.

The commands below are current as of Talos 1.14, which changed one thing every existing upgrade runbook needs to pick up: the ghcr.io/siderolabs/installer image is no longer published, and installer images come from the Image Factory instead.

What talosctl upgrade does, and what it doesn't

When a node receives an upgrade request, it cordons itself, drains its pods, stops its services, unmounts its filesystems, writes the new image, and sets the bootloader to boot the new version once. After it comes back and verifies itself, it makes that permanent, rejoins the cluster, and uncordons. Upgrades use an A-B scheme that keeps the previous kernel and OS image, so a node that fails to boot the new version falls back on its own. A node that boots fine but runs your workloads badly needs a manual talosctl rollback.

On control plane nodes Talos adds a guard that matters a lot for no-downtime work: it refuses to upgrade a control plane node if doing so would cost etcd quorum, and if several control plane nodes are asked to upgrade at once it only lets one proceed at a time. The docs are equally clear about what that guard doesn't cover. Nothing stops you from sending near-simultaneous upgrades to every node, and software that keeps its own quorum (their example is Rook/Ceph) may not recover from more than one node rebooting at a time.

Two flag changes trip up older runbooks:

  • --preserve is gone. Since Talos 1.8 the installer never wipes the system disk on upgrade, so the release notes describe --preserve as always set, and current talosctl upgrade doesn't have the flag.
  • --image should always be explicit. It defaults to the installer for the talosctl version in use, with the empty schematic. Omit it on a node built from a schematic with system extensions and the node upgrades without them. Look up the node's schematic first; Image Factory appends it to the extension list:
bash
talosctl get extensions --nodes 10.10.20.11
text
NODE          NAMESPACE   TYPE              ID   VERSION   NAME        VERSION
10.10.20.11   runtime     ExtensionStatus   0    1         schematic   376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba

That ID (the "vanilla" schematic, in this example) becomes factory.talos.dev/metal-installer/<schematic-id>:<version>.

The precondition is workload placement, not the upgrade tool

Both talosctl upgrade (which cordons and drains a node before touching it) and talosctl upgrade-k8s are built to be run safely against a live cluster. None of that matters if the cluster's workloads can't survive a node leaving. A drain goes through the Kubernetes Eviction API, which honours PodDisruptionBudgets, so an upgrade tool that drains correctly is necessary but not sufficient: if three replicas of a service all sit on the node being drained, the graceful drain just makes the outage orderly instead of preventing it.

For anything customer-facing, that means more than one replica, a PDB that reflects real availability requirements, and a spread constraint that keeps replicas on different nodes:

yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: invoice-api
  namespace: billing
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: invoice-api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: invoice-api
  namespace: billing
spec:
  replicas: 3
  selector:
    matchLabels:
      app: invoice-api
  template:
    metadata:
      labels:
        app: invoice-api
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: invoice-api
      containers:
        - name: api
          image: registry.example.com/billing/invoice-api:2.4.1

The mirror-image mistake is a PDB that can never be satisfied, such as minAvailable: 1 on a single-replica Deployment. That PDB permanently allows zero disruptions, and the drain waits on it until --drain-timeout (five minutes by default) runs out. So before every rolling operation I run a preflight that answers one question: can this cluster lose one node right now?

python
#!/usr/bin/env python3
"""
upgrade_preflight.py

Answers "can this cluster lose one node right now?" before a rolling Talos or
Kubernetes upgrade. Blocks (exit 1) on:
  - nodes that are not Ready or are already cordoned
  - PodDisruptionBudgets that currently allow zero disruptions (a drain will stall)
  - Deployments/StatefulSets outside the ignored namespaces with fewer than 2 replicas
  - multi-replica Deployments whose Ready pods all sit on one node

Requires: Python 3.10+, `pip install kubernetes`, a kubeconfig that can list
nodes, pods, deployments, statefulsets and poddisruptionbudgets.
"""

import argparse
import sys
from collections import defaultdict

from kubernetes import client, config


def main():
    parser = argparse.ArgumentParser(description="Pre-flight checks before a rolling node upgrade.")
    parser.add_argument("--kubeconfig", default=None, help="Path to kubeconfig (default: standard lookup)")
    parser.add_argument("--ignore-namespace", action="append", default=["kube-system"],
                        help="Namespace to skip in the workload checks (repeatable; kube-system is skipped by default)")
    args = parser.parse_args()

    config.load_kube_config(config_file=args.kubeconfig)
    core, apps, policy = client.CoreV1Api(), client.AppsV1Api(), client.PolicyV1Api()
    ignored = set(args.ignore_namespace)
    blockers = []

    for node in core.list_node().items:
        ready = next((c for c in node.status.conditions or [] if c.type == "Ready"), None)
        if ready is None or ready.status != "True":
            blockers.append(f"node {node.metadata.name} is not Ready")
        if node.spec.unschedulable:
            blockers.append(f"node {node.metadata.name} is already cordoned")

    for pdb in policy.list_pod_disruption_budget_for_all_namespaces().items:
        if pdb.metadata.namespace in ignored:
            continue
        if (pdb.status.disruptions_allowed or 0) < 1:
            blockers.append(
                f"PDB {pdb.metadata.namespace}/{pdb.metadata.name} allows 0 disruptions "
                f"(healthy {pdb.status.current_healthy}/{pdb.status.desired_healthy} desired)"
            )

    workloads = [("Deployment", d) for d in apps.list_deployment_for_all_namespaces().items]
    workloads += [("StatefulSet", s) for s in apps.list_stateful_set_for_all_namespaces().items]
    for kind, obj in workloads:
        ns, name = obj.metadata.namespace, obj.metadata.name
        replicas = obj.spec.replicas if obj.spec.replicas is not None else 1
        if ns not in ignored and 0 < replicas < 2:
            blockers.append(f"{kind} {ns}/{name} has {replicas} replica: it will go down while its node drains")

    # Ready pods per ReplicaSet owner, grouped by node: catches "3 replicas, 1 node".
    placement = defaultdict(set)
    for pod in core.list_pod_for_all_namespaces(field_selector="status.phase=Running").items:
        if pod.metadata.namespace in ignored or not pod.spec.node_name:
            continue
        for owner in pod.metadata.owner_references or []:
            if owner.kind == "ReplicaSet":
                placement[(pod.metadata.namespace, owner.name)].add(pod.spec.node_name)
    for (ns, rs), nodes in placement.items():
        replica_set = apps.read_namespaced_replica_set(rs, ns)
        if (replica_set.spec.replicas or 0) >= 2 and len(nodes) == 1:
            blockers.append(f"ReplicaSet {ns}/{rs}: all {replica_set.spec.replicas} replicas run on {next(iter(nodes))}")

    if blockers:
        print("NOT SAFE to take a node out:")
        for item in blockers:
            print(f"  - {item}")
        sys.exit(1)

    print("OK: every node Ready, no PDB at zero, no single-replica or single-node workloads.")
    sys.exit(0)


if __name__ == "__main__":
    main()
text
NOT SAFE to take a node out:
  - PDB billing/invoice-worker allows 0 disruptions (healthy 1/1 desired)
  - Deployment billing/invoice-worker has 1 replica: it will go down while its node drains

Etcd is the part that actually requires care

Worker nodes are the easy case; losing one briefly is exactly what the scheduler and PodDisruptionBudgets exist to absorb. Control plane nodes running etcd are where I slow down, because etcd's quorum math is unforgiving. Talos' disaster recovery guide states it plainly: a three-node control plane tolerates the failure of any single node, and losing more than one at the same time is a complete loss of service. Upgrading a control plane node takes its etcd member away for the reboot, so the cluster runs with no margin until it's back.

Talos' quorum guard stops the worst mistake, but I still gate each control plane node on a clean etcd status across all members before moving to the next:

bash
talosctl etcd status --nodes 10.10.20.11,10.10.20.12,10.10.20.13
text
NODE          MEMBER             DB SIZE   IN USE            LEADER             RAFT INDEX   RAFT TERM   RAFT APPLIED INDEX   LEARNER   ERRORS
10.10.20.11   a49c021e76e707db   48 MB     21 MB (43.75%)    ecebb05b59a776f1   1982231      7           1982231              false
10.10.20.12   ecebb05b59a776f1   47 MB     21 MB (44.68%)    ecebb05b59a776f1   1982231      7           1982231              false
10.10.20.13   eb47fb33e59bf0e2   48 MB     21 MB (43.75%)    ecebb05b59a776f1   1982231      7           1982231              false

All three members present, one agreed leader, matching raft indexes, nothing in ERRORS. And before the first node is touched, I take an etcd snapshot. talosctl etcd snapshot produces a consistent snapshot from any healthy control plane node, and it's the file talosctl bootstrap --recover-from needs if everything goes wrong. A snapshot taken five minutes before the upgrade is worth far more than last night's.

The rolling upgrade as a script

This is the script, with the health gate between every node. talosctl health confirms etcd, the control plane components, node readiness and schedulability; the etcd status and alarm checks catch anything subtler. It stops at the first failure and leaves the remaining nodes on the old version, which is a perfectly supportable state for as long as it takes to investigate.

bash
#!/usr/bin/env bash
# rolling-talos-upgrade.sh
#
# Upgrades Talos one node at a time: etcd snapshot first, then each control
# plane node, then each worker, with a full health gate and an etcd check
# between nodes. Stops at the first failure and leaves the rest untouched.
#
# Needs: talosctl matching the version the cluster runs now, an os:admin
# talosconfig with the node IPs as endpoints, kubectl, and
# upgrade_preflight.py next to this script.
set -euo pipefail

TARGET_VERSION="v1.14.0"   # the latest patch of the next minor release; never skip a minor
SCHEMATIC="<schematic-id>" # from: talosctl get extensions --nodes <ip>
IMAGE="factory.talos.dev/metal-installer/${SCHEMATIC}:${TARGET_VERSION}"
CONTROL_PLANE=(10.10.20.11 10.10.20.12 10.10.20.13)
WORKERS=(10.10.20.21 10.10.20.22)
SNAPSHOT_DIR="/var/backups/etcd"

CP_CSV="$(IFS=,; echo "${CONTROL_PLANE[*]}")"
WORKER_CSV="$(IFS=,; echo "${WORKERS[*]}")"

health_gate() {
    local args=(--nodes "${CONTROL_PLANE[0]}" --control-plane-nodes "$CP_CSV" --wait-timeout 10m)
    if [[ ${#WORKERS[@]} -gt 0 ]]; then
        args+=(--worker-nodes "$WORKER_CSV")
    fi
    talosctl health "${args[@]}"
    talosctl etcd status --nodes "$CP_CSV"
    if talosctl etcd alarm list --nodes "$CP_CSV" | grep -v '^NODE' | grep -q .; then
        echo "etcd has active alarms; stopping" >&2
        exit 1
    fi
}

upgrade_node() {
    local ip="$1"
    echo "=== $(date -u +%FT%TZ) upgrading $ip to $TARGET_VERSION"
    python3 "$(dirname "$0")/upgrade_preflight.py"
    talosctl upgrade --nodes "$ip" --image "$IMAGE" --wait --drain-timeout 10m
    health_gate
    talosctl version --nodes "$ip" | grep -A2 '^Server'
}

echo "=== pre-upgrade health and etcd snapshot"
health_gate
mkdir -p "$SNAPSHOT_DIR"
talosctl etcd snapshot "$SNAPSHOT_DIR/etcd-$(date -u +%Y%m%dT%H%M%SZ)-pre-${TARGET_VERSION}.snapshot" \
    --nodes "${CONTROL_PLANE[0]}"

for ip in "${CONTROL_PLANE[@]}"; do
    upgrade_node "$ip"
done

for ip in "${WORKERS[@]}"; do
    upgrade_node "$ip"
done

echo "=== all nodes on $TARGET_VERSION"
kubectl get nodes -o wide

Upgrade to the latest patch of each intermediate minor release rather than jumping: Talos only tests config migrations between adjacent minors. Going from 1.12 to 1.14 means two passes of this script, 1.12 to the latest 1.13 patch and then 1.13 to 1.14. Sidero also recommends running the talosctl version that matches the version the cluster currently runs.

Sequence the OS and Kubernetes upgrades separately

It's tempting to bump the Talos version and the Kubernetes version in the same pass, but I treat them as two rolling operations with a validation gap in between. Talos made that the default years ago: since 1.0, a Talos OS upgrade doesn't upgrade Kubernetes. talosctl upgrade handles the OS, and talosctl upgrade-k8s handles the control plane components, kube-proxy, the kubelet on every node, and the bootstrap manifests. When both changes land in the same pass and something breaks, the first question is always "which of the two things I changed caused this," and separating them removes that question entirely.

The Kubernetes step has a dry run that I always read first. It lists resources using APIs that are being deprecated in the target version and shows every component it would update:

bash
talosctl --nodes 10.10.20.11 upgrade-k8s --to 1.37.0 --dry-run
talosctl --nodes 10.10.20.11 upgrade-k8s --to 1.37.0

--nodes names the control plane node that receives the API call, but every node in the cluster gets upgraded. The command pre-pulls images, patches each control plane node's config with the new component versions, updates kube-proxy, then upgrades and verifies the kubelet node by node. If it fails partway, you can rerun it and it continues from where it stopped. Go one Kubernetes minor at a time: the version skew policy doesn't allow kube-apiserver to skip minor versions. Check the target against the Talos support matrix, too; Talos 1.14 supports Kubernetes 1.33 through 1.37.

One follow-up belongs in the change ticket. upgrade-k8s rewrites component versions in each node's live config, so any full controlplane.yaml you kept in Git is now stale and would downgrade components if re-applied. That's the drift Sidero's reproducible-configuration guide is written around: keep secrets.yaml and patches, and regenerate full configs with the Kubernetes version the cluster now runs.

What actually made this boring

The upgrades that go well are the ones where I've already tested the exact upgrade path, not just "some Talos upgrade" but the specific current-version-to-target-version jump, on a non-production cluster running comparable workloads. Every release's upgrade guide has a "before upgrade" section, and it's not boilerplate: the 1.14 guide, for example, requires migrating the multipath-tools extension's configuration from an ExtensionServiceConfig document to an EtcFileConfig document before upgrading, or multipathd waits forever after the reboot. Finding that in a test cluster costs nothing. Finding it mid-rollout on a production control plane costs the calm that makes a no-downtime upgrade worth attempting in the first place. Once the workload placement is right and the upgrade path is proven, the rolling upgrade itself really is close to a non-event, which is the whole point.

The cron health check in Python: Talos – Health-Checking a Kubernetes Cluster from a Cron Job is what tells me, the morning after, that nothing was left cordoned or crash-looping.

References