tom@techcolumnist:~$

~/2026/03/25/talos-linux-what-immutable-infrastructure-actually-buys-you-in-production.md

Talos Linux: What Immutable Infrastructure Buys You in Production

---
author: 
date: 
updated: 
read: 6 min
in:   [engineering, strategy]
tags: [talos, kubernetes]
---

$ grep -n '^#' post.md

I've spent enough years patching Ubuntu and RHEL nodes under Kubernetes clusters to be skeptical of any pitch that starts with "just don't SSH into it." Talos Linux makes that pitch literally: there's no shell, no SSH daemon, no package manager, not even busybox, and the entire OS is managed through a gRPC API and a declarative machine configuration. After running it under a few production clusters, the skepticism was partly warranted, but the parts of the pitch that hold up are worth more than I expected.

Everything below is written against Talos 1.14.

What "immutable" actually removes

Talos' philosophy page is specific about what immutability means. The root filesystem is a read-only SquashFS image, signed and delivered as a single versioned file, and Talos runs from that image even when installed to disk. There are a few controlled writable locations, and the main writable partition is deliberately called EPHEMERAL to remind everyone not to keep anything unique there. PID 1 is Talos' own machined, not systemd. User space is a ground-up rewrite in Go, not a trimmed-down distribution.

The obvious sell is security surface: no shell means no shell-based lateral movement, and no accumulated cruft from an admin who SSH'd in at 2am and left a fix in place that never made it back into config management. That's real, but the bigger day-to-day win is different. There's no longer a category of incident where the node's actual state has quietly drifted from what your infrastructure-as-code says it should be. On a normal distro, drift is a slow leak: a manual apt install here, a kernel parameter tweaked to fix an incident there, none of it ever reverted. On Talos the only way to change a node is to change its machine config through the API, and the running config is a resource you can read back.

That last property is what makes drift checkable, not just unlikely. The running config comes back with talosctl get machineconfig v1alpha1 -o jsonpath='{.spec}'. If you manage clusters the way Sidero Labs now recommends (a secrets.yaml bundle plus small patch files, with full configs regenerated on demand rather than committed), you can regenerate what each node should be running and compare. This is the script I run weekly and before every upgrade:

python
#!/usr/bin/env python3
"""
talos_drift_report.py

Compares the machine configuration running on each Talos node with the
configuration regenerated from secrets.yaml + patches (the declared state).
Both sides are parsed and normalized (documents keyed by kind/name, keys
sorted) so comment and ordering noise doesn't show up as drift.

Exit code 0 when every node matches, 1 when any node has drifted.

Requires: talosctl on PATH, PyYAML (pip install pyyaml).
Output can contain machine config values: keep the report private.
"""

import argparse
import difflib
import subprocess
import sys
import tempfile
from pathlib import Path

import yaml


def normalize(text):
    """Return the config as sorted YAML, one entry per (kind, name) document."""
    docs = {}
    for doc in yaml.safe_load_all(text):
        if not doc:
            continue
        key = f"{doc.get('kind', 'v1alpha1')}/{doc.get('name', '')}"
        docs[key] = doc
    return yaml.safe_dump(docs, sort_keys=True, default_flow_style=False).splitlines()


def declared_config(args, node_name):
    """Regenerate the node's controlplane config the same way it was created."""
    with tempfile.TemporaryDirectory() as tmp:
        out = Path(tmp) / f"{node_name}.yaml"
        cmd = [
            "talosctl", "gen", "config", args.cluster_name, args.endpoint,
            "--with-secrets", args.secrets,
            "--kubernetes-version", args.kubernetes_version,
            "--talos-version", args.talos_contract,
            "--with-docs=false", "--with-examples=false",
            "--output-types", "controlplane",
            "--output", str(out),
        ]
        for patch in ("common.yaml", "controlplane.yaml", f"nodes/{node_name}.yaml"):
            cmd += ["--config-patch", f"@{Path(args.patch_dir) / patch}"]
        subprocess.run(cmd, check=True, capture_output=True, text=True)
        return out.read_text()


def live_config(ip):
    """Fetch the running config: the .spec of the MachineConfig resource."""
    result = subprocess.run(
        ["talosctl", "--nodes", ip, "get", "machineconfig", "v1alpha1", "-o", "jsonpath={.spec}"],
        check=True, capture_output=True, text=True,
    )
    return result.stdout


def main():
    parser = argparse.ArgumentParser(description="Report drift between declared and running Talos configs.")
    parser.add_argument("--cluster-name", required=True)
    parser.add_argument("--endpoint", required=True, help="Kubernetes endpoint, e.g. https://10.10.20.10:6443")
    parser.add_argument("--secrets", default="secrets.yaml")
    parser.add_argument("--patch-dir", default="patches")
    parser.add_argument("--kubernetes-version", required=True, help="Version the cluster runs now")
    parser.add_argument("--talos-contract", required=True, help="Talos version contract used at creation, e.g. v1.14")
    parser.add_argument("--node", action="append", required=True, metavar="NAME=IP",
                        help="Node name (matching patches/nodes/NAME.yaml) and IP; repeatable")
    args = parser.parse_args()

    drifted = 0
    for entry in args.node:
        name, ip = entry.split("=", 1)
        try:
            declared = normalize(declared_config(args, name))
            live = normalize(live_config(ip))
        except subprocess.CalledProcessError as exc:
            print(f"[{name}] ERROR: {' '.join(exc.cmd[:3])} failed: {exc.stderr.strip()}")
            drifted += 1
            continue

        diff = list(difflib.unified_diff(declared, live, fromfile=f"{name} declared", tofile=f"{name} running", lineterm=""))
        if diff:
            drifted += 1
            print(f"[{name}] DRIFT ({ip}):")
            print("\n".join(diff))
        else:
            print(f"[{name}] in sync ({ip})")

    sys.exit(1 if drifted else 0)


if __name__ == "__main__":
    main()

Run it with the same inputs you used to build the cluster:

bash
python3 talos_drift_report.py \
    --cluster-name lab \
    --endpoint https://10.10.20.10:6443 \
    --kubernetes-version 1.37.0 \
    --talos-contract v1.14 \
    --node talos-cp1=10.10.20.11 \
    --node talos-cp2=10.10.20.12 \
    --node talos-cp3=10.10.20.13
text
[talos-cp1] in sync (10.10.20.11)
[talos-cp2] DRIFT (10.10.20.12):
--- talos-cp2 declared
+++ talos-cp2 running
@@ -41,6 +41,8 @@
   kind: KubeNodeConfig
   labels:
     node-role.kubernetes.io/control-plane: ''
+    topology.kubernetes.io/zone: rack-b
[talos-cp3] in sync (10.10.20.13)

In that example someone ran talosctl patch machineconfig to add a label and never wrote a patch file for it. The fix is a one-line patch in Git, not an archaeology session. Pass the Kubernetes version the cluster runs now: talosctl upgrade-k8s updates component images in the live configs, which is exactly the kind of drift the reproducible-configuration guide warns about if you regenerate with an older version.

The layout and bootstrap flow this assumes are in Talos Linux: Bootstrapping a Three-Node Cluster on Bare Metal.

Where it actually saves operational time

Patching is the concrete thing people ask about. Upgrading Talos is an API call that hands the node an installer image. The node cordons and drains itself, stops its services, unmounts its filesystems, writes the new image, and sets the bootloader to boot the new version once. Only after it comes back and verifies itself does it make that choice permanent, rejoin, and uncordon. Upgrades use an A-B scheme that keeps the previous kernel and OS image, so a node that fails to boot the new version falls back to the old one on the next reboot. If the node boots fine but your workloads don't like it, talosctl rollback switches back to the previous image. There's no apt or yum transaction that can half-complete, and no reboot that comes back into a broken initramfs because a kernel module got orphaned. Talos also refuses to upgrade a control plane node when doing so would cost etcd quorum. I haven't had a single "the patch broke the box and now I'm on the iDRAC console at midnight" incident on Talos nodes. On traditional distros at similar fleet sizes that was a recurring event, and it's a category of pager alert that simply stopped.

Configuration changes got safer in a way I didn't expect. In Talos 1.14, config changes are applied to the running node without a reboot. --mode=try applies a change and reverts it automatically after a timeout (one minute by default) if no further configuration update is applied. That's the right tool for anything that could cut off your own access, like a network change. I use it as a rehearsal: try the change, check from a second session that the node is still reachable and healthy, let it revert, then apply it for real:

bash
# Rehearse an MTU change; Talos reverts it after 2 minutes
talosctl -n 10.10.20.12 patch machineconfig --mode=try --timeout 2m -p @patches/mtu-9000.yaml

# Preview what the real change will do, then apply it in the default mode
talosctl -n 10.10.20.12 patch machineconfig --dry-run -p @patches/mtu-9000.yaml
talosctl -n 10.10.20.12 patch machineconfig -p @patches/mtu-9000.yaml

The patch itself is an ordinary document:

yaml
apiVersion: v1alpha1
kind: LinkConfig
name: net0
mtu: 9000

Talos' configuration patching docs are explicit that "Talos supports patching multi-document machine configuration," and list talosctl patch as the way to patch a running node's configuration, so a per-topic document like this one is a complete patch on its own.

One caveat from the same guide: a few settings are only read when a long-running service starts. etcd settings are the notable one, and Talos deliberately doesn't restart etcd on a config change, so those still need an explicit talosctl reboot, sequenced like any other control plane maintenance.

Where the trade-off actually bites

The cost is that every troubleshooting habit built over a career of ssh and grep stops working. When a node is behaving strangely you don't tail a log file. You pull what you need through the API, and it pays to have the equivalents memorized before the incident, not during it:

bash
# Text UI with node overview, logs and real-time metrics
talosctl -n 10.10.20.12 dashboard

# Service logs (Talos services), and Kubernetes container logs via the cri namespace
talosctl -n 10.10.20.12 logs kubelet --tail 200
talosctl -n 10.10.20.12 containers --namespace cri
talosctl -n 10.10.20.12 logs --namespace cri "$CONTAINER_ID"

# Kernel log, streaming
talosctl -n 10.10.20.12 dmesg --follow

# Service state, including etcd on control plane nodes
talosctl -n 10.10.20.12 service etcd

# Listening sockets with owning processes (note: -n is --nodes in talosctl, not "numeric")
talosctl -n 10.10.20.12 netstat --tcp --listening --programs

# Packet capture streamed to your workstation's tcpdump (physical link name from talosctl get links)
talosctl -n 10.10.20.12 pcap --interface enp1s0 --duration 30s -o - | tcpdump -nn -r -

# Read a file under /proc or /sys
talosctl -n 10.10.20.12 read /proc/cmdline

# Everything above plus COSI resources, bundled for a support case
talosctl -n 10.10.20.12 support --output support-talos-cp2.zip

If the thing you need to inspect isn't exposed, you're stuck, or at least slowed down. I hit exactly this with a NIC firmware quirk that would have been a five-minute ethtool session on a normal box and instead took an afternoon of working around the API's more limited surface. The surface has grown since then. talosctl get ethernetstatus <link> -o yaml shows ring sizes and offload features in ethtool terms, and the EthernetConfig document can change them declaratively. talosctl debug <image> runs a debug container on the node from an image reference or a local tarball, so you can bring your own tools for the rare case that needs them. Any team adopting Talos still needs to budget real time for this relearning, and needs to accept that a certain class of "let me just poke at it directly" debugging is gone for good, not just discouraged.

Where I'd actually recommend it

Talos earns its keep on infrastructure where the node's only job is to run Kubernetes: bare metal or VM pools dedicated to a cluster, no other tenancy, no other reason for a human to touch the box directly. It's a worse fit for mixed-use infrastructure where a node also needs to run something outside Kubernetes, or for a team that isn't already comfortable operating Kubernetes without SSH-shaped crutches. The immutability isn't a security feature bolted onto a Linux distro. It's a constraint the whole operating model is built around, and it only pays off when the rest of your operations already fit that constraint.

The honest tradeoff

Immutable infrastructure doesn't remove operational risk, it relocates it: from "did the sysadmin's manual fix get documented" to "do our secrets bundle and patches in version control actually represent what we want running." That's a better place for the risk to live, and the drift report above makes it measurable, but it still needs the same discipline: patch review, staged rollouts, a backed-up secrets.yaml, and someone who owns the machine config the way they used to own the runbook. Talos didn't make operations easier so much as it made the easy parts easier and the hard parts more honest about being hard.

References