tom@techcolumnist:~$

~/2026/02/04/python-talos-health-checking-a-kubernetes-cluster-from-a-cron-job.md

Python: Talos – Health-Checking a Kubernetes Cluster from a Cron Job

---
author: 
date: 
updated: 
read: 5 min
in:   [scripts, engineering]
tags: [python, talos, kubernetes]
---

$ grep -n '^#' post.md

talosctl health is a genuinely good command: it waits for etcd to be healthy and its membership to be consistent, the control plane static pods and components to be ready, every node to report Ready and schedulable, and kube-proxy and CoreDNS to be up, and it tells you clearly when the control plane isn't happy. What it doesn't tell you is whether an application pod has been crash-looping for two days on an otherwise healthy cluster, or whether etcd has raised a NOSPACE alarm. I run this script from cron on a jump box that holds both a talosconfig and a kubeconfig, and it combines the Talos-level checks with Kubernetes-level ones so both kinds of problem land in the same alert instead of two dashboards nobody checks at the same time.

The script is written against Talos 1.14 and the official kubernetes Python client (36.x at the time of writing). It pairs with the rolling upgrade workflow in Talos Linux: Upgrading a Cluster Without a Maintenance Window. talosctl health already fails when a node is unschedulable, but the separate cordon check names the node in the alert, which is what you want after an upgrade that stopped halfway.

Requirements

  • Python 3.10 or later with the official client: pip install kubernetes.
  • talosctl on the PATH. Sidero Labs recommends a talosctl version that matches the Talos version running on the cluster.
  • A talosconfig whose context can reach every node you list. Sidero Labs' RBAC guide places etcd alarm management in the os:operator role, so os:reader isn't enough for the alarm check. Generate a dedicated client config for the monitoring box with talosctl config new --roles=os:operator healthcheck rather than copying the os:admin one.
  • A kubeconfig bound to a role that can list nodes and pods cluster-wide. The admin kubeconfig from talosctl kubeconfig works, but a read-only ServiceAccount token is the better habit.
  • Network reach from the cron host to the Talos API (TCP 50000) on the nodes and to the Kubernetes API server (TCP 6443).

Parameters

ParameterRequiredDescription
--talosconfigyesPath to the Talos client config.
--kubeconfigyesPath to the kubeconfig.
--control-plane-nodesyesControl plane node IPs. The first one is the node talosctl health runs through.
--worker-nodesnoWorker node IPs. Omit on a cluster where every node is a control plane node.
--restart-thresholdnoRestart count that counts as unhealthy (default 5).
--timeoutnoSeconds passed to talosctl health --wait-timeout (default 120; the talosctl default is 20 minutes, far too long for cron).
--ignore-namespacenoNamespace to skip in the pod check, repeatable (for example a namespace full of short-lived test Jobs).
--webhook-urlnoURL that receives the JSON summary as a POST when something is wrong.
--lock-filenoLock file used to skip a run while the previous one is still going (default /tmp/talos-healthcheck.lock).

Usage

Run it by hand against a three-node cluster where all three nodes are control plane nodes:

bash
python3 talos_cluster_healthcheck.py \
    --talosconfig /etc/talos/talosconfig \
    --kubeconfig /etc/kubernetes/kubeconfig \
    --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 \
    --webhook-url "https://alerts.example.com/hooks/talos"

With dedicated workers, list them separately so talosctl health knows what membership to expect:

bash
python3 talos_cluster_healthcheck.py \
    --talosconfig /etc/talos/talosconfig \
    --kubeconfig /etc/kubernetes/kubeconfig \
    --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 \
    --worker-nodes 10.10.20.21 10.10.20.22

Sample output when the control plane is fine but one worker was left cordoned and one pod is crash-looping:

json
{
  "checkedAt": "2026-02-03T14:20:01+00:00",
  "healthy": false,
  "problems": [
    {
      "check": "cordoned nodes",
      "detail": [
        "talos-w02"
      ]
    },
    {
      "check": "pod health",
      "detail": [
        "billing/invoice-worker-6c9d8f7b8-x2p4q: container invoice-worker is CrashLoopBackOff"
      ]
    }
  ]
}

Schedule it every 10 minutes and rely on the webhook for alerting (cron's own mail-on-output gets noisy fast). The exit code is 0 when healthy, 1 when a check found a problem, and 2 when the check itself couldn't run:

text
*/10 * * * * /usr/bin/python3 /opt/scripts/talos_cluster_healthcheck.py --talosconfig /etc/talos/talosconfig --kubeconfig /etc/kubernetes/kubeconfig --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 --webhook-url "https://alerts.example.com/hooks/talos" >> /var/log/talos-healthcheck.log 2>&1

Script

python
#!/usr/bin/env python3
"""
talos_cluster_healthcheck.py

Health-checks a Talos Linux Kubernetes cluster from a cron job:

  1. `talosctl health` (server-side), run through one control plane node
     against an explicit list of control plane and worker nodes.
  2. `talosctl etcd alarm list` across the control plane nodes (NOSPACE and
     CORRUPT alarms stop etcd writes long before anything else complains).
  3. Kubernetes node readiness and cordon state, through the Kubernetes API.
  4. Pod health: Failed/Unknown phases, init and app containers stuck in a
     back-off waiting state, and containers over a restart threshold.

Prints a JSON summary, posts it to an optional webhook on failure, and exits:
    0  healthy
    1  one or more checks found a problem
    2  the health check itself could not run (lock held, talosctl missing, ...)

Requires: Python 3.10+, the `kubernetes` package, `talosctl` on PATH.
"""

import argparse
import fcntl
import json
import subprocess
import sys
import urllib.request
from datetime import datetime, timezone

from kubernetes import client, config

BACKOFF_REASONS = {"CrashLoopBackOff", "ImagePullBackOff", "ErrImagePull", "CreateContainerConfigError"}


def run_talosctl(args, talosconfig, timeout):
    """Run a talosctl command and return (returncode, combined output)."""
    cmd = ["talosctl", "--talosconfig", talosconfig, *args]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
    except subprocess.TimeoutExpired:
        return 124, f"timed out after {timeout}s: {' '.join(cmd)}"
    return result.returncode, (result.stdout + result.stderr).strip()


def check_talos_health(talosconfig, control_plane_nodes, worker_nodes, timeout_seconds):
    """talosctl health talks to exactly one node, which checks the whole cluster."""
    args = [
        "health",
        "--nodes", control_plane_nodes[0],
        "--control-plane-nodes", ",".join(control_plane_nodes),
        "--wait-timeout", f"{timeout_seconds}s",
    ]
    if worker_nodes:
        args += ["--worker-nodes", ",".join(worker_nodes)]

    code, output = run_talosctl(args, talosconfig, timeout_seconds + 30)
    return code == 0, output


def check_etcd_alarms(talosconfig, control_plane_nodes):
    """Return a list of alarm lines; an empty list means no alarms."""
    code, output = run_talosctl(
        ["etcd", "alarm", "list", "--nodes", ",".join(control_plane_nodes)], talosconfig, 60
    )
    if code != 0:
        return [f"talosctl etcd alarm list failed: {output}"]

    lines = [line for line in output.splitlines() if line.strip()]
    return [line for line in lines if not line.startswith("NODE")]


def check_nodes(v1):
    """Return (not_ready, cordoned) lists of node names."""
    not_ready, cordoned = [], []
    for node in v1.list_node(_request_timeout=30).items:
        conditions = node.status.conditions or []
        ready = next((c for c in conditions if c.type == "Ready"), None)
        if ready is None or ready.status != "True":
            reason = ready.reason if ready else "no Ready condition"
            not_ready.append(f"{node.metadata.name} ({reason})")
        if node.spec.unschedulable:
            cordoned.append(node.metadata.name)
    return not_ready, cordoned


def check_pods(v1, restart_threshold, ignore_namespaces):
    """Return a list of 'namespace/pod: reason' strings for unhealthy pods."""
    problems = []
    for pod in v1.list_pod_for_all_namespaces(_request_timeout=60).items:
        ns, name = pod.metadata.namespace, pod.metadata.name
        if ns in ignore_namespaces:
            continue

        phase = pod.status.phase
        if phase == "Succeeded":
            continue
        if phase in ("Failed", "Unknown"):
            problems.append(f"{ns}/{name}: phase={phase} reason={pod.status.reason}")
            continue

        # A failing init container keeps the pod Pending (Init:CrashLoopBackOff),
        # so check init container statuses as well as the app containers.
        statuses = [("init container", cs) for cs in pod.status.init_container_statuses or []]
        statuses += [("container", cs) for cs in pod.status.container_statuses or []]
        for kind, cs in statuses:
            waiting = cs.state.waiting if cs.state else None
            if waiting and waiting.reason in BACKOFF_REASONS:
                problems.append(f"{ns}/{name}: {kind} {cs.name} is {waiting.reason}")
            elif cs.restart_count >= restart_threshold:
                problems.append(f"{ns}/{name}: {kind} {cs.name} restarted {cs.restart_count} times")
    return problems


def send_webhook(webhook_url, summary):
    """POST the JSON summary. Best effort: a failed webhook is logged, not fatal."""
    payload = json.dumps(summary).encode("utf-8")
    request = urllib.request.Request(
        webhook_url, data=payload, headers={"Content-Type": "application/json"}, method="POST"
    )
    try:
        with urllib.request.urlopen(request, timeout=10) as response:
            response.read()
    except Exception as exc:  # noqa: BLE001 - alerting must never crash the check
        print(f"warning: webhook POST failed: {exc}", file=sys.stderr)


def main():
    parser = argparse.ArgumentParser(description="Health-check a Talos Linux Kubernetes cluster.")
    parser.add_argument("--talosconfig", required=True, help="Path to talosconfig")
    parser.add_argument("--kubeconfig", required=True, help="Path to kubeconfig")
    parser.add_argument("--control-plane-nodes", nargs="+", required=True, help="Control plane node IPs")
    parser.add_argument("--worker-nodes", nargs="*", default=[], help="Worker node IPs (omit if none)")
    parser.add_argument("--restart-threshold", type=int, default=5, help="Restart count treated as unhealthy (default 5)")
    parser.add_argument("--timeout", type=int, default=120, help="talosctl health --wait-timeout in seconds (default 120)")
    parser.add_argument("--ignore-namespace", action="append", default=[], help="Namespace to skip in the pod check (repeatable)")
    parser.add_argument("--webhook-url", help="Webhook to POST the JSON summary to on failure")
    parser.add_argument("--lock-file", default="/tmp/talos-healthcheck.lock", help="Lock file that prevents overlapping runs")
    args = parser.parse_args()

    lock = open(args.lock_file, "w")
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        print("another health check is still running; skipping this run", file=sys.stderr)
        sys.exit(2)

    problems = []
    try:
        talos_ok, talos_output = check_talos_health(
            args.talosconfig, args.control_plane_nodes, args.worker_nodes, args.timeout
        )
        if not talos_ok:
            problems.append({"check": "talosctl health", "detail": talos_output.splitlines()[-20:]})

        alarms = check_etcd_alarms(args.talosconfig, args.control_plane_nodes)
        if alarms:
            problems.append({"check": "etcd alarms", "detail": alarms})
    except FileNotFoundError:
        print("talosctl not found on PATH", file=sys.stderr)
        sys.exit(2)

    try:
        config.load_kube_config(config_file=args.kubeconfig)
        v1 = client.CoreV1Api()
        not_ready, cordoned = check_nodes(v1)
        if not_ready:
            problems.append({"check": "node readiness", "detail": not_ready})
        if cordoned:
            problems.append({"check": "cordoned nodes", "detail": cordoned})

        pod_problems = check_pods(v1, args.restart_threshold, set(args.ignore_namespace))
        if pod_problems:
            problems.append({"check": "pod health", "detail": pod_problems})
    except Exception as exc:  # noqa: BLE001 - an unreachable API server is itself a finding
        problems.append({"check": "kubernetes api", "detail": [f"{type(exc).__name__}: {exc}"]})

    summary = {
        "checkedAt": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "healthy": not problems,
        "problems": problems,
    }
    print(json.dumps(summary, indent=2))

    if problems:
        if args.webhook_url:
            send_webhook(args.webhook_url, summary)
        sys.exit(1)
    sys.exit(0)


if __name__ == "__main__":
    main()

Notes

  • One node for talosctl health. talosctl health is built to talk to a single node, which then checks the rest of the cluster server-side; passing several addresses to --nodes makes it refuse to run. That is why the script sends the request through the first control plane node and passes the full membership with --control-plane-nodes and --worker-nodes. An earlier version of this script passed every node to --nodes, which doesn't work.
  • Keep the timeout under the cron interval. talosctl health blocks until its checks pass or --wait-timeout expires, and the default is 20 minutes. The lock file is a second line of defence: if a run is still going, the next one exits with code 2 instead of piling up.
  • etcd alarms matter more than they look. Talos' etcd maintenance guide notes that etcd's default space quota is 2 GiB and that etcd stops operations when the database exceeds it; the condition shows up as a NOSPACE row in talosctl etcd alarm list. The fix is defragmentation (talosctl -n <IP> etcd defrag, one node at a time) or a larger quota-backend-bytes, followed by talosctl etcd alarm disarm.
  • Restart counts are a blunt instrument on purpose. The kubelet resets a container's back-off timer after 10 minutes of clean running, but the restart count stays with the pod, so a pod that crashed six times last month and has been fine since keeps tripping the threshold until it is replaced. The back-off check (CrashLoopBackOff, ImagePullBackOff and friends, read from state.waiting.reason for init containers as well as app containers, since a failing init container leaves the pod Pending rather than Failed) is the precise signal; the restart threshold catches the slow leak without needing a metrics backend. Raise the threshold or add --ignore-namespace rather than deleting the check.
  • The Kubernetes API being unreachable is a finding, not a crash. Any exception from the client (a TLS error, a timeout, a 403 from an RBAC mistake) is reported as a kubernetes api problem so the webhook still fires.
  • Read-only by design. The script doesn't restart, uncordon or delete anything. A health check that also remediates needs far more thought about blast radius than a cron job should be trusted with unsupervised.
  • Don't point the talosconfig at the VIP. If the cluster uses a Talos Layer 2 VIP, Sidero Labs warns against using it as a talosconfig endpoint, because the VIP depends on etcd and the API server; list the node IPs instead.

Source