[{"data":1,"prerenderedAt":3440},["ShallowReactive",2],{"post:\u002F2026\u002F02\u002F04\u002Fpython-talos-health-checking-a-kubernetes-cluster-from-a-cron-job\u002F":3},{"post":4,"newer":3392,"older":3404,"related":3413,"series":3438},{"id":5,"title":6,"body":7,"canonical":3373,"categories":3374,"date":3377,"description":3378,"extension":3379,"featured":3380,"hero":3381,"image":3381,"meta":3382,"navigation":589,"path":3383,"readingTime":335,"seo":3384,"series":3381,"seriesOrder":3381,"sites":3385,"source":3381,"stem":3386,"tags":3387,"updated":3389,"url":3390,"__hash__":3391},"blog\u002Fblog\u002F2026\u002F02\u002F04\u002Fpython-talos-health-checking-a-kubernetes-cluster-from-a-cron-job.md","Python: Talos – Health-Checking a Kubernetes Cluster from a Cron Job",{"type":8,"value":9,"toc":3365},"minimark",[10,30,46,51,113,117,261,265,268,342,348,400,403,547,550,558,562,3164,3168,3284,3288,3361],[11,12,13,17,18,21,22,25,26,29],"p",{},[14,15,16],"code",{},"talosctl health"," is a genuinely good command: it waits for etcd to be healthy and its membership to be consistent, the control plane static pods and components to be ready, every node to report Ready and schedulable, and kube-proxy and CoreDNS to be up, and it tells you clearly when the control plane isn't happy. What it doesn't tell you is whether an application pod has been crash-looping for two days on an otherwise healthy cluster, or whether etcd has raised a ",[14,19,20],{},"NOSPACE"," alarm. I run this script from cron on a jump box that holds both a ",[14,23,24],{},"talosconfig"," and a ",[14,27,28],{},"kubeconfig",", and it combines the Talos-level checks with Kubernetes-level ones so both kinds of problem land in the same alert instead of two dashboards nobody checks at the same time.",[11,31,32,33,36,37,42,43,45],{},"The script is written against Talos 1.14 and the official ",[14,34,35],{},"kubernetes"," Python client (36.x at the time of writing). It pairs with the rolling upgrade workflow in ",[38,39,41],"a",{"href":40},"\u002F2026\u002F08\u002F26\u002Ftalos-linux-upgrading-a-cluster-without-a-maintenance-window\u002F","Talos Linux: Upgrading a Cluster Without a Maintenance Window",". ",[14,44,16],{}," already fails when a node is unschedulable, but the separate cordon check names the node in the alert, which is what you want after an upgrade that stopped halfway.",[47,48,50],"h2",{"id":49},"requirements","Requirements",[52,53,54,62,75,97,110],"ul",{},[55,56,57,58,61],"li",{},"Python 3.10 or later with the official client: ",[14,59,60],{},"pip install kubernetes",".",[55,63,64,67,68,71,72,74],{},[14,65,66],{},"talosctl"," on the ",[14,69,70],{},"PATH",". Sidero Labs recommends a ",[14,73,66],{}," version that matches the Talos version running on the cluster.",[55,76,77,78,80,81,84,85,88,89,92,93,96],{},"A ",[14,79,24],{}," whose context can reach every node you list. Sidero Labs' RBAC guide places etcd alarm management in the ",[14,82,83],{},"os:operator"," role, so ",[14,86,87],{},"os:reader"," isn't enough for the alarm check. Generate a dedicated client config for the monitoring box with ",[14,90,91],{},"talosctl config new --roles=os:operator healthcheck"," rather than copying the ",[14,94,95],{},"os:admin"," one.",[55,98,77,99,101,102,105,106,109],{},[14,100,28],{}," bound to a role that can ",[14,103,104],{},"list"," nodes and pods cluster-wide. The admin kubeconfig from ",[14,107,108],{},"talosctl kubeconfig"," works, but a read-only ServiceAccount token is the better habit.",[55,111,112],{},"Network reach from the cron host to the Talos API (TCP 50000) on the nodes and to the Kubernetes API server (TCP 6443).",[47,114,116],{"id":115},"parameters","Parameters",[118,119,120,136],"table",{},[121,122,123],"thead",{},[124,125,126,130,133],"tr",{},[127,128,129],"th",{},"Parameter",[127,131,132],{},"Required",[127,134,135],{},"Description",[137,138,139,153,165,180,193,205,221,233,245],"tbody",{},[124,140,141,147,150],{},[142,143,144],"td",{},[14,145,146],{},"--talosconfig",[142,148,149],{},"yes",[142,151,152],{},"Path to the Talos client config.",[124,154,155,160,162],{},[142,156,157],{},[14,158,159],{},"--kubeconfig",[142,161,149],{},[142,163,164],{},"Path to the kubeconfig.",[124,166,167,172,174],{},[142,168,169],{},[14,170,171],{},"--control-plane-nodes",[142,173,149],{},[142,175,176,177,179],{},"Control plane node IPs. The first one is the node ",[14,178,16],{}," runs through.",[124,181,182,187,190],{},[142,183,184],{},[14,185,186],{},"--worker-nodes",[142,188,189],{},"no",[142,191,192],{},"Worker node IPs. Omit on a cluster where every node is a control plane node.",[124,194,195,200,202],{},[142,196,197],{},[14,198,199],{},"--restart-threshold",[142,201,189],{},[142,203,204],{},"Restart count that counts as unhealthy (default 5).",[124,206,207,212,214],{},[142,208,209],{},[14,210,211],{},"--timeout",[142,213,189],{},[142,215,216,217,220],{},"Seconds passed to ",[14,218,219],{},"talosctl health --wait-timeout"," (default 120; the talosctl default is 20 minutes, far too long for cron).",[124,222,223,228,230],{},[142,224,225],{},[14,226,227],{},"--ignore-namespace",[142,229,189],{},[142,231,232],{},"Namespace to skip in the pod check, repeatable (for example a namespace full of short-lived test Jobs).",[124,234,235,240,242],{},[142,236,237],{},[14,238,239],{},"--webhook-url",[142,241,189],{},[142,243,244],{},"URL that receives the JSON summary as a POST when something is wrong.",[124,246,247,252,254],{},[142,248,249],{},[14,250,251],{},"--lock-file",[142,253,189],{},[142,255,256,257,260],{},"Lock file used to skip a run while the previous one is still going (default ",[14,258,259],{},"\u002Ftmp\u002Ftalos-healthcheck.lock",").",[47,262,264],{"id":263},"usage","Usage",[11,266,267],{},"Run it by hand against a three-node cluster where all three nodes are control plane nodes:",[269,270,275],"pre",{"className":271,"code":272,"language":273,"meta":274,"style":274},"language-bash shiki shiki-themes github-dark","python3 talos_cluster_healthcheck.py \\\n    --talosconfig \u002Fetc\u002Ftalos\u002Ftalosconfig \\\n    --kubeconfig \u002Fetc\u002Fkubernetes\u002Fkubeconfig \\\n    --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 \\\n    --webhook-url \"https:\u002F\u002Falerts.example.com\u002Fhooks\u002Ftalos\"\n","bash","",[14,276,277,294,305,316,333],{"__ignoreMap":274},[278,279,282,286,290],"span",{"class":280,"line":281},"line",1,[278,283,285],{"class":284},"svObZ","python3",[278,287,289],{"class":288},"sU2Wk"," talos_cluster_healthcheck.py",[278,291,293],{"class":292},"sDLfK"," \\\n",[278,295,297,300,303],{"class":280,"line":296},2,[278,298,299],{"class":292},"    --talosconfig",[278,301,302],{"class":288}," \u002Fetc\u002Ftalos\u002Ftalosconfig",[278,304,293],{"class":292},[278,306,308,311,314],{"class":280,"line":307},3,[278,309,310],{"class":292},"    --kubeconfig",[278,312,313],{"class":288}," \u002Fetc\u002Fkubernetes\u002Fkubeconfig",[278,315,293],{"class":292},[278,317,319,322,325,328,331],{"class":280,"line":318},4,[278,320,321],{"class":292},"    --control-plane-nodes",[278,323,324],{"class":292}," 10.10.20.11",[278,326,327],{"class":292}," 10.10.20.12",[278,329,330],{"class":292}," 10.10.20.13",[278,332,293],{"class":292},[278,334,336,339],{"class":280,"line":335},5,[278,337,338],{"class":292},"    --webhook-url",[278,340,341],{"class":288}," \"https:\u002F\u002Falerts.example.com\u002Fhooks\u002Ftalos\"\n",[11,343,344,345,347],{},"With dedicated workers, list them separately so ",[14,346,16],{}," knows what membership to expect:",[269,349,351],{"className":271,"code":350,"language":273,"meta":274,"style":274},"python3 talos_cluster_healthcheck.py \\\n    --talosconfig \u002Fetc\u002Ftalos\u002Ftalosconfig \\\n    --kubeconfig \u002Fetc\u002Fkubernetes\u002Fkubeconfig \\\n    --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 \\\n    --worker-nodes 10.10.20.21 10.10.20.22\n",[14,352,353,361,369,377,389],{"__ignoreMap":274},[278,354,355,357,359],{"class":280,"line":281},[278,356,285],{"class":284},[278,358,289],{"class":288},[278,360,293],{"class":292},[278,362,363,365,367],{"class":280,"line":296},[278,364,299],{"class":292},[278,366,302],{"class":288},[278,368,293],{"class":292},[278,370,371,373,375],{"class":280,"line":307},[278,372,310],{"class":292},[278,374,313],{"class":288},[278,376,293],{"class":292},[278,378,379,381,383,385,387],{"class":280,"line":318},[278,380,321],{"class":292},[278,382,324],{"class":292},[278,384,327],{"class":292},[278,386,330],{"class":292},[278,388,293],{"class":292},[278,390,391,394,397],{"class":280,"line":335},[278,392,393],{"class":292},"    --worker-nodes",[278,395,396],{"class":292}," 10.10.20.21",[278,398,399],{"class":292}," 10.10.20.22\n",[11,401,402],{},"Sample output when the control plane is fine but one worker was left cordoned and one pod is crash-looping:",[269,404,408],{"className":405,"code":406,"language":407,"meta":274,"style":274},"language-json shiki shiki-themes github-dark","{\n  \"checkedAt\": \"2026-02-03T14:20:01+00:00\",\n  \"healthy\": false,\n  \"problems\": [\n    {\n      \"check\": \"cordoned nodes\",\n      \"detail\": [\n        \"talos-w02\"\n      ]\n    },\n    {\n      \"check\": \"pod health\",\n      \"detail\": [\n        \"billing\u002Finvoice-worker-6c9d8f7b8-x2p4q: container invoice-worker is CrashLoopBackOff\"\n      ]\n    }\n  ]\n}\n","json",[14,409,410,416,430,442,450,455,468,476,482,488,494,499,511,518,524,529,535,541],{"__ignoreMap":274},[278,411,412],{"class":280,"line":281},[278,413,415],{"class":414},"s95oV","{\n",[278,417,418,421,424,427],{"class":280,"line":296},[278,419,420],{"class":292},"  \"checkedAt\"",[278,422,423],{"class":414},": ",[278,425,426],{"class":288},"\"2026-02-03T14:20:01+00:00\"",[278,428,429],{"class":414},",\n",[278,431,432,435,437,440],{"class":280,"line":307},[278,433,434],{"class":292},"  \"healthy\"",[278,436,423],{"class":414},[278,438,439],{"class":292},"false",[278,441,429],{"class":414},[278,443,444,447],{"class":280,"line":318},[278,445,446],{"class":292},"  \"problems\"",[278,448,449],{"class":414},": [\n",[278,451,452],{"class":280,"line":335},[278,453,454],{"class":414},"    {\n",[278,456,458,461,463,466],{"class":280,"line":457},6,[278,459,460],{"class":292},"      \"check\"",[278,462,423],{"class":414},[278,464,465],{"class":288},"\"cordoned nodes\"",[278,467,429],{"class":414},[278,469,471,474],{"class":280,"line":470},7,[278,472,473],{"class":292},"      \"detail\"",[278,475,449],{"class":414},[278,477,479],{"class":280,"line":478},8,[278,480,481],{"class":288},"        \"talos-w02\"\n",[278,483,485],{"class":280,"line":484},9,[278,486,487],{"class":414},"      ]\n",[278,489,491],{"class":280,"line":490},10,[278,492,493],{"class":414},"    },\n",[278,495,497],{"class":280,"line":496},11,[278,498,454],{"class":414},[278,500,502,504,506,509],{"class":280,"line":501},12,[278,503,460],{"class":292},[278,505,423],{"class":414},[278,507,508],{"class":288},"\"pod health\"",[278,510,429],{"class":414},[278,512,514,516],{"class":280,"line":513},13,[278,515,473],{"class":292},[278,517,449],{"class":414},[278,519,521],{"class":280,"line":520},14,[278,522,523],{"class":288},"        \"billing\u002Finvoice-worker-6c9d8f7b8-x2p4q: container invoice-worker is CrashLoopBackOff\"\n",[278,525,527],{"class":280,"line":526},15,[278,528,487],{"class":414},[278,530,532],{"class":280,"line":531},16,[278,533,534],{"class":414},"    }\n",[278,536,538],{"class":280,"line":537},17,[278,539,540],{"class":414},"  ]\n",[278,542,544],{"class":280,"line":543},18,[278,545,546],{"class":414},"}\n",[11,548,549],{},"Schedule it every 10 minutes and rely on the webhook for alerting (cron's own mail-on-output gets noisy fast). The exit code is 0 when healthy, 1 when a check found a problem, and 2 when the check itself couldn't run:",[269,551,556],{"className":552,"code":554,"language":555,"meta":274},[553],"language-text","*\u002F10 * * * * \u002Fusr\u002Fbin\u002Fpython3 \u002Fopt\u002Fscripts\u002Ftalos_cluster_healthcheck.py --talosconfig \u002Fetc\u002Ftalos\u002Ftalosconfig --kubeconfig \u002Fetc\u002Fkubernetes\u002Fkubeconfig --control-plane-nodes 10.10.20.11 10.10.20.12 10.10.20.13 --webhook-url \"https:\u002F\u002Falerts.example.com\u002Fhooks\u002Ftalos\" >> \u002Fvar\u002Flog\u002Ftalos-healthcheck.log 2>&1\n","text",[14,557,554],{"__ignoreMap":274},[47,559,561],{"id":560},"script","Script",[269,563,567],{"className":564,"code":565,"language":566,"meta":274,"style":274},"language-python shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"\ntalos_cluster_healthcheck.py\n\nHealth-checks a Talos Linux Kubernetes cluster from a cron job:\n\n  1. `talosctl health` (server-side), run through one control plane node\n     against an explicit list of control plane and worker nodes.\n  2. `talosctl etcd alarm list` across the control plane nodes (NOSPACE and\n     CORRUPT alarms stop etcd writes long before anything else complains).\n  3. Kubernetes node readiness and cordon state, through the Kubernetes API.\n  4. Pod health: Failed\u002FUnknown phases, init and app containers stuck in a\n     back-off waiting state, and containers over a restart threshold.\n\nPrints a JSON summary, posts it to an optional webhook on failure, and exits:\n    0  healthy\n    1  one or more checks found a problem\n    2  the health check itself could not run (lock held, talosctl missing, ...)\n\nRequires: Python 3.10+, the `kubernetes` package, `talosctl` on PATH.\n\"\"\"\n\nimport argparse\nimport fcntl\nimport json\nimport subprocess\nimport sys\nimport urllib.request\nfrom datetime import datetime, timezone\n\nfrom kubernetes import client, config\n\nBACKOFF_REASONS = {\"CrashLoopBackOff\", \"ImagePullBackOff\", \"ErrImagePull\", \"CreateContainerConfigError\"}\n\n\ndef run_talosctl(args, talosconfig, timeout):\n    \"\"\"Run a talosctl command and return (returncode, combined output).\"\"\"\n    cmd = [\"talosctl\", \"--talosconfig\", talosconfig, *args]\n    try:\n        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)\n    except subprocess.TimeoutExpired:\n        return 124, f\"timed out after {timeout}s: {' '.join(cmd)}\"\n    return result.returncode, (result.stdout + result.stderr).strip()\n\n\ndef check_talos_health(talosconfig, control_plane_nodes, worker_nodes, timeout_seconds):\n    \"\"\"talosctl health talks to exactly one node, which checks the whole cluster.\"\"\"\n    args = [\n        \"health\",\n        \"--nodes\", control_plane_nodes[0],\n        \"--control-plane-nodes\", \",\".join(control_plane_nodes),\n        \"--wait-timeout\", f\"{timeout_seconds}s\",\n    ]\n    if worker_nodes:\n        args += [\"--worker-nodes\", \",\".join(worker_nodes)]\n\n    code, output = run_talosctl(args, talosconfig, timeout_seconds + 30)\n    return code == 0, output\n\n\ndef check_etcd_alarms(talosconfig, control_plane_nodes):\n    \"\"\"Return a list of alarm lines; an empty list means no alarms.\"\"\"\n    code, output = run_talosctl(\n        [\"etcd\", \"alarm\", \"list\", \"--nodes\", \",\".join(control_plane_nodes)], talosconfig, 60\n    )\n    if code != 0:\n        return [f\"talosctl etcd alarm list failed: {output}\"]\n\n    lines = [line for line in output.splitlines() if line.strip()]\n    return [line for line in lines if not line.startswith(\"NODE\")]\n\n\ndef check_nodes(v1):\n    \"\"\"Return (not_ready, cordoned) lists of node names.\"\"\"\n    not_ready, cordoned = [], []\n    for node in v1.list_node(_request_timeout=30).items:\n        conditions = node.status.conditions or []\n        ready = next((c for c in conditions if c.type == \"Ready\"), None)\n        if ready is None or ready.status != \"True\":\n            reason = ready.reason if ready else \"no Ready condition\"\n            not_ready.append(f\"{node.metadata.name} ({reason})\")\n        if node.spec.unschedulable:\n            cordoned.append(node.metadata.name)\n    return not_ready, cordoned\n\n\ndef check_pods(v1, restart_threshold, ignore_namespaces):\n    \"\"\"Return a list of 'namespace\u002Fpod: reason' strings for unhealthy pods.\"\"\"\n    problems = []\n    for pod in v1.list_pod_for_all_namespaces(_request_timeout=60).items:\n        ns, name = pod.metadata.namespace, pod.metadata.name\n        if ns in ignore_namespaces:\n            continue\n\n        phase = pod.status.phase\n        if phase == \"Succeeded\":\n            continue\n        if phase in (\"Failed\", \"Unknown\"):\n            problems.append(f\"{ns}\u002F{name}: phase={phase} reason={pod.status.reason}\")\n            continue\n\n        # A failing init container keeps the pod Pending (Init:CrashLoopBackOff),\n        # so check init container statuses as well as the app containers.\n        statuses = [(\"init container\", cs) for cs in pod.status.init_container_statuses or []]\n        statuses += [(\"container\", cs) for cs in pod.status.container_statuses or []]\n        for kind, cs in statuses:\n            waiting = cs.state.waiting if cs.state else None\n            if waiting and waiting.reason in BACKOFF_REASONS:\n                problems.append(f\"{ns}\u002F{name}: {kind} {cs.name} is {waiting.reason}\")\n            elif cs.restart_count >= restart_threshold:\n                problems.append(f\"{ns}\u002F{name}: {kind} {cs.name} restarted {cs.restart_count} times\")\n    return problems\n\n\ndef send_webhook(webhook_url, summary):\n    \"\"\"POST the JSON summary. Best effort: a failed webhook is logged, not fatal.\"\"\"\n    payload = json.dumps(summary).encode(\"utf-8\")\n    request = urllib.request.Request(\n        webhook_url, data=payload, headers={\"Content-Type\": \"application\u002Fjson\"}, method=\"POST\"\n    )\n    try:\n        with urllib.request.urlopen(request, timeout=10) as response:\n            response.read()\n    except Exception as exc:  # noqa: BLE001 - alerting must never crash the check\n        print(f\"warning: webhook POST failed: {exc}\", file=sys.stderr)\n\n\ndef main():\n    parser = argparse.ArgumentParser(description=\"Health-check a Talos Linux Kubernetes cluster.\")\n    parser.add_argument(\"--talosconfig\", required=True, help=\"Path to talosconfig\")\n    parser.add_argument(\"--kubeconfig\", required=True, help=\"Path to kubeconfig\")\n    parser.add_argument(\"--control-plane-nodes\", nargs=\"+\", required=True, help=\"Control plane node IPs\")\n    parser.add_argument(\"--worker-nodes\", nargs=\"*\", default=[], help=\"Worker node IPs (omit if none)\")\n    parser.add_argument(\"--restart-threshold\", type=int, default=5, help=\"Restart count treated as unhealthy (default 5)\")\n    parser.add_argument(\"--timeout\", type=int, default=120, help=\"talosctl health --wait-timeout in seconds (default 120)\")\n    parser.add_argument(\"--ignore-namespace\", action=\"append\", default=[], help=\"Namespace to skip in the pod check (repeatable)\")\n    parser.add_argument(\"--webhook-url\", help=\"Webhook to POST the JSON summary to on failure\")\n    parser.add_argument(\"--lock-file\", default=\"\u002Ftmp\u002Ftalos-healthcheck.lock\", help=\"Lock file that prevents overlapping runs\")\n    args = parser.parse_args()\n\n    lock = open(args.lock_file, \"w\")\n    try:\n        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)\n    except BlockingIOError:\n        print(\"another health check is still running; skipping this run\", file=sys.stderr)\n        sys.exit(2)\n\n    problems = []\n    try:\n        talos_ok, talos_output = check_talos_health(\n            args.talosconfig, args.control_plane_nodes, args.worker_nodes, args.timeout\n        )\n        if not talos_ok:\n            problems.append({\"check\": \"talosctl health\", \"detail\": talos_output.splitlines()[-20:]})\n\n        alarms = check_etcd_alarms(args.talosconfig, args.control_plane_nodes)\n        if alarms:\n            problems.append({\"check\": \"etcd alarms\", \"detail\": alarms})\n    except FileNotFoundError:\n        print(\"talosctl not found on PATH\", file=sys.stderr)\n        sys.exit(2)\n\n    try:\n        config.load_kube_config(config_file=args.kubeconfig)\n        v1 = client.CoreV1Api()\n        not_ready, cordoned = check_nodes(v1)\n        if not_ready:\n            problems.append({\"check\": \"node readiness\", \"detail\": not_ready})\n        if cordoned:\n            problems.append({\"check\": \"cordoned nodes\", \"detail\": cordoned})\n\n        pod_problems = check_pods(v1, args.restart_threshold, set(args.ignore_namespace))\n        if pod_problems:\n            problems.append({\"check\": \"pod health\", \"detail\": pod_problems})\n    except Exception as exc:  # noqa: BLE001 - an unreachable API server is itself a finding\n        problems.append({\"check\": \"kubernetes api\", \"detail\": [f\"{type(exc).__name__}: {exc}\"]})\n\n    summary = {\n        \"checkedAt\": datetime.now(timezone.utc).isoformat(timespec=\"seconds\"),\n        \"healthy\": not problems,\n        \"problems\": problems,\n    }\n    print(json.dumps(summary, indent=2))\n\n    if problems:\n        if args.webhook_url:\n            send_webhook(args.webhook_url, summary)\n        sys.exit(1)\n    sys.exit(0)\n\n\nif __name__ == \"__main__\":\n    main()\n","python",[14,568,569,575,580,585,591,596,600,605,610,615,620,625,630,635,639,644,649,654,659,664,670,675,680,690,698,706,714,722,730,744,749,762,767,800,805,810,822,828,857,866,904,913,954,969,974,979,990,996,1007,1015,1030,1044,1069,1075,1084,1105,1110,1129,1146,1151,1156,1167,1173,1183,1217,1223,1237,1261,1266,1295,1325,1330,1335,1346,1352,1363,1388,1405,1447,1475,1496,1528,1536,1542,1550,1555,1560,1571,1577,1587,1609,1620,1633,1639,1644,1655,1670,1675,1697,1748,1753,1758,1764,1770,1802,1829,1843,1864,1886,1940,1955,2007,2015,2020,2025,2036,2042,2058,2069,2109,2114,2121,2146,2152,2169,2202,2207,2212,2223,2244,2273,2300,2337,2372,2410,2446,2481,2500,2528,2538,2543,2562,2569,2589,2599,2617,2628,2633,2642,2649,2660,2666,2672,2682,2713,2718,2729,2737,2756,2766,2784,2793,2798,2805,2819,2830,2841,2849,2868,2876,2894,2899,2916,2924,2942,2956,3002,3007,3018,3038,3052,3061,3066,3085,3090,3098,3106,3112,3122,3132,3137,3142,3158],{"__ignoreMap":274},[278,570,571],{"class":280,"line":281},[278,572,574],{"class":573},"sAwPA","#!\u002Fusr\u002Fbin\u002Fenv python3\n",[278,576,577],{"class":280,"line":296},[278,578,579],{"class":288},"\"\"\"\n",[278,581,582],{"class":280,"line":307},[278,583,584],{"class":288},"talos_cluster_healthcheck.py\n",[278,586,587],{"class":280,"line":318},[278,588,590],{"emptyLinePlaceholder":589},true,"\n",[278,592,593],{"class":280,"line":335},[278,594,595],{"class":288},"Health-checks a Talos Linux Kubernetes cluster from a cron job:\n",[278,597,598],{"class":280,"line":457},[278,599,590],{"emptyLinePlaceholder":589},[278,601,602],{"class":280,"line":470},[278,603,604],{"class":288},"  1. `talosctl health` (server-side), run through one control plane node\n",[278,606,607],{"class":280,"line":478},[278,608,609],{"class":288},"     against an explicit list of control plane and worker nodes.\n",[278,611,612],{"class":280,"line":484},[278,613,614],{"class":288},"  2. `talosctl etcd alarm list` across the control plane nodes (NOSPACE and\n",[278,616,617],{"class":280,"line":490},[278,618,619],{"class":288},"     CORRUPT alarms stop etcd writes long before anything else complains).\n",[278,621,622],{"class":280,"line":496},[278,623,624],{"class":288},"  3. Kubernetes node readiness and cordon state, through the Kubernetes API.\n",[278,626,627],{"class":280,"line":501},[278,628,629],{"class":288},"  4. Pod health: Failed\u002FUnknown phases, init and app containers stuck in a\n",[278,631,632],{"class":280,"line":513},[278,633,634],{"class":288},"     back-off waiting state, and containers over a restart threshold.\n",[278,636,637],{"class":280,"line":520},[278,638,590],{"emptyLinePlaceholder":589},[278,640,641],{"class":280,"line":526},[278,642,643],{"class":288},"Prints a JSON summary, posts it to an optional webhook on failure, and exits:\n",[278,645,646],{"class":280,"line":531},[278,647,648],{"class":288},"    0  healthy\n",[278,650,651],{"class":280,"line":537},[278,652,653],{"class":288},"    1  one or more checks found a problem\n",[278,655,656],{"class":280,"line":543},[278,657,658],{"class":288},"    2  the health check itself could not run (lock held, talosctl missing, ...)\n",[278,660,662],{"class":280,"line":661},19,[278,663,590],{"emptyLinePlaceholder":589},[278,665,667],{"class":280,"line":666},20,[278,668,669],{"class":288},"Requires: Python 3.10+, the `kubernetes` package, `talosctl` on PATH.\n",[278,671,673],{"class":280,"line":672},21,[278,674,579],{"class":288},[278,676,678],{"class":280,"line":677},22,[278,679,590],{"emptyLinePlaceholder":589},[278,681,683,687],{"class":280,"line":682},23,[278,684,686],{"class":685},"snl16","import",[278,688,689],{"class":414}," argparse\n",[278,691,693,695],{"class":280,"line":692},24,[278,694,686],{"class":685},[278,696,697],{"class":414}," fcntl\n",[278,699,701,703],{"class":280,"line":700},25,[278,702,686],{"class":685},[278,704,705],{"class":414}," json\n",[278,707,709,711],{"class":280,"line":708},26,[278,710,686],{"class":685},[278,712,713],{"class":414}," subprocess\n",[278,715,717,719],{"class":280,"line":716},27,[278,718,686],{"class":685},[278,720,721],{"class":414}," sys\n",[278,723,725,727],{"class":280,"line":724},28,[278,726,686],{"class":685},[278,728,729],{"class":414}," urllib.request\n",[278,731,733,736,739,741],{"class":280,"line":732},29,[278,734,735],{"class":685},"from",[278,737,738],{"class":414}," datetime ",[278,740,686],{"class":685},[278,742,743],{"class":414}," datetime, timezone\n",[278,745,747],{"class":280,"line":746},30,[278,748,590],{"emptyLinePlaceholder":589},[278,750,752,754,757,759],{"class":280,"line":751},31,[278,753,735],{"class":685},[278,755,756],{"class":414}," kubernetes ",[278,758,686],{"class":685},[278,760,761],{"class":414}," client, config\n",[278,763,765],{"class":280,"line":764},32,[278,766,590],{"emptyLinePlaceholder":589},[278,768,770,773,776,779,782,785,788,790,793,795,798],{"class":280,"line":769},33,[278,771,772],{"class":292},"BACKOFF_REASONS",[278,774,775],{"class":685}," =",[278,777,778],{"class":414}," {",[278,780,781],{"class":288},"\"CrashLoopBackOff\"",[278,783,784],{"class":414},", ",[278,786,787],{"class":288},"\"ImagePullBackOff\"",[278,789,784],{"class":414},[278,791,792],{"class":288},"\"ErrImagePull\"",[278,794,784],{"class":414},[278,796,797],{"class":288},"\"CreateContainerConfigError\"",[278,799,546],{"class":414},[278,801,803],{"class":280,"line":802},34,[278,804,590],{"emptyLinePlaceholder":589},[278,806,808],{"class":280,"line":807},35,[278,809,590],{"emptyLinePlaceholder":589},[278,811,813,816,819],{"class":280,"line":812},36,[278,814,815],{"class":685},"def",[278,817,818],{"class":284}," run_talosctl",[278,820,821],{"class":414},"(args, talosconfig, timeout):\n",[278,823,825],{"class":280,"line":824},37,[278,826,827],{"class":288},"    \"\"\"Run a talosctl command and return (returncode, combined output).\"\"\"\n",[278,829,831,834,837,840,843,845,848,851,854],{"class":280,"line":830},38,[278,832,833],{"class":414},"    cmd ",[278,835,836],{"class":685},"=",[278,838,839],{"class":414}," [",[278,841,842],{"class":288},"\"talosctl\"",[278,844,784],{"class":414},[278,846,847],{"class":288},"\"--talosconfig\"",[278,849,850],{"class":414},", talosconfig, ",[278,852,853],{"class":685},"*",[278,855,856],{"class":414},"args]\n",[278,858,860,863],{"class":280,"line":859},39,[278,861,862],{"class":685},"    try",[278,864,865],{"class":414},":\n",[278,867,869,872,874,877,881,883,886,888,890,892,894,896,899,901],{"class":280,"line":868},40,[278,870,871],{"class":414},"        result ",[278,873,836],{"class":685},[278,875,876],{"class":414}," subprocess.run(cmd, ",[278,878,880],{"class":879},"s9osk","capture_output",[278,882,836],{"class":685},[278,884,885],{"class":292},"True",[278,887,784],{"class":414},[278,889,555],{"class":879},[278,891,836],{"class":685},[278,893,885],{"class":292},[278,895,784],{"class":414},[278,897,898],{"class":879},"timeout",[278,900,836],{"class":685},[278,902,903],{"class":414},"timeout)\n",[278,905,907,910],{"class":280,"line":906},41,[278,908,909],{"class":685},"    except",[278,911,912],{"class":414}," subprocess.TimeoutExpired:\n",[278,914,916,919,922,924,927,930,933,935,938,941,943,946,949,951],{"class":280,"line":915},42,[278,917,918],{"class":685},"        return",[278,920,921],{"class":292}," 124",[278,923,784],{"class":414},[278,925,926],{"class":685},"f",[278,928,929],{"class":288},"\"timed out after ",[278,931,932],{"class":292},"{",[278,934,898],{"class":414},[278,936,937],{"class":292},"}",[278,939,940],{"class":288},"s: ",[278,942,932],{"class":292},[278,944,945],{"class":288},"' '",[278,947,948],{"class":414},".join(cmd)",[278,950,937],{"class":292},[278,952,953],{"class":288},"\"\n",[278,955,957,960,963,966],{"class":280,"line":956},43,[278,958,959],{"class":685},"    return",[278,961,962],{"class":414}," result.returncode, (result.stdout ",[278,964,965],{"class":685},"+",[278,967,968],{"class":414}," result.stderr).strip()\n",[278,970,972],{"class":280,"line":971},44,[278,973,590],{"emptyLinePlaceholder":589},[278,975,977],{"class":280,"line":976},45,[278,978,590],{"emptyLinePlaceholder":589},[278,980,982,984,987],{"class":280,"line":981},46,[278,983,815],{"class":685},[278,985,986],{"class":284}," check_talos_health",[278,988,989],{"class":414},"(talosconfig, control_plane_nodes, worker_nodes, timeout_seconds):\n",[278,991,993],{"class":280,"line":992},47,[278,994,995],{"class":288},"    \"\"\"talosctl health talks to exactly one node, which checks the whole cluster.\"\"\"\n",[278,997,999,1002,1004],{"class":280,"line":998},48,[278,1000,1001],{"class":414},"    args ",[278,1003,836],{"class":685},[278,1005,1006],{"class":414}," [\n",[278,1008,1010,1013],{"class":280,"line":1009},49,[278,1011,1012],{"class":288},"        \"health\"",[278,1014,429],{"class":414},[278,1016,1018,1021,1024,1027],{"class":280,"line":1017},50,[278,1019,1020],{"class":288},"        \"--nodes\"",[278,1022,1023],{"class":414},", control_plane_nodes[",[278,1025,1026],{"class":292},"0",[278,1028,1029],{"class":414},"],\n",[278,1031,1033,1036,1038,1041],{"class":280,"line":1032},51,[278,1034,1035],{"class":288},"        \"--control-plane-nodes\"",[278,1037,784],{"class":414},[278,1039,1040],{"class":288},"\",\"",[278,1042,1043],{"class":414},".join(control_plane_nodes),\n",[278,1045,1047,1050,1052,1054,1057,1059,1062,1064,1067],{"class":280,"line":1046},52,[278,1048,1049],{"class":288},"        \"--wait-timeout\"",[278,1051,784],{"class":414},[278,1053,926],{"class":685},[278,1055,1056],{"class":288},"\"",[278,1058,932],{"class":292},[278,1060,1061],{"class":414},"timeout_seconds",[278,1063,937],{"class":292},[278,1065,1066],{"class":288},"s\"",[278,1068,429],{"class":414},[278,1070,1072],{"class":280,"line":1071},53,[278,1073,1074],{"class":414},"    ]\n",[278,1076,1078,1081],{"class":280,"line":1077},54,[278,1079,1080],{"class":685},"    if",[278,1082,1083],{"class":414}," worker_nodes:\n",[278,1085,1087,1090,1093,1095,1098,1100,1102],{"class":280,"line":1086},55,[278,1088,1089],{"class":414},"        args ",[278,1091,1092],{"class":685},"+=",[278,1094,839],{"class":414},[278,1096,1097],{"class":288},"\"--worker-nodes\"",[278,1099,784],{"class":414},[278,1101,1040],{"class":288},[278,1103,1104],{"class":414},".join(worker_nodes)]\n",[278,1106,1108],{"class":280,"line":1107},56,[278,1109,590],{"emptyLinePlaceholder":589},[278,1111,1113,1116,1118,1121,1123,1126],{"class":280,"line":1112},57,[278,1114,1115],{"class":414},"    code, output ",[278,1117,836],{"class":685},[278,1119,1120],{"class":414}," run_talosctl(args, talosconfig, timeout_seconds ",[278,1122,965],{"class":685},[278,1124,1125],{"class":292}," 30",[278,1127,1128],{"class":414},")\n",[278,1130,1132,1134,1137,1140,1143],{"class":280,"line":1131},58,[278,1133,959],{"class":685},[278,1135,1136],{"class":414}," code ",[278,1138,1139],{"class":685},"==",[278,1141,1142],{"class":292}," 0",[278,1144,1145],{"class":414},", output\n",[278,1147,1149],{"class":280,"line":1148},59,[278,1150,590],{"emptyLinePlaceholder":589},[278,1152,1154],{"class":280,"line":1153},60,[278,1155,590],{"emptyLinePlaceholder":589},[278,1157,1159,1161,1164],{"class":280,"line":1158},61,[278,1160,815],{"class":685},[278,1162,1163],{"class":284}," check_etcd_alarms",[278,1165,1166],{"class":414},"(talosconfig, control_plane_nodes):\n",[278,1168,1170],{"class":280,"line":1169},62,[278,1171,1172],{"class":288},"    \"\"\"Return a list of alarm lines; an empty list means no alarms.\"\"\"\n",[278,1174,1176,1178,1180],{"class":280,"line":1175},63,[278,1177,1115],{"class":414},[278,1179,836],{"class":685},[278,1181,1182],{"class":414}," run_talosctl(\n",[278,1184,1186,1189,1192,1194,1197,1199,1202,1204,1207,1209,1211,1214],{"class":280,"line":1185},64,[278,1187,1188],{"class":414},"        [",[278,1190,1191],{"class":288},"\"etcd\"",[278,1193,784],{"class":414},[278,1195,1196],{"class":288},"\"alarm\"",[278,1198,784],{"class":414},[278,1200,1201],{"class":288},"\"list\"",[278,1203,784],{"class":414},[278,1205,1206],{"class":288},"\"--nodes\"",[278,1208,784],{"class":414},[278,1210,1040],{"class":288},[278,1212,1213],{"class":414},".join(control_plane_nodes)], talosconfig, ",[278,1215,1216],{"class":292},"60\n",[278,1218,1220],{"class":280,"line":1219},65,[278,1221,1222],{"class":414},"    )\n",[278,1224,1226,1228,1230,1233,1235],{"class":280,"line":1225},66,[278,1227,1080],{"class":685},[278,1229,1136],{"class":414},[278,1231,1232],{"class":685},"!=",[278,1234,1142],{"class":292},[278,1236,865],{"class":414},[278,1238,1240,1242,1244,1246,1249,1251,1254,1256,1258],{"class":280,"line":1239},67,[278,1241,918],{"class":685},[278,1243,839],{"class":414},[278,1245,926],{"class":685},[278,1247,1248],{"class":288},"\"talosctl etcd alarm list failed: ",[278,1250,932],{"class":292},[278,1252,1253],{"class":414},"output",[278,1255,937],{"class":292},[278,1257,1056],{"class":288},[278,1259,1260],{"class":414},"]\n",[278,1262,1264],{"class":280,"line":1263},68,[278,1265,590],{"emptyLinePlaceholder":589},[278,1267,1269,1272,1274,1277,1280,1283,1286,1289,1292],{"class":280,"line":1268},69,[278,1270,1271],{"class":414},"    lines ",[278,1273,836],{"class":685},[278,1275,1276],{"class":414}," [line ",[278,1278,1279],{"class":685},"for",[278,1281,1282],{"class":414}," line ",[278,1284,1285],{"class":685},"in",[278,1287,1288],{"class":414}," output.splitlines() ",[278,1290,1291],{"class":685},"if",[278,1293,1294],{"class":414}," line.strip()]\n",[278,1296,1298,1300,1302,1304,1306,1308,1311,1313,1316,1319,1322],{"class":280,"line":1297},70,[278,1299,959],{"class":685},[278,1301,1276],{"class":414},[278,1303,1279],{"class":685},[278,1305,1282],{"class":414},[278,1307,1285],{"class":685},[278,1309,1310],{"class":414}," lines ",[278,1312,1291],{"class":685},[278,1314,1315],{"class":685}," not",[278,1317,1318],{"class":414}," line.startswith(",[278,1320,1321],{"class":288},"\"NODE\"",[278,1323,1324],{"class":414},")]\n",[278,1326,1328],{"class":280,"line":1327},71,[278,1329,590],{"emptyLinePlaceholder":589},[278,1331,1333],{"class":280,"line":1332},72,[278,1334,590],{"emptyLinePlaceholder":589},[278,1336,1338,1340,1343],{"class":280,"line":1337},73,[278,1339,815],{"class":685},[278,1341,1342],{"class":284}," check_nodes",[278,1344,1345],{"class":414},"(v1):\n",[278,1347,1349],{"class":280,"line":1348},74,[278,1350,1351],{"class":288},"    \"\"\"Return (not_ready, cordoned) lists of node names.\"\"\"\n",[278,1353,1355,1358,1360],{"class":280,"line":1354},75,[278,1356,1357],{"class":414},"    not_ready, cordoned ",[278,1359,836],{"class":685},[278,1361,1362],{"class":414}," [], []\n",[278,1364,1366,1369,1372,1374,1377,1380,1382,1385],{"class":280,"line":1365},76,[278,1367,1368],{"class":685},"    for",[278,1370,1371],{"class":414}," node ",[278,1373,1285],{"class":685},[278,1375,1376],{"class":414}," v1.list_node(",[278,1378,1379],{"class":879},"_request_timeout",[278,1381,836],{"class":685},[278,1383,1384],{"class":292},"30",[278,1386,1387],{"class":414},").items:\n",[278,1389,1391,1394,1396,1399,1402],{"class":280,"line":1390},77,[278,1392,1393],{"class":414},"        conditions ",[278,1395,836],{"class":685},[278,1397,1398],{"class":414}," node.status.conditions ",[278,1400,1401],{"class":685},"or",[278,1403,1404],{"class":414}," []\n",[278,1406,1408,1411,1413,1416,1419,1421,1424,1426,1429,1431,1434,1436,1439,1442,1445],{"class":280,"line":1407},78,[278,1409,1410],{"class":414},"        ready ",[278,1412,836],{"class":685},[278,1414,1415],{"class":292}," next",[278,1417,1418],{"class":414},"((c ",[278,1420,1279],{"class":685},[278,1422,1423],{"class":414}," c ",[278,1425,1285],{"class":685},[278,1427,1428],{"class":414}," conditions ",[278,1430,1291],{"class":685},[278,1432,1433],{"class":414}," c.type ",[278,1435,1139],{"class":685},[278,1437,1438],{"class":288}," \"Ready\"",[278,1440,1441],{"class":414},"), ",[278,1443,1444],{"class":292},"None",[278,1446,1128],{"class":414},[278,1448,1450,1453,1456,1459,1462,1465,1468,1470,1473],{"class":280,"line":1449},79,[278,1451,1452],{"class":685},"        if",[278,1454,1455],{"class":414}," ready ",[278,1457,1458],{"class":685},"is",[278,1460,1461],{"class":292}," None",[278,1463,1464],{"class":685}," or",[278,1466,1467],{"class":414}," ready.status ",[278,1469,1232],{"class":685},[278,1471,1472],{"class":288}," \"True\"",[278,1474,865],{"class":414},[278,1476,1478,1481,1483,1486,1488,1490,1493],{"class":280,"line":1477},80,[278,1479,1480],{"class":414},"            reason ",[278,1482,836],{"class":685},[278,1484,1485],{"class":414}," ready.reason ",[278,1487,1291],{"class":685},[278,1489,1455],{"class":414},[278,1491,1492],{"class":685},"else",[278,1494,1495],{"class":288}," \"no Ready condition\"\n",[278,1497,1499,1502,1504,1506,1508,1511,1513,1516,1518,1521,1523,1526],{"class":280,"line":1498},81,[278,1500,1501],{"class":414},"            not_ready.append(",[278,1503,926],{"class":685},[278,1505,1056],{"class":288},[278,1507,932],{"class":292},[278,1509,1510],{"class":414},"node.metadata.name",[278,1512,937],{"class":292},[278,1514,1515],{"class":288}," (",[278,1517,932],{"class":292},[278,1519,1520],{"class":414},"reason",[278,1522,937],{"class":292},[278,1524,1525],{"class":288},")\"",[278,1527,1128],{"class":414},[278,1529,1531,1533],{"class":280,"line":1530},82,[278,1532,1452],{"class":685},[278,1534,1535],{"class":414}," node.spec.unschedulable:\n",[278,1537,1539],{"class":280,"line":1538},83,[278,1540,1541],{"class":414},"            cordoned.append(node.metadata.name)\n",[278,1543,1545,1547],{"class":280,"line":1544},84,[278,1546,959],{"class":685},[278,1548,1549],{"class":414}," not_ready, cordoned\n",[278,1551,1553],{"class":280,"line":1552},85,[278,1554,590],{"emptyLinePlaceholder":589},[278,1556,1558],{"class":280,"line":1557},86,[278,1559,590],{"emptyLinePlaceholder":589},[278,1561,1563,1565,1568],{"class":280,"line":1562},87,[278,1564,815],{"class":685},[278,1566,1567],{"class":284}," check_pods",[278,1569,1570],{"class":414},"(v1, restart_threshold, ignore_namespaces):\n",[278,1572,1574],{"class":280,"line":1573},88,[278,1575,1576],{"class":288},"    \"\"\"Return a list of 'namespace\u002Fpod: reason' strings for unhealthy pods.\"\"\"\n",[278,1578,1580,1583,1585],{"class":280,"line":1579},89,[278,1581,1582],{"class":414},"    problems ",[278,1584,836],{"class":685},[278,1586,1404],{"class":414},[278,1588,1590,1592,1595,1597,1600,1602,1604,1607],{"class":280,"line":1589},90,[278,1591,1368],{"class":685},[278,1593,1594],{"class":414}," pod ",[278,1596,1285],{"class":685},[278,1598,1599],{"class":414}," v1.list_pod_for_all_namespaces(",[278,1601,1379],{"class":879},[278,1603,836],{"class":685},[278,1605,1606],{"class":292},"60",[278,1608,1387],{"class":414},[278,1610,1612,1615,1617],{"class":280,"line":1611},91,[278,1613,1614],{"class":414},"        ns, name ",[278,1616,836],{"class":685},[278,1618,1619],{"class":414}," pod.metadata.namespace, pod.metadata.name\n",[278,1621,1623,1625,1628,1630],{"class":280,"line":1622},92,[278,1624,1452],{"class":685},[278,1626,1627],{"class":414}," ns ",[278,1629,1285],{"class":685},[278,1631,1632],{"class":414}," ignore_namespaces:\n",[278,1634,1636],{"class":280,"line":1635},93,[278,1637,1638],{"class":685},"            continue\n",[278,1640,1642],{"class":280,"line":1641},94,[278,1643,590],{"emptyLinePlaceholder":589},[278,1645,1647,1650,1652],{"class":280,"line":1646},95,[278,1648,1649],{"class":414},"        phase ",[278,1651,836],{"class":685},[278,1653,1654],{"class":414}," pod.status.phase\n",[278,1656,1658,1660,1663,1665,1668],{"class":280,"line":1657},96,[278,1659,1452],{"class":685},[278,1661,1662],{"class":414}," phase ",[278,1664,1139],{"class":685},[278,1666,1667],{"class":288}," \"Succeeded\"",[278,1669,865],{"class":414},[278,1671,1673],{"class":280,"line":1672},97,[278,1674,1638],{"class":685},[278,1676,1678,1680,1682,1684,1686,1689,1691,1694],{"class":280,"line":1677},98,[278,1679,1452],{"class":685},[278,1681,1662],{"class":414},[278,1683,1285],{"class":685},[278,1685,1515],{"class":414},[278,1687,1688],{"class":288},"\"Failed\"",[278,1690,784],{"class":414},[278,1692,1693],{"class":288},"\"Unknown\"",[278,1695,1696],{"class":414},"):\n",[278,1698,1700,1703,1705,1707,1709,1712,1714,1717,1719,1722,1724,1727,1729,1732,1734,1737,1739,1742,1744,1746],{"class":280,"line":1699},99,[278,1701,1702],{"class":414},"            problems.append(",[278,1704,926],{"class":685},[278,1706,1056],{"class":288},[278,1708,932],{"class":292},[278,1710,1711],{"class":414},"ns",[278,1713,937],{"class":292},[278,1715,1716],{"class":288},"\u002F",[278,1718,932],{"class":292},[278,1720,1721],{"class":414},"name",[278,1723,937],{"class":292},[278,1725,1726],{"class":288},": phase=",[278,1728,932],{"class":292},[278,1730,1731],{"class":414},"phase",[278,1733,937],{"class":292},[278,1735,1736],{"class":288}," reason=",[278,1738,932],{"class":292},[278,1740,1741],{"class":414},"pod.status.reason",[278,1743,937],{"class":292},[278,1745,1056],{"class":288},[278,1747,1128],{"class":414},[278,1749,1751],{"class":280,"line":1750},100,[278,1752,1638],{"class":685},[278,1754,1756],{"class":280,"line":1755},101,[278,1757,590],{"emptyLinePlaceholder":589},[278,1759,1761],{"class":280,"line":1760},102,[278,1762,1763],{"class":573},"        # A failing init container keeps the pod Pending (Init:CrashLoopBackOff),\n",[278,1765,1767],{"class":280,"line":1766},103,[278,1768,1769],{"class":573},"        # so check init container statuses as well as the app containers.\n",[278,1771,1773,1776,1778,1781,1784,1787,1789,1792,1794,1797,1799],{"class":280,"line":1772},104,[278,1774,1775],{"class":414},"        statuses ",[278,1777,836],{"class":685},[278,1779,1780],{"class":414}," [(",[278,1782,1783],{"class":288},"\"init container\"",[278,1785,1786],{"class":414},", cs) ",[278,1788,1279],{"class":685},[278,1790,1791],{"class":414}," cs ",[278,1793,1285],{"class":685},[278,1795,1796],{"class":414}," pod.status.init_container_statuses ",[278,1798,1401],{"class":685},[278,1800,1801],{"class":414}," []]\n",[278,1803,1805,1807,1809,1811,1814,1816,1818,1820,1822,1825,1827],{"class":280,"line":1804},105,[278,1806,1775],{"class":414},[278,1808,1092],{"class":685},[278,1810,1780],{"class":414},[278,1812,1813],{"class":288},"\"container\"",[278,1815,1786],{"class":414},[278,1817,1279],{"class":685},[278,1819,1791],{"class":414},[278,1821,1285],{"class":685},[278,1823,1824],{"class":414}," pod.status.container_statuses ",[278,1826,1401],{"class":685},[278,1828,1801],{"class":414},[278,1830,1832,1835,1838,1840],{"class":280,"line":1831},106,[278,1833,1834],{"class":685},"        for",[278,1836,1837],{"class":414}," kind, cs ",[278,1839,1285],{"class":685},[278,1841,1842],{"class":414}," statuses:\n",[278,1844,1846,1849,1851,1854,1856,1859,1861],{"class":280,"line":1845},107,[278,1847,1848],{"class":414},"            waiting ",[278,1850,836],{"class":685},[278,1852,1853],{"class":414}," cs.state.waiting ",[278,1855,1291],{"class":685},[278,1857,1858],{"class":414}," cs.state ",[278,1860,1492],{"class":685},[278,1862,1863],{"class":292}," None\n",[278,1865,1867,1870,1873,1876,1879,1881,1884],{"class":280,"line":1866},108,[278,1868,1869],{"class":685},"            if",[278,1871,1872],{"class":414}," waiting ",[278,1874,1875],{"class":685},"and",[278,1877,1878],{"class":414}," waiting.reason ",[278,1880,1285],{"class":685},[278,1882,1883],{"class":292}," BACKOFF_REASONS",[278,1885,865],{"class":414},[278,1887,1889,1892,1894,1896,1898,1900,1902,1904,1906,1908,1910,1912,1914,1917,1919,1921,1924,1926,1929,1931,1934,1936,1938],{"class":280,"line":1888},109,[278,1890,1891],{"class":414},"                problems.append(",[278,1893,926],{"class":685},[278,1895,1056],{"class":288},[278,1897,932],{"class":292},[278,1899,1711],{"class":414},[278,1901,937],{"class":292},[278,1903,1716],{"class":288},[278,1905,932],{"class":292},[278,1907,1721],{"class":414},[278,1909,937],{"class":292},[278,1911,423],{"class":288},[278,1913,932],{"class":292},[278,1915,1916],{"class":414},"kind",[278,1918,937],{"class":292},[278,1920,778],{"class":292},[278,1922,1923],{"class":414},"cs.name",[278,1925,937],{"class":292},[278,1927,1928],{"class":288}," is ",[278,1930,932],{"class":292},[278,1932,1933],{"class":414},"waiting.reason",[278,1935,937],{"class":292},[278,1937,1056],{"class":288},[278,1939,1128],{"class":414},[278,1941,1943,1946,1949,1952],{"class":280,"line":1942},110,[278,1944,1945],{"class":685},"            elif",[278,1947,1948],{"class":414}," cs.restart_count ",[278,1950,1951],{"class":685},">=",[278,1953,1954],{"class":414}," restart_threshold:\n",[278,1956,1958,1960,1962,1964,1966,1968,1970,1972,1974,1976,1978,1980,1982,1984,1986,1988,1990,1992,1995,1997,2000,2002,2005],{"class":280,"line":1957},111,[278,1959,1891],{"class":414},[278,1961,926],{"class":685},[278,1963,1056],{"class":288},[278,1965,932],{"class":292},[278,1967,1711],{"class":414},[278,1969,937],{"class":292},[278,1971,1716],{"class":288},[278,1973,932],{"class":292},[278,1975,1721],{"class":414},[278,1977,937],{"class":292},[278,1979,423],{"class":288},[278,1981,932],{"class":292},[278,1983,1916],{"class":414},[278,1985,937],{"class":292},[278,1987,778],{"class":292},[278,1989,1923],{"class":414},[278,1991,937],{"class":292},[278,1993,1994],{"class":288}," restarted ",[278,1996,932],{"class":292},[278,1998,1999],{"class":414},"cs.restart_count",[278,2001,937],{"class":292},[278,2003,2004],{"class":288}," times\"",[278,2006,1128],{"class":414},[278,2008,2010,2012],{"class":280,"line":2009},112,[278,2011,959],{"class":685},[278,2013,2014],{"class":414}," problems\n",[278,2016,2018],{"class":280,"line":2017},113,[278,2019,590],{"emptyLinePlaceholder":589},[278,2021,2023],{"class":280,"line":2022},114,[278,2024,590],{"emptyLinePlaceholder":589},[278,2026,2028,2030,2033],{"class":280,"line":2027},115,[278,2029,815],{"class":685},[278,2031,2032],{"class":284}," send_webhook",[278,2034,2035],{"class":414},"(webhook_url, summary):\n",[278,2037,2039],{"class":280,"line":2038},116,[278,2040,2041],{"class":288},"    \"\"\"POST the JSON summary. Best effort: a failed webhook is logged, not fatal.\"\"\"\n",[278,2043,2045,2048,2050,2053,2056],{"class":280,"line":2044},117,[278,2046,2047],{"class":414},"    payload ",[278,2049,836],{"class":685},[278,2051,2052],{"class":414}," json.dumps(summary).encode(",[278,2054,2055],{"class":288},"\"utf-8\"",[278,2057,1128],{"class":414},[278,2059,2061,2064,2066],{"class":280,"line":2060},118,[278,2062,2063],{"class":414},"    request ",[278,2065,836],{"class":685},[278,2067,2068],{"class":414}," urllib.request.Request(\n",[278,2070,2072,2075,2078,2080,2083,2086,2088,2090,2093,2095,2098,2101,2104,2106],{"class":280,"line":2071},119,[278,2073,2074],{"class":414},"        webhook_url, ",[278,2076,2077],{"class":879},"data",[278,2079,836],{"class":685},[278,2081,2082],{"class":414},"payload, ",[278,2084,2085],{"class":879},"headers",[278,2087,836],{"class":685},[278,2089,932],{"class":414},[278,2091,2092],{"class":288},"\"Content-Type\"",[278,2094,423],{"class":414},[278,2096,2097],{"class":288},"\"application\u002Fjson\"",[278,2099,2100],{"class":414},"}, ",[278,2102,2103],{"class":879},"method",[278,2105,836],{"class":685},[278,2107,2108],{"class":288},"\"POST\"\n",[278,2110,2112],{"class":280,"line":2111},120,[278,2113,1222],{"class":414},[278,2115,2117,2119],{"class":280,"line":2116},121,[278,2118,862],{"class":685},[278,2120,865],{"class":414},[278,2122,2124,2127,2130,2132,2134,2137,2140,2143],{"class":280,"line":2123},122,[278,2125,2126],{"class":685},"        with",[278,2128,2129],{"class":414}," urllib.request.urlopen(request, ",[278,2131,898],{"class":879},[278,2133,836],{"class":685},[278,2135,2136],{"class":292},"10",[278,2138,2139],{"class":414},") ",[278,2141,2142],{"class":685},"as",[278,2144,2145],{"class":414}," response:\n",[278,2147,2149],{"class":280,"line":2148},123,[278,2150,2151],{"class":414},"            response.read()\n",[278,2153,2155,2157,2160,2163,2166],{"class":280,"line":2154},124,[278,2156,909],{"class":685},[278,2158,2159],{"class":292}," Exception",[278,2161,2162],{"class":685}," as",[278,2164,2165],{"class":414}," exc:  ",[278,2167,2168],{"class":573},"# noqa: BLE001 - alerting must never crash the check\n",[278,2170,2172,2175,2178,2180,2183,2185,2188,2190,2192,2194,2197,2199],{"class":280,"line":2171},125,[278,2173,2174],{"class":292},"        print",[278,2176,2177],{"class":414},"(",[278,2179,926],{"class":685},[278,2181,2182],{"class":288},"\"warning: webhook POST failed: ",[278,2184,932],{"class":292},[278,2186,2187],{"class":414},"exc",[278,2189,937],{"class":292},[278,2191,1056],{"class":288},[278,2193,784],{"class":414},[278,2195,2196],{"class":879},"file",[278,2198,836],{"class":685},[278,2200,2201],{"class":414},"sys.stderr)\n",[278,2203,2205],{"class":280,"line":2204},126,[278,2206,590],{"emptyLinePlaceholder":589},[278,2208,2210],{"class":280,"line":2209},127,[278,2211,590],{"emptyLinePlaceholder":589},[278,2213,2215,2217,2220],{"class":280,"line":2214},128,[278,2216,815],{"class":685},[278,2218,2219],{"class":284}," main",[278,2221,2222],{"class":414},"():\n",[278,2224,2226,2229,2231,2234,2237,2239,2242],{"class":280,"line":2225},129,[278,2227,2228],{"class":414},"    parser ",[278,2230,836],{"class":685},[278,2232,2233],{"class":414}," argparse.ArgumentParser(",[278,2235,2236],{"class":879},"description",[278,2238,836],{"class":685},[278,2240,2241],{"class":288},"\"Health-check a Talos Linux Kubernetes cluster.\"",[278,2243,1128],{"class":414},[278,2245,2247,2250,2252,2254,2257,2259,2261,2263,2266,2268,2271],{"class":280,"line":2246},130,[278,2248,2249],{"class":414},"    parser.add_argument(",[278,2251,847],{"class":288},[278,2253,784],{"class":414},[278,2255,2256],{"class":879},"required",[278,2258,836],{"class":685},[278,2260,885],{"class":292},[278,2262,784],{"class":414},[278,2264,2265],{"class":879},"help",[278,2267,836],{"class":685},[278,2269,2270],{"class":288},"\"Path to talosconfig\"",[278,2272,1128],{"class":414},[278,2274,2276,2278,2281,2283,2285,2287,2289,2291,2293,2295,2298],{"class":280,"line":2275},131,[278,2277,2249],{"class":414},[278,2279,2280],{"class":288},"\"--kubeconfig\"",[278,2282,784],{"class":414},[278,2284,2256],{"class":879},[278,2286,836],{"class":685},[278,2288,885],{"class":292},[278,2290,784],{"class":414},[278,2292,2265],{"class":879},[278,2294,836],{"class":685},[278,2296,2297],{"class":288},"\"Path to kubeconfig\"",[278,2299,1128],{"class":414},[278,2301,2303,2305,2308,2310,2313,2315,2318,2320,2322,2324,2326,2328,2330,2332,2335],{"class":280,"line":2302},132,[278,2304,2249],{"class":414},[278,2306,2307],{"class":288},"\"--control-plane-nodes\"",[278,2309,784],{"class":414},[278,2311,2312],{"class":879},"nargs",[278,2314,836],{"class":685},[278,2316,2317],{"class":288},"\"+\"",[278,2319,784],{"class":414},[278,2321,2256],{"class":879},[278,2323,836],{"class":685},[278,2325,885],{"class":292},[278,2327,784],{"class":414},[278,2329,2265],{"class":879},[278,2331,836],{"class":685},[278,2333,2334],{"class":288},"\"Control plane node IPs\"",[278,2336,1128],{"class":414},[278,2338,2340,2342,2344,2346,2348,2350,2353,2355,2358,2360,2363,2365,2367,2370],{"class":280,"line":2339},133,[278,2341,2249],{"class":414},[278,2343,1097],{"class":288},[278,2345,784],{"class":414},[278,2347,2312],{"class":879},[278,2349,836],{"class":685},[278,2351,2352],{"class":288},"\"*\"",[278,2354,784],{"class":414},[278,2356,2357],{"class":879},"default",[278,2359,836],{"class":685},[278,2361,2362],{"class":414},"[], ",[278,2364,2265],{"class":879},[278,2366,836],{"class":685},[278,2368,2369],{"class":288},"\"Worker node IPs (omit if none)\"",[278,2371,1128],{"class":414},[278,2373,2375,2377,2380,2382,2385,2387,2390,2392,2394,2396,2399,2401,2403,2405,2408],{"class":280,"line":2374},134,[278,2376,2249],{"class":414},[278,2378,2379],{"class":288},"\"--restart-threshold\"",[278,2381,784],{"class":414},[278,2383,2384],{"class":879},"type",[278,2386,836],{"class":685},[278,2388,2389],{"class":292},"int",[278,2391,784],{"class":414},[278,2393,2357],{"class":879},[278,2395,836],{"class":685},[278,2397,2398],{"class":292},"5",[278,2400,784],{"class":414},[278,2402,2265],{"class":879},[278,2404,836],{"class":685},[278,2406,2407],{"class":288},"\"Restart count treated as unhealthy (default 5)\"",[278,2409,1128],{"class":414},[278,2411,2413,2415,2418,2420,2422,2424,2426,2428,2430,2432,2435,2437,2439,2441,2444],{"class":280,"line":2412},135,[278,2414,2249],{"class":414},[278,2416,2417],{"class":288},"\"--timeout\"",[278,2419,784],{"class":414},[278,2421,2384],{"class":879},[278,2423,836],{"class":685},[278,2425,2389],{"class":292},[278,2427,784],{"class":414},[278,2429,2357],{"class":879},[278,2431,836],{"class":685},[278,2433,2434],{"class":292},"120",[278,2436,784],{"class":414},[278,2438,2265],{"class":879},[278,2440,836],{"class":685},[278,2442,2443],{"class":288},"\"talosctl health --wait-timeout in seconds (default 120)\"",[278,2445,1128],{"class":414},[278,2447,2449,2451,2454,2456,2459,2461,2464,2466,2468,2470,2472,2474,2476,2479],{"class":280,"line":2448},136,[278,2450,2249],{"class":414},[278,2452,2453],{"class":288},"\"--ignore-namespace\"",[278,2455,784],{"class":414},[278,2457,2458],{"class":879},"action",[278,2460,836],{"class":685},[278,2462,2463],{"class":288},"\"append\"",[278,2465,784],{"class":414},[278,2467,2357],{"class":879},[278,2469,836],{"class":685},[278,2471,2362],{"class":414},[278,2473,2265],{"class":879},[278,2475,836],{"class":685},[278,2477,2478],{"class":288},"\"Namespace to skip in the pod check (repeatable)\"",[278,2480,1128],{"class":414},[278,2482,2484,2486,2489,2491,2493,2495,2498],{"class":280,"line":2483},137,[278,2485,2249],{"class":414},[278,2487,2488],{"class":288},"\"--webhook-url\"",[278,2490,784],{"class":414},[278,2492,2265],{"class":879},[278,2494,836],{"class":685},[278,2496,2497],{"class":288},"\"Webhook to POST the JSON summary to on failure\"",[278,2499,1128],{"class":414},[278,2501,2503,2505,2508,2510,2512,2514,2517,2519,2521,2523,2526],{"class":280,"line":2502},138,[278,2504,2249],{"class":414},[278,2506,2507],{"class":288},"\"--lock-file\"",[278,2509,784],{"class":414},[278,2511,2357],{"class":879},[278,2513,836],{"class":685},[278,2515,2516],{"class":288},"\"\u002Ftmp\u002Ftalos-healthcheck.lock\"",[278,2518,784],{"class":414},[278,2520,2265],{"class":879},[278,2522,836],{"class":685},[278,2524,2525],{"class":288},"\"Lock file that prevents overlapping runs\"",[278,2527,1128],{"class":414},[278,2529,2531,2533,2535],{"class":280,"line":2530},139,[278,2532,1001],{"class":414},[278,2534,836],{"class":685},[278,2536,2537],{"class":414}," parser.parse_args()\n",[278,2539,2541],{"class":280,"line":2540},140,[278,2542,590],{"emptyLinePlaceholder":589},[278,2544,2546,2549,2551,2554,2557,2560],{"class":280,"line":2545},141,[278,2547,2548],{"class":414},"    lock ",[278,2550,836],{"class":685},[278,2552,2553],{"class":292}," open",[278,2555,2556],{"class":414},"(args.lock_file, ",[278,2558,2559],{"class":288},"\"w\"",[278,2561,1128],{"class":414},[278,2563,2565,2567],{"class":280,"line":2564},142,[278,2566,862],{"class":685},[278,2568,865],{"class":414},[278,2570,2572,2575,2578,2581,2584,2587],{"class":280,"line":2571},143,[278,2573,2574],{"class":414},"        fcntl.flock(lock, fcntl.",[278,2576,2577],{"class":292},"LOCK_EX",[278,2579,2580],{"class":685}," |",[278,2582,2583],{"class":414}," fcntl.",[278,2585,2586],{"class":292},"LOCK_NB",[278,2588,1128],{"class":414},[278,2590,2592,2594,2597],{"class":280,"line":2591},144,[278,2593,909],{"class":685},[278,2595,2596],{"class":292}," BlockingIOError",[278,2598,865],{"class":414},[278,2600,2602,2604,2606,2609,2611,2613,2615],{"class":280,"line":2601},145,[278,2603,2174],{"class":292},[278,2605,2177],{"class":414},[278,2607,2608],{"class":288},"\"another health check is still running; skipping this run\"",[278,2610,784],{"class":414},[278,2612,2196],{"class":879},[278,2614,836],{"class":685},[278,2616,2201],{"class":414},[278,2618,2620,2623,2626],{"class":280,"line":2619},146,[278,2621,2622],{"class":414},"        sys.exit(",[278,2624,2625],{"class":292},"2",[278,2627,1128],{"class":414},[278,2629,2631],{"class":280,"line":2630},147,[278,2632,590],{"emptyLinePlaceholder":589},[278,2634,2636,2638,2640],{"class":280,"line":2635},148,[278,2637,1582],{"class":414},[278,2639,836],{"class":685},[278,2641,1404],{"class":414},[278,2643,2645,2647],{"class":280,"line":2644},149,[278,2646,862],{"class":685},[278,2648,865],{"class":414},[278,2650,2652,2655,2657],{"class":280,"line":2651},150,[278,2653,2654],{"class":414},"        talos_ok, talos_output ",[278,2656,836],{"class":685},[278,2658,2659],{"class":414}," check_talos_health(\n",[278,2661,2663],{"class":280,"line":2662},151,[278,2664,2665],{"class":414},"            args.talosconfig, args.control_plane_nodes, args.worker_nodes, args.timeout\n",[278,2667,2669],{"class":280,"line":2668},152,[278,2670,2671],{"class":414},"        )\n",[278,2673,2675,2677,2679],{"class":280,"line":2674},153,[278,2676,1452],{"class":685},[278,2678,1315],{"class":685},[278,2680,2681],{"class":414}," talos_ok:\n",[278,2683,2685,2688,2691,2693,2696,2698,2701,2704,2707,2710],{"class":280,"line":2684},154,[278,2686,2687],{"class":414},"            problems.append({",[278,2689,2690],{"class":288},"\"check\"",[278,2692,423],{"class":414},[278,2694,2695],{"class":288},"\"talosctl health\"",[278,2697,784],{"class":414},[278,2699,2700],{"class":288},"\"detail\"",[278,2702,2703],{"class":414},": talos_output.splitlines()[",[278,2705,2706],{"class":685},"-",[278,2708,2709],{"class":292},"20",[278,2711,2712],{"class":414},":]})\n",[278,2714,2716],{"class":280,"line":2715},155,[278,2717,590],{"emptyLinePlaceholder":589},[278,2719,2721,2724,2726],{"class":280,"line":2720},156,[278,2722,2723],{"class":414},"        alarms ",[278,2725,836],{"class":685},[278,2727,2728],{"class":414}," check_etcd_alarms(args.talosconfig, args.control_plane_nodes)\n",[278,2730,2732,2734],{"class":280,"line":2731},157,[278,2733,1452],{"class":685},[278,2735,2736],{"class":414}," alarms:\n",[278,2738,2740,2742,2744,2746,2749,2751,2753],{"class":280,"line":2739},158,[278,2741,2687],{"class":414},[278,2743,2690],{"class":288},[278,2745,423],{"class":414},[278,2747,2748],{"class":288},"\"etcd alarms\"",[278,2750,784],{"class":414},[278,2752,2700],{"class":288},[278,2754,2755],{"class":414},": alarms})\n",[278,2757,2759,2761,2764],{"class":280,"line":2758},159,[278,2760,909],{"class":685},[278,2762,2763],{"class":292}," FileNotFoundError",[278,2765,865],{"class":414},[278,2767,2769,2771,2773,2776,2778,2780,2782],{"class":280,"line":2768},160,[278,2770,2174],{"class":292},[278,2772,2177],{"class":414},[278,2774,2775],{"class":288},"\"talosctl not found on PATH\"",[278,2777,784],{"class":414},[278,2779,2196],{"class":879},[278,2781,836],{"class":685},[278,2783,2201],{"class":414},[278,2785,2787,2789,2791],{"class":280,"line":2786},161,[278,2788,2622],{"class":414},[278,2790,2625],{"class":292},[278,2792,1128],{"class":414},[278,2794,2796],{"class":280,"line":2795},162,[278,2797,590],{"emptyLinePlaceholder":589},[278,2799,2801,2803],{"class":280,"line":2800},163,[278,2802,862],{"class":685},[278,2804,865],{"class":414},[278,2806,2808,2811,2814,2816],{"class":280,"line":2807},164,[278,2809,2810],{"class":414},"        config.load_kube_config(",[278,2812,2813],{"class":879},"config_file",[278,2815,836],{"class":685},[278,2817,2818],{"class":414},"args.kubeconfig)\n",[278,2820,2822,2825,2827],{"class":280,"line":2821},165,[278,2823,2824],{"class":414},"        v1 ",[278,2826,836],{"class":685},[278,2828,2829],{"class":414}," client.CoreV1Api()\n",[278,2831,2833,2836,2838],{"class":280,"line":2832},166,[278,2834,2835],{"class":414},"        not_ready, cordoned ",[278,2837,836],{"class":685},[278,2839,2840],{"class":414}," check_nodes(v1)\n",[278,2842,2844,2846],{"class":280,"line":2843},167,[278,2845,1452],{"class":685},[278,2847,2848],{"class":414}," not_ready:\n",[278,2850,2852,2854,2856,2858,2861,2863,2865],{"class":280,"line":2851},168,[278,2853,2687],{"class":414},[278,2855,2690],{"class":288},[278,2857,423],{"class":414},[278,2859,2860],{"class":288},"\"node readiness\"",[278,2862,784],{"class":414},[278,2864,2700],{"class":288},[278,2866,2867],{"class":414},": not_ready})\n",[278,2869,2871,2873],{"class":280,"line":2870},169,[278,2872,1452],{"class":685},[278,2874,2875],{"class":414}," cordoned:\n",[278,2877,2879,2881,2883,2885,2887,2889,2891],{"class":280,"line":2878},170,[278,2880,2687],{"class":414},[278,2882,2690],{"class":288},[278,2884,423],{"class":414},[278,2886,465],{"class":288},[278,2888,784],{"class":414},[278,2890,2700],{"class":288},[278,2892,2893],{"class":414},": cordoned})\n",[278,2895,2897],{"class":280,"line":2896},171,[278,2898,590],{"emptyLinePlaceholder":589},[278,2900,2902,2905,2907,2910,2913],{"class":280,"line":2901},172,[278,2903,2904],{"class":414},"        pod_problems ",[278,2906,836],{"class":685},[278,2908,2909],{"class":414}," check_pods(v1, args.restart_threshold, ",[278,2911,2912],{"class":292},"set",[278,2914,2915],{"class":414},"(args.ignore_namespace))\n",[278,2917,2919,2921],{"class":280,"line":2918},173,[278,2920,1452],{"class":685},[278,2922,2923],{"class":414}," pod_problems:\n",[278,2925,2927,2929,2931,2933,2935,2937,2939],{"class":280,"line":2926},174,[278,2928,2687],{"class":414},[278,2930,2690],{"class":288},[278,2932,423],{"class":414},[278,2934,508],{"class":288},[278,2936,784],{"class":414},[278,2938,2700],{"class":288},[278,2940,2941],{"class":414},": pod_problems})\n",[278,2943,2945,2947,2949,2951,2953],{"class":280,"line":2944},175,[278,2946,909],{"class":685},[278,2948,2159],{"class":292},[278,2950,2162],{"class":685},[278,2952,2165],{"class":414},[278,2954,2955],{"class":573},"# noqa: BLE001 - an unreachable API server is itself a finding\n",[278,2957,2959,2962,2964,2966,2969,2971,2973,2976,2978,2980,2983,2986,2989,2991,2993,2995,2997,2999],{"class":280,"line":2958},176,[278,2960,2961],{"class":414},"        problems.append({",[278,2963,2690],{"class":288},[278,2965,423],{"class":414},[278,2967,2968],{"class":288},"\"kubernetes api\"",[278,2970,784],{"class":414},[278,2972,2700],{"class":288},[278,2974,2975],{"class":414},": [",[278,2977,926],{"class":685},[278,2979,1056],{"class":288},[278,2981,2982],{"class":292},"{type",[278,2984,2985],{"class":414},"(exc).",[278,2987,2988],{"class":292},"__name__}",[278,2990,423],{"class":288},[278,2992,932],{"class":292},[278,2994,2187],{"class":414},[278,2996,937],{"class":292},[278,2998,1056],{"class":288},[278,3000,3001],{"class":414},"]})\n",[278,3003,3005],{"class":280,"line":3004},177,[278,3006,590],{"emptyLinePlaceholder":589},[278,3008,3010,3013,3015],{"class":280,"line":3009},178,[278,3011,3012],{"class":414},"    summary ",[278,3014,836],{"class":685},[278,3016,3017],{"class":414}," {\n",[278,3019,3021,3024,3027,3030,3032,3035],{"class":280,"line":3020},179,[278,3022,3023],{"class":288},"        \"checkedAt\"",[278,3025,3026],{"class":414},": datetime.now(timezone.utc).isoformat(",[278,3028,3029],{"class":879},"timespec",[278,3031,836],{"class":685},[278,3033,3034],{"class":288},"\"seconds\"",[278,3036,3037],{"class":414},"),\n",[278,3039,3041,3044,3046,3049],{"class":280,"line":3040},180,[278,3042,3043],{"class":288},"        \"healthy\"",[278,3045,423],{"class":414},[278,3047,3048],{"class":685},"not",[278,3050,3051],{"class":414}," problems,\n",[278,3053,3055,3058],{"class":280,"line":3054},181,[278,3056,3057],{"class":288},"        \"problems\"",[278,3059,3060],{"class":414},": problems,\n",[278,3062,3064],{"class":280,"line":3063},182,[278,3065,534],{"class":414},[278,3067,3069,3072,3075,3078,3080,3082],{"class":280,"line":3068},183,[278,3070,3071],{"class":292},"    print",[278,3073,3074],{"class":414},"(json.dumps(summary, ",[278,3076,3077],{"class":879},"indent",[278,3079,836],{"class":685},[278,3081,2625],{"class":292},[278,3083,3084],{"class":414},"))\n",[278,3086,3088],{"class":280,"line":3087},184,[278,3089,590],{"emptyLinePlaceholder":589},[278,3091,3093,3095],{"class":280,"line":3092},185,[278,3094,1080],{"class":685},[278,3096,3097],{"class":414}," problems:\n",[278,3099,3101,3103],{"class":280,"line":3100},186,[278,3102,1452],{"class":685},[278,3104,3105],{"class":414}," args.webhook_url:\n",[278,3107,3109],{"class":280,"line":3108},187,[278,3110,3111],{"class":414},"            send_webhook(args.webhook_url, summary)\n",[278,3113,3115,3117,3120],{"class":280,"line":3114},188,[278,3116,2622],{"class":414},[278,3118,3119],{"class":292},"1",[278,3121,1128],{"class":414},[278,3123,3125,3128,3130],{"class":280,"line":3124},189,[278,3126,3127],{"class":414},"    sys.exit(",[278,3129,1026],{"class":292},[278,3131,1128],{"class":414},[278,3133,3135],{"class":280,"line":3134},190,[278,3136,590],{"emptyLinePlaceholder":589},[278,3138,3140],{"class":280,"line":3139},191,[278,3141,590],{"emptyLinePlaceholder":589},[278,3143,3145,3147,3150,3153,3156],{"class":280,"line":3144},192,[278,3146,1291],{"class":685},[278,3148,3149],{"class":292}," __name__",[278,3151,3152],{"class":685}," ==",[278,3154,3155],{"class":288}," \"__main__\"",[278,3157,865],{"class":414},[278,3159,3161],{"class":280,"line":3160},193,[278,3162,3163],{"class":414},"    main()\n",[47,3165,3167],{"id":3166},"notes","Notes",[52,3169,3170,3195,3207,3231,3259,3269,3275],{},[55,3171,3172,3178,3179,3181,3182,3185,3186,3188,3189,3191,3192,3194],{},[3173,3174,3175,3176,61],"strong",{},"One node for ",[14,3177,16],{}," ",[14,3180,16],{}," is built to talk to a single node, which then checks the rest of the cluster server-side; passing several addresses to ",[14,3183,3184],{},"--nodes"," makes it refuse to run. That is why the script sends the request through the first control plane node and passes the full membership with ",[14,3187,171],{}," and ",[14,3190,186],{},". An earlier version of this script passed every node to ",[14,3193,3184],{},", which doesn't work.",[55,3196,3197,3178,3200,3202,3203,3206],{},[3173,3198,3199],{},"Keep the timeout under the cron interval.",[14,3201,16],{}," blocks until its checks pass or ",[14,3204,3205],{},"--wait-timeout"," expires, and the default is 20 minutes. The lock file is a second line of defence: if a run is still going, the next one exits with code 2 instead of piling up.",[55,3208,3209,3212,3213,3215,3216,3219,3220,3223,3224,3227,3228,61],{},[3173,3210,3211],{},"etcd alarms matter more than they look."," Talos' etcd maintenance guide notes that etcd's default space quota is 2 GiB and that etcd stops operations when the database exceeds it; the condition shows up as a ",[14,3214,20],{}," row in ",[14,3217,3218],{},"talosctl etcd alarm list",". The fix is defragmentation (",[14,3221,3222],{},"talosctl -n \u003CIP> etcd defrag",", one node at a time) or a larger ",[14,3225,3226],{},"quota-backend-bytes",", followed by ",[14,3229,3230],{},"talosctl etcd alarm disarm",[55,3232,3233,3236,3237,784,3240,3243,3244,3247,3248,3251,3252,3255,3256,3258],{},[3173,3234,3235],{},"Restart counts are a blunt instrument on purpose."," The kubelet resets a container's back-off timer after 10 minutes of clean running, but the restart count stays with the pod, so a pod that crashed six times last month and has been fine since keeps tripping the threshold until it is replaced. The back-off check (",[14,3238,3239],{},"CrashLoopBackOff",[14,3241,3242],{},"ImagePullBackOff"," and friends, read from ",[14,3245,3246],{},"state.waiting.reason"," for init containers as well as app containers, since a failing init container leaves the pod ",[14,3249,3250],{},"Pending"," rather than ",[14,3253,3254],{},"Failed",") is the precise signal; the restart threshold catches the slow leak without needing a metrics backend. Raise the threshold or add ",[14,3257,227],{}," rather than deleting the check.",[55,3260,3261,3264,3265,3268],{},[3173,3262,3263],{},"The Kubernetes API being unreachable is a finding, not a crash."," Any exception from the client (a TLS error, a timeout, a 403 from an RBAC mistake) is reported as a ",[14,3266,3267],{},"kubernetes api"," problem so the webhook still fires.",[55,3270,3271,3274],{},[3173,3272,3273],{},"Read-only by design."," The script doesn't restart, uncordon or delete anything. A health check that also remediates needs far more thought about blast radius than a cron job should be trusted with unsupervised.",[55,3276,3277,3280,3281,3283],{},[3173,3278,3279],{},"Don't point the talosconfig at the VIP."," If the cluster uses a Talos Layer 2 VIP, Sidero Labs warns against using it as a ",[14,3282,24],{}," endpoint, because the VIP depends on etcd and the API server; list the node IPs instead.",[47,3285,3287],{"id":3286},"source","Source",[52,3289,3290,3308,3322,3330,3338,3354],{},[55,3291,3292,1515,3298,784,3301,784,3304,3307],{},[38,3293,3297],{"href":3294,"rel":3295},"https:\u002F\u002Fdocs.siderolabs.com\u002Ftalos\u002Fv1.14\u002Freference\u002Fcli",[3296],"nofollow","talosctl CLI reference",[14,3299,3300],{},"health",[14,3302,3303],{},"etcd alarm list",[14,3305,3306],{},"etcd status",")",[55,3309,3310,1515,3315,784,3317,784,3319,3307],{},[38,3311,3314],{"href":3312,"rel":3313},"https:\u002F\u002Fdocs.siderolabs.com\u002Ftalos\u002Fv1.14\u002Fsecurity\u002Frbac",[3296],"Role-based access control (RBAC)",[14,3316,87],{},[14,3318,83],{},[14,3320,3321],{},"talosctl config new --roles",[55,3323,3324,3329],{},[38,3325,3328],{"href":3326,"rel":3327},"https:\u002F\u002Fdocs.siderolabs.com\u002Ftalos\u002Fv1.14\u002Fbuild-and-extend-talos\u002Fcluster-operations-and-maintenance\u002Fetcd-maintenance",[3296],"etcd Maintenance"," (space quota, alarms, defragmentation)",[55,3331,3332,3337],{},[38,3333,3336],{"href":3334,"rel":3335},"https:\u002F\u002Fdocs.siderolabs.com\u002Ftalos\u002Fv1.14\u002Fnetworking\u002Fadvanced\u002Fvip",[3296],"Virtual (shared) IP"," (why the VIP is not a talosconfig endpoint)",[55,3339,3340,1515,3345,784,3348,784,3351,3307],{},[38,3341,3344],{"href":3342,"rel":3343},"https:\u002F\u002Fgithub.com\u002Fkubernetes-client\u002Fpython",[3296],"Kubernetes Python client",[14,3346,3347],{},"CoreV1Api.list_node",[14,3349,3350],{},"CoreV1Api.list_pod_for_all_namespaces",[14,3352,3353],{},"V1ContainerStatus",[55,3355,3356],{},[38,3357,3360],{"href":3358,"rel":3359},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F",[3296],"Pod Lifecycle: container states and restart policy",[3362,3363,3364],"style",{},"html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}",{"title":274,"searchDepth":296,"depth":296,"links":3366},[3367,3368,3369,3370,3371,3372],{"id":49,"depth":296,"text":50},{"id":115,"depth":296,"text":116},{"id":263,"depth":296,"text":264},{"id":560,"depth":296,"text":561},{"id":3166,"depth":296,"text":3167},{"id":3286,"depth":296,"text":3287},"techcolumnist",[3375,3376],"scripts","engineering","2026-02-04T14:00:00Z","A cron-friendly Python health check for Talos clusters: talosctl health, etcd alarms, node readiness, cordons and crash-looping pods, with a webhook alert.","md",false,null,{},"\u002Fblog\u002F2026\u002F02\u002F04\u002Fpython-talos-health-checking-a-kubernetes-cluster-from-a-cron-job",{"title":6,"description":3378},[3373],"blog\u002F2026\u002F02\u002F04\u002Fpython-talos-health-checking-a-kubernetes-cluster-from-a-cron-job",[566,3388,35],"talos","2026-09-15T05:12:15Z","\u002F2026\u002F02\u002F04\u002Fpython-talos-health-checking-a-kubernetes-cluster-from-a-cron-job\u002F","gSoM_bZ2wfk8zJAj1y9ZVMDBiKy7knD1oujFCLENQPY",{"title":3393,"description":3394,"date":3395,"url":3396,"categories":3397,"tags":3398,"image":3381,"readingTime":307,"canonical":3373,"sites":3403,"series":3381,"seriesOrder":3381},"Node.js: Discovery – Scanning a Subnet for Open RDP and SSH","A dependency-free Node.js TCP connect scanner that sweeps a subnet for RDP and SSH, separates closed from filtered ports, grabs SSH banners and writes CSV.","2026-02-11T14:00:00Z","\u002F2026\u002F02\u002F11\u002Fnode-js-discovery-scanning-a-subnet-for-open-rdp-and-ssh\u002F",[3375,3376],[3399,3400,3401,3402],"nodejs","discovery","network","ssh",[3373],{"title":3405,"description":3406,"date":3407,"url":3408,"categories":3409,"tags":3410,"image":3381,"readingTime":307,"canonical":3373,"sites":3412,"series":3381,"seriesOrder":3381},"Python: Discovery – Crawling DNS Zones for Orphaned Records","A dnspython script that transfers a zone with AXFR, probes every A\u002FAAAA record for signs of life, flags dangling CNAMEs, and reports orphan cleanup candidates.","2026-01-28T14:00:00Z","\u002F2026\u002F01\u002F28\u002Fpython-discovery-crawling-dns-zones-for-orphaned-records\u002F",[3375,3376],[566,3400,3401,3411],"windows",[3373],[3414,3424,3430],{"title":3415,"description":3416,"date":3417,"url":3418,"categories":3419,"tags":3421,"image":3381,"readingTime":335,"canonical":3373,"sites":3423,"series":3381,"seriesOrder":3381},"Hyper-V to Talos: Rethinking the Virtualization Stack","Why moving workloads off Hyper-V VMs onto a Talos Kubernetes cluster changed the whole stack, with an inventory script and a VM-to-manifest example.","2026-09-16T14:00:00Z","\u002F2026\u002F09\u002F16\u002Fhyper-v-to-talos-rethinking-the-virtualization-stack\u002F",[3376,3420],"strategy",[3422,3388,35],"hyper-v",[3373],{"title":41,"description":3425,"date":3426,"url":40,"categories":3427,"tags":3428,"image":3381,"readingTime":470,"canonical":3373,"sites":3429,"series":3381,"seriesOrder":3381},"Rolling Talos and Kubernetes upgrades through a cluster one node at a time: preflight checks, etcd snapshots, Image Factory installers, and scripts.","2026-08-26T14:00:00Z",[3376],[3388,35],[3373],{"title":3431,"description":3432,"date":3433,"url":3434,"categories":3435,"tags":3436,"image":3381,"readingTime":470,"canonical":3373,"sites":3437,"series":3381,"seriesOrder":3381},"Kubernetes: Running Talos Alongside a Legacy Hyper-V Estate","Introducing a Talos Kubernetes cluster into an existing Hyper-V estate: VM build script, Image Factory schematic, VLAN, VIP, firewall and storage boundaries.","2026-07-08T14:00:00Z","\u002F2026\u002F07\u002F08\u002Fkubernetes-running-talos-alongside-a-legacy-hyper-v-estate\u002F",[3376],[3388,35,3422],[3373],{"doc":3381,"posts":3439},[],1790052513419]