[{"data":1,"prerenderedAt":3430},["ShallowReactive",2],{"post:\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run\u002F":3},{"post":4,"newer":3382,"older":3393,"related":3402,"series":3428},{"id":5,"title":6,"body":7,"canonical":3362,"categories":3363,"date":3366,"description":3367,"extension":3368,"featured":3369,"hero":3370,"image":3370,"meta":3371,"navigation":615,"path":3372,"readingTime":139,"seo":3373,"series":3370,"seriesOrder":3370,"sites":3374,"source":3370,"stem":3375,"tags":3376,"updated":3370,"url":3380,"__hash__":3381},"blog\u002Fblog\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run.md","Python: Data Warehouse – Catching Schema Drift Before an ETL Run",{"type":8,"value":9,"toc":3355},"minimark",[10,23,50,55,86,90,97,157,160,480,483,564,567,575,585,589,3123,3127,3316,3320,3351],[11,12,13,14,18,19,22],"p",{},"The ETL failures that cost the most time are never the ones that crash loudly. They are the ones where a source table quietly loses a column, or a ",[15,16,17],"code",{},"varchar(50)"," becomes ",[15,20,21],{},"varchar(255)",", and the load \"succeeds\" while writing garbage, truncated values, or nulls downstream. I run this script as a pre-flight check before every scheduled ETL job: snapshot the expected schema once, commit it, then diff the live schema against it on every run and fail fast if something moved.",[11,24,25,26,30,31,35,36,39,40,45,46,49],{},"This version compares more than column names and types. It also compares length, numeric and timestamp precision, interval fields, nullability, the schema-qualified underlying type for arrays and user-defined types, and the domain a column uses. It sorts findings into ",[27,28,29],"strong",{},"breaking"," changes (a column removed, a type changed, a length or precision ",[32,33,34],"em",{},"reduced",", a column becoming nullable) and ",[27,37,38],{},"additive"," ones (a new column, a length increased). A pipeline that selects an explicit column list, like the ",[41,42,44],"a",{"href":43},"\u002F2025\u002F12\u002F17\u002Fpython-data-warehouse-incremental-loads-from-postgres-to-bigquery\u002F","incremental Postgres-to-BigQuery load",", usually survives additive changes, so ",[15,47,48],{},"--allow-additive"," lets those pass with a warning.",[51,52,54],"h2",{"id":53},"requirements","Requirements",[56,57,58,62,72,79],"ul",{},[59,60,61],"li",{},"Python 3.10 or later.",[59,63,64,67,68,71],{},[15,65,66],{},"psycopg2-binary"," 2.9.1 or later (",[15,69,70],{},"pip install psycopg2-binary","), the first release with Python 3.10 wheels.",[59,73,74,75,78],{},"A role that can see the tables. ",[15,76,77],{},"information_schema.columns"," only shows columns the current user has access to, as owner or through some privilege. A table the ETL role can't read looks exactly like a table that was dropped, which is what you want the check to catch anyway.",[59,80,81,82,85],{},"A baseline snapshot generated once with ",[15,83,84],{},"--snapshot",", committed to version control next to the pipeline that depends on it.",[51,87,89],{"id":88},"usage","Usage",[11,91,92,93,96],{},"Take the baseline the first time and commit the file. Table names can be schema-qualified; unqualified names default to ",[15,94,95],{},"public",".",[98,99,104],"pre",{"className":100,"code":101,"language":102,"meta":103,"style":103},"language-bash shiki shiki-themes github-dark","python check_schema_drift.py --snapshot \\\n    --dsn \"postgresql:\u002F\u002Fetl_reader:\u003Cpassword>@\u003Chost>:5432\u002Fapp\" \\\n    --tables public.orders,public.customers,billing.line_items \\\n    --output schema_baseline.json\n","bash","",[15,105,106,126,137,148],{"__ignoreMap":103},[107,108,111,115,119,123],"span",{"class":109,"line":110},"line",1,[107,112,114],{"class":113},"svObZ","python",[107,116,118],{"class":117},"sU2Wk"," check_schema_drift.py",[107,120,122],{"class":121},"sDLfK"," --snapshot",[107,124,125],{"class":121}," \\\n",[107,127,129,132,135],{"class":109,"line":128},2,[107,130,131],{"class":121},"    --dsn",[107,133,134],{"class":117}," \"postgresql:\u002F\u002Fetl_reader:\u003Cpassword>@\u003Chost>:5432\u002Fapp\"",[107,136,125],{"class":121},[107,138,140,143,146],{"class":109,"line":139},3,[107,141,142],{"class":121},"    --tables",[107,144,145],{"class":117}," public.orders,public.customers,billing.line_items",[107,147,125],{"class":121},[107,149,151,154],{"class":109,"line":150},4,[107,152,153],{"class":121},"    --output",[107,155,156],{"class":117}," schema_baseline.json\n",[11,158,159],{},"The baseline records every attribute that is compared:",[98,161,165],{"className":162,"code":163,"language":164,"meta":103,"style":103},"language-json shiki shiki-themes github-dark","{\n  \"public.orders\": {\n    \"discount_code\": {\n      \"character_maximum_length\": 32,\n      \"data_type\": \"character varying\",\n      \"datetime_precision\": null,\n      \"domain_name\": null,\n      \"domain_schema\": null,\n      \"interval_type\": null,\n      \"is_nullable\": \"YES\",\n      \"numeric_precision\": null,\n      \"numeric_scale\": null,\n      \"udt_name\": \"varchar\",\n      \"udt_schema\": \"pg_catalog\"\n    },\n    \"total_cents\": {\n      \"character_maximum_length\": null,\n      \"data_type\": \"integer\",\n      \"datetime_precision\": null,\n      \"domain_name\": null,\n      \"domain_schema\": null,\n      \"interval_type\": null,\n      \"is_nullable\": \"NO\",\n      \"numeric_precision\": 32,\n      \"numeric_scale\": 0,\n      \"udt_name\": \"int4\",\n      \"udt_schema\": \"pg_catalog\"\n    }\n  }\n}\n","json",[15,166,167,173,181,188,202,215,228,240,252,264,277,289,301,314,325,331,339,350,362,373,384,395,406,418,429,441,453,462,468,474],{"__ignoreMap":103},[107,168,169],{"class":109,"line":110},[107,170,172],{"class":171},"s95oV","{\n",[107,174,175,178],{"class":109,"line":128},[107,176,177],{"class":121},"  \"public.orders\"",[107,179,180],{"class":171},": {\n",[107,182,183,186],{"class":109,"line":139},[107,184,185],{"class":121},"    \"discount_code\"",[107,187,180],{"class":171},[107,189,190,193,196,199],{"class":109,"line":150},[107,191,192],{"class":121},"      \"character_maximum_length\"",[107,194,195],{"class":171},": ",[107,197,198],{"class":121},"32",[107,200,201],{"class":171},",\n",[107,203,205,208,210,213],{"class":109,"line":204},5,[107,206,207],{"class":121},"      \"data_type\"",[107,209,195],{"class":171},[107,211,212],{"class":117},"\"character varying\"",[107,214,201],{"class":171},[107,216,218,221,223,226],{"class":109,"line":217},6,[107,219,220],{"class":121},"      \"datetime_precision\"",[107,222,195],{"class":171},[107,224,225],{"class":121},"null",[107,227,201],{"class":171},[107,229,231,234,236,238],{"class":109,"line":230},7,[107,232,233],{"class":121},"      \"domain_name\"",[107,235,195],{"class":171},[107,237,225],{"class":121},[107,239,201],{"class":171},[107,241,243,246,248,250],{"class":109,"line":242},8,[107,244,245],{"class":121},"      \"domain_schema\"",[107,247,195],{"class":171},[107,249,225],{"class":121},[107,251,201],{"class":171},[107,253,255,258,260,262],{"class":109,"line":254},9,[107,256,257],{"class":121},"      \"interval_type\"",[107,259,195],{"class":171},[107,261,225],{"class":121},[107,263,201],{"class":171},[107,265,267,270,272,275],{"class":109,"line":266},10,[107,268,269],{"class":121},"      \"is_nullable\"",[107,271,195],{"class":171},[107,273,274],{"class":117},"\"YES\"",[107,276,201],{"class":171},[107,278,280,283,285,287],{"class":109,"line":279},11,[107,281,282],{"class":121},"      \"numeric_precision\"",[107,284,195],{"class":171},[107,286,225],{"class":121},[107,288,201],{"class":171},[107,290,292,295,297,299],{"class":109,"line":291},12,[107,293,294],{"class":121},"      \"numeric_scale\"",[107,296,195],{"class":171},[107,298,225],{"class":121},[107,300,201],{"class":171},[107,302,304,307,309,312],{"class":109,"line":303},13,[107,305,306],{"class":121},"      \"udt_name\"",[107,308,195],{"class":171},[107,310,311],{"class":117},"\"varchar\"",[107,313,201],{"class":171},[107,315,317,320,322],{"class":109,"line":316},14,[107,318,319],{"class":121},"      \"udt_schema\"",[107,321,195],{"class":171},[107,323,324],{"class":117},"\"pg_catalog\"\n",[107,326,328],{"class":109,"line":327},15,[107,329,330],{"class":171},"    },\n",[107,332,334,337],{"class":109,"line":333},16,[107,335,336],{"class":121},"    \"total_cents\"",[107,338,180],{"class":171},[107,340,342,344,346,348],{"class":109,"line":341},17,[107,343,192],{"class":121},[107,345,195],{"class":171},[107,347,225],{"class":121},[107,349,201],{"class":171},[107,351,353,355,357,360],{"class":109,"line":352},18,[107,354,207],{"class":121},[107,356,195],{"class":171},[107,358,359],{"class":117},"\"integer\"",[107,361,201],{"class":171},[107,363,365,367,369,371],{"class":109,"line":364},19,[107,366,220],{"class":121},[107,368,195],{"class":171},[107,370,225],{"class":121},[107,372,201],{"class":171},[107,374,376,378,380,382],{"class":109,"line":375},20,[107,377,233],{"class":121},[107,379,195],{"class":171},[107,381,225],{"class":121},[107,383,201],{"class":171},[107,385,387,389,391,393],{"class":109,"line":386},21,[107,388,245],{"class":121},[107,390,195],{"class":171},[107,392,225],{"class":121},[107,394,201],{"class":171},[107,396,398,400,402,404],{"class":109,"line":397},22,[107,399,257],{"class":121},[107,401,195],{"class":171},[107,403,225],{"class":121},[107,405,201],{"class":171},[107,407,409,411,413,416],{"class":109,"line":408},23,[107,410,269],{"class":121},[107,412,195],{"class":171},[107,414,415],{"class":117},"\"NO\"",[107,417,201],{"class":171},[107,419,421,423,425,427],{"class":109,"line":420},24,[107,422,282],{"class":121},[107,424,195],{"class":171},[107,426,198],{"class":121},[107,428,201],{"class":171},[107,430,432,434,436,439],{"class":109,"line":431},25,[107,433,294],{"class":121},[107,435,195],{"class":171},[107,437,438],{"class":121},"0",[107,440,201],{"class":171},[107,442,444,446,448,451],{"class":109,"line":443},26,[107,445,306],{"class":121},[107,447,195],{"class":171},[107,449,450],{"class":117},"\"int4\"",[107,452,201],{"class":171},[107,454,456,458,460],{"class":109,"line":455},27,[107,457,319],{"class":121},[107,459,195],{"class":171},[107,461,324],{"class":117},[107,463,465],{"class":109,"line":464},28,[107,466,467],{"class":171},"    }\n",[107,469,471],{"class":109,"line":470},29,[107,472,473],{"class":171},"  }\n",[107,475,477],{"class":109,"line":476},30,[107,478,479],{"class":171},"}\n",[11,481,482],{},"Run the check as the first step of the job and chain the load after it, so a non-zero exit stops the pipeline before any extraction:",[98,484,486],{"className":100,"code":485,"language":102,"meta":103,"style":103},"python check_schema_drift.py \\\n    --dsn \"$SOURCE_DSN\" \\\n    --tables public.orders,public.customers,billing.line_items \\\n    --baseline schema_baseline.json \\\n    --allow-additive \\\n  && python postgres_to_bigquery_incremental.py --pg-dsn \"$SOURCE_DSN\" --source-table public.orders ...\n",[15,487,488,496,511,519,529,536],{"__ignoreMap":103},[107,489,490,492,494],{"class":109,"line":110},[107,491,114],{"class":113},[107,493,118],{"class":117},[107,495,125],{"class":121},[107,497,498,500,503,506,509],{"class":109,"line":128},[107,499,131],{"class":121},[107,501,502],{"class":117}," \"",[107,504,505],{"class":171},"$SOURCE_DSN",[107,507,508],{"class":117},"\"",[107,510,125],{"class":121},[107,512,513,515,517],{"class":109,"line":139},[107,514,142],{"class":121},[107,516,145],{"class":117},[107,518,125],{"class":121},[107,520,521,524,527],{"class":109,"line":150},[107,522,523],{"class":121},"    --baseline",[107,525,526],{"class":117}," schema_baseline.json",[107,528,125],{"class":121},[107,530,531,534],{"class":109,"line":204},[107,532,533],{"class":121},"    --allow-additive",[107,535,125],{"class":121},[107,537,538,541,543,546,549,551,553,555,558,561],{"class":109,"line":217},[107,539,540],{"class":171},"  && ",[107,542,114],{"class":113},[107,544,545],{"class":117}," postgres_to_bigquery_incremental.py",[107,547,548],{"class":121}," --pg-dsn",[107,550,502],{"class":117},[107,552,505],{"class":171},[107,554,508],{"class":117},[107,556,557],{"class":121}," --source-table",[107,559,560],{"class":117}," public.orders",[107,562,563],{"class":117}," ...\n",[11,565,566],{},"Sample output when drift is found:",[98,568,573],{"className":569,"code":571,"language":572,"meta":103},[570],"language-text","BREAKING billing.line_items: sku: character_maximum_length 64 -> 32\nADDITIVE public.customers: column added: loyalty_tier (character varying)\nBREAKING public.orders: column removed: discount_code\nBREAKING public.orders: total_cents: type int4 -> numeric\n3 breaking, 1 additive change(s). Exit 1.\n","text",[15,574,571],{"__ignoreMap":103},[11,576,577,578,580,581,584],{},"With only additive changes and ",[15,579,48],{},", it prints the same ",[15,582,583],{},"ADDITIVE"," lines and exits 0.",[51,586,588],{"id":587},"script","Script",[98,590,593],{"className":591,"code":592,"language":114,"meta":103,"style":103},"language-python shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"\ncheck_schema_drift.py\n\nCompares live PostgreSQL column definitions against a saved baseline and\nreports added, removed, retyped (including a change of domain or of the\nschema a user-defined type lives in), resized, precision-changed, or\nnullability-changed columns before an ETL job runs.\n\nUsage:\n    Baseline: python check_schema_drift.py --snapshot --dsn \u003Cdsn> --tables s.t1,t2 --output baseline.json\n    Check:    python check_schema_drift.py --dsn \u003Cdsn> --tables s.t1,t2 --baseline baseline.json [--allow-additive]\n\nReads:  information_schema.columns for each table through the given DSN.\nWrites: a JSON snapshot in --snapshot mode; nothing in check mode.\nExit:   0 no drift (or only additive drift with --allow-additive),\n        1 drift found, 2 usage or connection error.\n\"\"\"\n\nimport argparse\nimport json\nimport sys\n\nimport psycopg2\n\n# The first five together identify the column's type; the rest are compared one by one.\nTYPE_ATTRIBUTES = (\"data_type\", \"udt_schema\", \"udt_name\", \"domain_schema\", \"domain_name\")\nDETAIL_ATTRIBUTES = (\n    \"character_maximum_length\",\n    \"numeric_precision\",\n    \"numeric_scale\",\n    \"datetime_precision\",\n    \"interval_type\",\n    \"is_nullable\",\n)\nATTRIBUTES = TYPE_ATTRIBUTES + DETAIL_ATTRIBUTES\n\nCOLUMN_QUERY = \"\"\"\n    SELECT column_name, data_type, udt_schema, udt_name, domain_schema, domain_name,\n           character_maximum_length, numeric_precision, numeric_scale,\n           datetime_precision, interval_type, is_nullable\n    FROM information_schema.columns\n    WHERE table_schema = %s AND table_name = %s\n    ORDER BY ordinal_position\n\"\"\"\n\n\ndef qualify(name: str) -> tuple[str, str]:\n    \"\"\"Split 'schema.table' (default schema public) into its parts.\"\"\"\n    schema, _, table = name.rpartition(\".\")\n    return (schema or \"public\", table)\n\n\ndef fetch_schema(dsn: str, tables: list[str]) -> dict[str, dict[str, dict]]:\n    \"\"\"Return {'schema.table': {column: {attribute: value}}}. Missing tables map to {}.\"\"\"\n    result: dict[str, dict[str, dict]] = {}\n    conn = psycopg2.connect(dsn)\n    try:\n        with conn, conn.cursor() as cursor:\n            for name in tables:\n                schema, table = qualify(name)\n                cursor.execute(COLUMN_QUERY, (schema, table))\n                columns = {}\n                for row in cursor.fetchall():\n                    column_name, *values = row\n                    columns[column_name] = dict(zip(ATTRIBUTES, values))\n                result[f\"{schema}.{table}\"] = columns\n    finally:\n        conn.close()\n    return result\n\n\ndef type_label(column: dict) -> str:\n    \"\"\"Domain name if the column uses one, else the underlying type; schema shown unless pg_catalog.\"\"\"\n    schema = column.get(\"domain_schema\") or column.get(\"udt_schema\")\n    name = column.get(\"domain_name\") or column.get(\"udt_name\")\n    return name if schema in (None, \"pg_catalog\") else f\"{schema}.{name}\"\n\n\ndef classify(attribute: str, old, new) -> str:\n    \"\"\"Return 'additive' for widening changes, 'breaking' for everything else.\"\"\"\n    widening = (\"character_maximum_length\", \"numeric_precision\", \"datetime_precision\")\n    if attribute in widening and old is not None and new is not None and new > old:\n        return \"additive\"\n    if attribute in widening and old is not None and new is None:\n        return \"additive\"  # length limit removed, e.g. varchar(50) -> varchar\n    if attribute == \"is_nullable\" and old == \"YES\" and new == \"NO\":\n        return \"additive\"  # stricter source never produces a value the load can't take\n    return \"breaking\"\n\n\ndef diff_schema(baseline: dict, current: dict) -> list[tuple[str, str, str]]:\n    \"\"\"Return [(severity, table, message)] for every difference.\"\"\"\n    findings = []\n    for table in sorted(set(baseline) | set(current)):\n        old_cols = baseline.get(table, {})\n        new_cols = current.get(table, {})\n\n        if old_cols and not new_cols:\n            findings.append((\"breaking\", table, \"table missing or not visible to this role\"))\n            continue\n        if new_cols and not old_cols:\n            findings.append((\"additive\", table, \"table not in baseline\"))\n            continue\n\n        for column in sorted(set(old_cols) - set(new_cols)):\n            findings.append((\"breaking\", table, f\"column removed: {column}\"))\n        for column in sorted(set(new_cols) - set(old_cols)):\n            findings.append((\"additive\", table, f\"column added: {column} ({new_cols[column]['data_type']})\"))\n        for column in sorted(set(old_cols) & set(new_cols)):\n            old_type = tuple(old_cols[column].get(a) for a in TYPE_ATTRIBUTES)\n            new_type = tuple(new_cols[column].get(a) for a in TYPE_ATTRIBUTES)\n            if old_type != new_type:\n                # One finding per retyped column; precision\u002Flength differences follow from it.\n                old_label, new_label = type_label(old_cols[column]), type_label(new_cols[column])\n                findings.append((\"breaking\", table, f\"{column}: type {old_label} -> {new_label}\"))\n                continue\n            for attribute in DETAIL_ATTRIBUTES:\n                old = old_cols[column].get(attribute)\n                new = new_cols[column].get(attribute)\n                if old != new:\n                    severity = classify(attribute, old, new)\n                    findings.append((severity, table, f\"{column}: {attribute} {old} -> {new}\"))\n    return findings\n\n\ndef main() -> int:\n    parser = argparse.ArgumentParser(description=\"Detect PostgreSQL schema drift before an ETL run.\")\n    parser.add_argument(\"--dsn\", required=True, help=\"libpq connection string or URI\")\n    parser.add_argument(\"--tables\", required=True, help=\"Comma-separated tables, optionally schema-qualified\")\n    parser.add_argument(\"--snapshot\", action=\"store_true\", help=\"Write a baseline instead of checking\")\n    parser.add_argument(\"--output\", default=\"schema_baseline.json\", help=\"Baseline path to write (--snapshot)\")\n    parser.add_argument(\"--baseline\", help=\"Baseline path to compare against\")\n    parser.add_argument(\"--allow-additive\", action=\"store_true\", help=\"Exit 0 when all drift is additive\")\n    args = parser.parse_args()\n\n    tables = [t.strip() for t in args.tables.split(\",\") if t.strip()]\n    try:\n        current = fetch_schema(args.dsn, tables)\n    except psycopg2.Error as error:\n        print(f\"error: could not read schema: {error}\", file=sys.stderr)\n        return 2\n\n    if args.snapshot:\n        empty = [t for t, cols in current.items() if not cols]\n        if empty:\n            print(f\"error: no visible columns for {', '.join(empty)}\", file=sys.stderr)\n            return 2\n        with open(args.output, \"w\", encoding=\"utf-8\") as handle:\n            json.dump(current, handle, indent=2, sort_keys=True)\n            handle.write(\"\\n\")\n        print(f\"Baseline for {len(current)} table(s) written to {args.output}\")\n        return 0\n\n    if not args.baseline:\n        print(\"error: --baseline is required unless --snapshot is given\", file=sys.stderr)\n        return 2\n\n    with open(args.baseline, encoding=\"utf-8\") as handle:\n        baseline = json.load(handle)\n    # Only compare the tables requested on this run.\n    baseline = {t: cols for t, cols in baseline.items() if t in current}\n\n    findings = diff_schema(baseline, current)\n    if not findings:\n        print(\"No schema drift detected.\")\n        return 0\n\n    for severity, table, message in findings:\n        print(f\"{severity.upper()} {table}: {message}\")\n\n    breaking = sum(1 for f in findings if f[0] == \"breaking\")\n    additive = len(findings) - breaking\n    status = 0 if breaking == 0 and args.allow_additive else 1\n    print(f\"{breaking} breaking, {additive} additive change(s). Exit {status}.\")\n    return status\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n",[15,594,595,601,606,611,617,622,627,632,637,641,646,651,656,660,665,670,675,680,684,688,697,704,711,715,722,726,731,769,779,786,793,801,809,817,825,830,847,852,863,869,875,881,887,902,908,913,918,923,950,956,973,991,996,1001,1037,1043,1067,1078,1087,1102,1117,1128,1139,1149,1163,1180,1204,1243,1251,1257,1265,1270,1275,1295,1301,1325,1347,1400,1405,1410,1430,1436,1461,1512,1521,1552,1563,1596,1606,1614,1619,1624,1657,1663,1674,1705,1716,1727,1732,1748,1766,1772,1787,1802,1807,1812,1840,1865,1890,1930,1954,1980,2003,2018,2024,2035,2079,2085,2099,2110,2121,2134,2145,2191,2199,2204,2209,2225,2247,2278,2305,2334,2363,2382,2409,2420,2425,2456,2463,2474,2488,2520,2528,2533,2541,2569,2577,2610,2618,2649,2674,2689,2723,2731,2736,2746,2764,2771,2776,2799,2810,2816,2845,2850,2860,2870,2882,2889,2894,2906,2943,2948,2991,3010,3041,3083,3091,3096,3101,3117],{"__ignoreMap":103},[107,596,597],{"class":109,"line":110},[107,598,600],{"class":599},"sAwPA","#!\u002Fusr\u002Fbin\u002Fenv python3\n",[107,602,603],{"class":109,"line":128},[107,604,605],{"class":117},"\"\"\"\n",[107,607,608],{"class":109,"line":139},[107,609,610],{"class":117},"check_schema_drift.py\n",[107,612,613],{"class":109,"line":150},[107,614,616],{"emptyLinePlaceholder":615},true,"\n",[107,618,619],{"class":109,"line":204},[107,620,621],{"class":117},"Compares live PostgreSQL column definitions against a saved baseline and\n",[107,623,624],{"class":109,"line":217},[107,625,626],{"class":117},"reports added, removed, retyped (including a change of domain or of the\n",[107,628,629],{"class":109,"line":230},[107,630,631],{"class":117},"schema a user-defined type lives in), resized, precision-changed, or\n",[107,633,634],{"class":109,"line":242},[107,635,636],{"class":117},"nullability-changed columns before an ETL job runs.\n",[107,638,639],{"class":109,"line":254},[107,640,616],{"emptyLinePlaceholder":615},[107,642,643],{"class":109,"line":266},[107,644,645],{"class":117},"Usage:\n",[107,647,648],{"class":109,"line":279},[107,649,650],{"class":117},"    Baseline: python check_schema_drift.py --snapshot --dsn \u003Cdsn> --tables s.t1,t2 --output baseline.json\n",[107,652,653],{"class":109,"line":291},[107,654,655],{"class":117},"    Check:    python check_schema_drift.py --dsn \u003Cdsn> --tables s.t1,t2 --baseline baseline.json [--allow-additive]\n",[107,657,658],{"class":109,"line":303},[107,659,616],{"emptyLinePlaceholder":615},[107,661,662],{"class":109,"line":316},[107,663,664],{"class":117},"Reads:  information_schema.columns for each table through the given DSN.\n",[107,666,667],{"class":109,"line":327},[107,668,669],{"class":117},"Writes: a JSON snapshot in --snapshot mode; nothing in check mode.\n",[107,671,672],{"class":109,"line":333},[107,673,674],{"class":117},"Exit:   0 no drift (or only additive drift with --allow-additive),\n",[107,676,677],{"class":109,"line":341},[107,678,679],{"class":117},"        1 drift found, 2 usage or connection error.\n",[107,681,682],{"class":109,"line":352},[107,683,605],{"class":117},[107,685,686],{"class":109,"line":364},[107,687,616],{"emptyLinePlaceholder":615},[107,689,690,694],{"class":109,"line":375},[107,691,693],{"class":692},"snl16","import",[107,695,696],{"class":171}," argparse\n",[107,698,699,701],{"class":109,"line":386},[107,700,693],{"class":692},[107,702,703],{"class":171}," json\n",[107,705,706,708],{"class":109,"line":397},[107,707,693],{"class":692},[107,709,710],{"class":171}," sys\n",[107,712,713],{"class":109,"line":408},[107,714,616],{"emptyLinePlaceholder":615},[107,716,717,719],{"class":109,"line":420},[107,718,693],{"class":692},[107,720,721],{"class":171}," psycopg2\n",[107,723,724],{"class":109,"line":431},[107,725,616],{"emptyLinePlaceholder":615},[107,727,728],{"class":109,"line":443},[107,729,730],{"class":599},"# The first five together identify the column's type; the rest are compared one by one.\n",[107,732,733,736,739,742,745,748,751,753,756,758,761,763,766],{"class":109,"line":455},[107,734,735],{"class":121},"TYPE_ATTRIBUTES",[107,737,738],{"class":692}," =",[107,740,741],{"class":171}," (",[107,743,744],{"class":117},"\"data_type\"",[107,746,747],{"class":171},", ",[107,749,750],{"class":117},"\"udt_schema\"",[107,752,747],{"class":171},[107,754,755],{"class":117},"\"udt_name\"",[107,757,747],{"class":171},[107,759,760],{"class":117},"\"domain_schema\"",[107,762,747],{"class":171},[107,764,765],{"class":117},"\"domain_name\"",[107,767,768],{"class":171},")\n",[107,770,771,774,776],{"class":109,"line":464},[107,772,773],{"class":121},"DETAIL_ATTRIBUTES",[107,775,738],{"class":692},[107,777,778],{"class":171}," (\n",[107,780,781,784],{"class":109,"line":470},[107,782,783],{"class":117},"    \"character_maximum_length\"",[107,785,201],{"class":171},[107,787,788,791],{"class":109,"line":476},[107,789,790],{"class":117},"    \"numeric_precision\"",[107,792,201],{"class":171},[107,794,796,799],{"class":109,"line":795},31,[107,797,798],{"class":117},"    \"numeric_scale\"",[107,800,201],{"class":171},[107,802,804,807],{"class":109,"line":803},32,[107,805,806],{"class":117},"    \"datetime_precision\"",[107,808,201],{"class":171},[107,810,812,815],{"class":109,"line":811},33,[107,813,814],{"class":117},"    \"interval_type\"",[107,816,201],{"class":171},[107,818,820,823],{"class":109,"line":819},34,[107,821,822],{"class":117},"    \"is_nullable\"",[107,824,201],{"class":171},[107,826,828],{"class":109,"line":827},35,[107,829,768],{"class":171},[107,831,833,836,838,841,844],{"class":109,"line":832},36,[107,834,835],{"class":121},"ATTRIBUTES",[107,837,738],{"class":692},[107,839,840],{"class":121}," TYPE_ATTRIBUTES",[107,842,843],{"class":692}," +",[107,845,846],{"class":121}," DETAIL_ATTRIBUTES\n",[107,848,850],{"class":109,"line":849},37,[107,851,616],{"emptyLinePlaceholder":615},[107,853,855,858,860],{"class":109,"line":854},38,[107,856,857],{"class":121},"COLUMN_QUERY",[107,859,738],{"class":692},[107,861,862],{"class":117}," \"\"\"\n",[107,864,866],{"class":109,"line":865},39,[107,867,868],{"class":117},"    SELECT column_name, data_type, udt_schema, udt_name, domain_schema, domain_name,\n",[107,870,872],{"class":109,"line":871},40,[107,873,874],{"class":117},"           character_maximum_length, numeric_precision, numeric_scale,\n",[107,876,878],{"class":109,"line":877},41,[107,879,880],{"class":117},"           datetime_precision, interval_type, is_nullable\n",[107,882,884],{"class":109,"line":883},42,[107,885,886],{"class":117},"    FROM information_schema.columns\n",[107,888,890,893,896,899],{"class":109,"line":889},43,[107,891,892],{"class":117},"    WHERE table_schema = ",[107,894,895],{"class":121},"%s",[107,897,898],{"class":117}," AND table_name = ",[107,900,901],{"class":121},"%s\n",[107,903,905],{"class":109,"line":904},44,[107,906,907],{"class":117},"    ORDER BY ordinal_position\n",[107,909,911],{"class":109,"line":910},45,[107,912,605],{"class":117},[107,914,916],{"class":109,"line":915},46,[107,917,616],{"emptyLinePlaceholder":615},[107,919,921],{"class":109,"line":920},47,[107,922,616],{"emptyLinePlaceholder":615},[107,924,926,929,932,935,938,941,943,945,947],{"class":109,"line":925},48,[107,927,928],{"class":692},"def",[107,930,931],{"class":113}," qualify",[107,933,934],{"class":171},"(name: ",[107,936,937],{"class":121},"str",[107,939,940],{"class":171},") -> tuple[",[107,942,937],{"class":121},[107,944,747],{"class":171},[107,946,937],{"class":121},[107,948,949],{"class":171},"]:\n",[107,951,953],{"class":109,"line":952},49,[107,954,955],{"class":117},"    \"\"\"Split 'schema.table' (default schema public) into its parts.\"\"\"\n",[107,957,959,962,965,968,971],{"class":109,"line":958},50,[107,960,961],{"class":171},"    schema, _, table ",[107,963,964],{"class":692},"=",[107,966,967],{"class":171}," name.rpartition(",[107,969,970],{"class":117},"\".\"",[107,972,768],{"class":171},[107,974,976,979,982,985,988],{"class":109,"line":975},51,[107,977,978],{"class":692},"    return",[107,980,981],{"class":171}," (schema ",[107,983,984],{"class":692},"or",[107,986,987],{"class":117}," \"public\"",[107,989,990],{"class":171},", table)\n",[107,992,994],{"class":109,"line":993},52,[107,995,616],{"emptyLinePlaceholder":615},[107,997,999],{"class":109,"line":998},53,[107,1000,616],{"emptyLinePlaceholder":615},[107,1002,1004,1006,1009,1012,1014,1017,1019,1022,1024,1027,1029,1031,1034],{"class":109,"line":1003},54,[107,1005,928],{"class":692},[107,1007,1008],{"class":113}," fetch_schema",[107,1010,1011],{"class":171},"(dsn: ",[107,1013,937],{"class":121},[107,1015,1016],{"class":171},", tables: list[",[107,1018,937],{"class":121},[107,1020,1021],{"class":171},"]) -> dict[",[107,1023,937],{"class":121},[107,1025,1026],{"class":171},", dict[",[107,1028,937],{"class":121},[107,1030,747],{"class":171},[107,1032,1033],{"class":121},"dict",[107,1035,1036],{"class":171},"]]:\n",[107,1038,1040],{"class":109,"line":1039},55,[107,1041,1042],{"class":117},"    \"\"\"Return {'schema.table': {column: {attribute: value}}}. Missing tables map to {}.\"\"\"\n",[107,1044,1046,1049,1051,1053,1055,1057,1059,1062,1064],{"class":109,"line":1045},56,[107,1047,1048],{"class":171},"    result: dict[",[107,1050,937],{"class":121},[107,1052,1026],{"class":171},[107,1054,937],{"class":121},[107,1056,747],{"class":171},[107,1058,1033],{"class":121},[107,1060,1061],{"class":171},"]] ",[107,1063,964],{"class":692},[107,1065,1066],{"class":171}," {}\n",[107,1068,1070,1073,1075],{"class":109,"line":1069},57,[107,1071,1072],{"class":171},"    conn ",[107,1074,964],{"class":692},[107,1076,1077],{"class":171}," psycopg2.connect(dsn)\n",[107,1079,1081,1084],{"class":109,"line":1080},58,[107,1082,1083],{"class":692},"    try",[107,1085,1086],{"class":171},":\n",[107,1088,1090,1093,1096,1099],{"class":109,"line":1089},59,[107,1091,1092],{"class":692},"        with",[107,1094,1095],{"class":171}," conn, conn.cursor() ",[107,1097,1098],{"class":692},"as",[107,1100,1101],{"class":171}," cursor:\n",[107,1103,1105,1108,1111,1114],{"class":109,"line":1104},60,[107,1106,1107],{"class":692},"            for",[107,1109,1110],{"class":171}," name ",[107,1112,1113],{"class":692},"in",[107,1115,1116],{"class":171}," tables:\n",[107,1118,1120,1123,1125],{"class":109,"line":1119},61,[107,1121,1122],{"class":171},"                schema, table ",[107,1124,964],{"class":692},[107,1126,1127],{"class":171}," qualify(name)\n",[107,1129,1131,1134,1136],{"class":109,"line":1130},62,[107,1132,1133],{"class":171},"                cursor.execute(",[107,1135,857],{"class":121},[107,1137,1138],{"class":171},", (schema, table))\n",[107,1140,1142,1145,1147],{"class":109,"line":1141},63,[107,1143,1144],{"class":171},"                columns ",[107,1146,964],{"class":692},[107,1148,1066],{"class":171},[107,1150,1152,1155,1158,1160],{"class":109,"line":1151},64,[107,1153,1154],{"class":692},"                for",[107,1156,1157],{"class":171}," row ",[107,1159,1113],{"class":692},[107,1161,1162],{"class":171}," cursor.fetchall():\n",[107,1164,1166,1169,1172,1175,1177],{"class":109,"line":1165},65,[107,1167,1168],{"class":171},"                    column_name, ",[107,1170,1171],{"class":692},"*",[107,1173,1174],{"class":171},"values ",[107,1176,964],{"class":692},[107,1178,1179],{"class":171}," row\n",[107,1181,1183,1186,1188,1191,1194,1197,1199,1201],{"class":109,"line":1182},66,[107,1184,1185],{"class":171},"                    columns[column_name] ",[107,1187,964],{"class":692},[107,1189,1190],{"class":121}," dict",[107,1192,1193],{"class":171},"(",[107,1195,1196],{"class":121},"zip",[107,1198,1193],{"class":171},[107,1200,835],{"class":121},[107,1202,1203],{"class":171},", values))\n",[107,1205,1207,1210,1213,1215,1218,1221,1224,1226,1228,1231,1233,1235,1238,1240],{"class":109,"line":1206},67,[107,1208,1209],{"class":171},"                result[",[107,1211,1212],{"class":692},"f",[107,1214,508],{"class":117},[107,1216,1217],{"class":121},"{",[107,1219,1220],{"class":171},"schema",[107,1222,1223],{"class":121},"}",[107,1225,96],{"class":117},[107,1227,1217],{"class":121},[107,1229,1230],{"class":171},"table",[107,1232,1223],{"class":121},[107,1234,508],{"class":117},[107,1236,1237],{"class":171},"] ",[107,1239,964],{"class":692},[107,1241,1242],{"class":171}," columns\n",[107,1244,1246,1249],{"class":109,"line":1245},68,[107,1247,1248],{"class":692},"    finally",[107,1250,1086],{"class":171},[107,1252,1254],{"class":109,"line":1253},69,[107,1255,1256],{"class":171},"        conn.close()\n",[107,1258,1260,1262],{"class":109,"line":1259},70,[107,1261,978],{"class":692},[107,1263,1264],{"class":171}," result\n",[107,1266,1268],{"class":109,"line":1267},71,[107,1269,616],{"emptyLinePlaceholder":615},[107,1271,1273],{"class":109,"line":1272},72,[107,1274,616],{"emptyLinePlaceholder":615},[107,1276,1278,1280,1283,1286,1288,1291,1293],{"class":109,"line":1277},73,[107,1279,928],{"class":692},[107,1281,1282],{"class":113}," type_label",[107,1284,1285],{"class":171},"(column: ",[107,1287,1033],{"class":121},[107,1289,1290],{"class":171},") -> ",[107,1292,937],{"class":121},[107,1294,1086],{"class":171},[107,1296,1298],{"class":109,"line":1297},74,[107,1299,1300],{"class":117},"    \"\"\"Domain name if the column uses one, else the underlying type; schema shown unless pg_catalog.\"\"\"\n",[107,1302,1304,1307,1309,1312,1314,1317,1319,1321,1323],{"class":109,"line":1303},75,[107,1305,1306],{"class":171},"    schema ",[107,1308,964],{"class":692},[107,1310,1311],{"class":171}," column.get(",[107,1313,760],{"class":117},[107,1315,1316],{"class":171},") ",[107,1318,984],{"class":692},[107,1320,1311],{"class":171},[107,1322,750],{"class":117},[107,1324,768],{"class":171},[107,1326,1328,1331,1333,1335,1337,1339,1341,1343,1345],{"class":109,"line":1327},76,[107,1329,1330],{"class":171},"    name ",[107,1332,964],{"class":692},[107,1334,1311],{"class":171},[107,1336,765],{"class":117},[107,1338,1316],{"class":171},[107,1340,984],{"class":692},[107,1342,1311],{"class":171},[107,1344,755],{"class":117},[107,1346,768],{"class":171},[107,1348,1350,1352,1354,1357,1360,1362,1364,1367,1369,1372,1374,1377,1380,1382,1384,1386,1388,1390,1392,1395,1397],{"class":109,"line":1349},77,[107,1351,978],{"class":692},[107,1353,1110],{"class":171},[107,1355,1356],{"class":692},"if",[107,1358,1359],{"class":171}," schema ",[107,1361,1113],{"class":692},[107,1363,741],{"class":171},[107,1365,1366],{"class":121},"None",[107,1368,747],{"class":171},[107,1370,1371],{"class":117},"\"pg_catalog\"",[107,1373,1316],{"class":171},[107,1375,1376],{"class":692},"else",[107,1378,1379],{"class":692}," f",[107,1381,508],{"class":117},[107,1383,1217],{"class":121},[107,1385,1220],{"class":171},[107,1387,1223],{"class":121},[107,1389,96],{"class":117},[107,1391,1217],{"class":121},[107,1393,1394],{"class":171},"name",[107,1396,1223],{"class":121},[107,1398,1399],{"class":117},"\"\n",[107,1401,1403],{"class":109,"line":1402},78,[107,1404,616],{"emptyLinePlaceholder":615},[107,1406,1408],{"class":109,"line":1407},79,[107,1409,616],{"emptyLinePlaceholder":615},[107,1411,1413,1415,1418,1421,1423,1426,1428],{"class":109,"line":1412},80,[107,1414,928],{"class":692},[107,1416,1417],{"class":113}," classify",[107,1419,1420],{"class":171},"(attribute: ",[107,1422,937],{"class":121},[107,1424,1425],{"class":171},", old, new) -> ",[107,1427,937],{"class":121},[107,1429,1086],{"class":171},[107,1431,1433],{"class":109,"line":1432},81,[107,1434,1435],{"class":117},"    \"\"\"Return 'additive' for widening changes, 'breaking' for everything else.\"\"\"\n",[107,1437,1439,1442,1444,1446,1449,1451,1454,1456,1459],{"class":109,"line":1438},82,[107,1440,1441],{"class":171},"    widening ",[107,1443,964],{"class":692},[107,1445,741],{"class":171},[107,1447,1448],{"class":117},"\"character_maximum_length\"",[107,1450,747],{"class":171},[107,1452,1453],{"class":117},"\"numeric_precision\"",[107,1455,747],{"class":171},[107,1457,1458],{"class":117},"\"datetime_precision\"",[107,1460,768],{"class":171},[107,1462,1464,1467,1470,1472,1475,1478,1481,1484,1487,1490,1493,1496,1498,1500,1502,1504,1506,1509],{"class":109,"line":1463},83,[107,1465,1466],{"class":692},"    if",[107,1468,1469],{"class":171}," attribute ",[107,1471,1113],{"class":692},[107,1473,1474],{"class":171}," widening ",[107,1476,1477],{"class":692},"and",[107,1479,1480],{"class":171}," old ",[107,1482,1483],{"class":692},"is",[107,1485,1486],{"class":692}," not",[107,1488,1489],{"class":121}," None",[107,1491,1492],{"class":692}," and",[107,1494,1495],{"class":171}," new ",[107,1497,1483],{"class":692},[107,1499,1486],{"class":692},[107,1501,1489],{"class":121},[107,1503,1492],{"class":692},[107,1505,1495],{"class":171},[107,1507,1508],{"class":692},">",[107,1510,1511],{"class":171}," old:\n",[107,1513,1515,1518],{"class":109,"line":1514},84,[107,1516,1517],{"class":692},"        return",[107,1519,1520],{"class":117}," \"additive\"\n",[107,1522,1524,1526,1528,1530,1532,1534,1536,1538,1540,1542,1544,1546,1548,1550],{"class":109,"line":1523},85,[107,1525,1466],{"class":692},[107,1527,1469],{"class":171},[107,1529,1113],{"class":692},[107,1531,1474],{"class":171},[107,1533,1477],{"class":692},[107,1535,1480],{"class":171},[107,1537,1483],{"class":692},[107,1539,1486],{"class":692},[107,1541,1489],{"class":121},[107,1543,1492],{"class":692},[107,1545,1495],{"class":171},[107,1547,1483],{"class":692},[107,1549,1489],{"class":121},[107,1551,1086],{"class":171},[107,1553,1555,1557,1560],{"class":109,"line":1554},86,[107,1556,1517],{"class":692},[107,1558,1559],{"class":117}," \"additive\"",[107,1561,1562],{"class":599},"  # length limit removed, e.g. varchar(50) -> varchar\n",[107,1564,1566,1568,1570,1573,1576,1578,1580,1582,1585,1587,1589,1591,1594],{"class":109,"line":1565},87,[107,1567,1466],{"class":692},[107,1569,1469],{"class":171},[107,1571,1572],{"class":692},"==",[107,1574,1575],{"class":117}," \"is_nullable\"",[107,1577,1492],{"class":692},[107,1579,1480],{"class":171},[107,1581,1572],{"class":692},[107,1583,1584],{"class":117}," \"YES\"",[107,1586,1492],{"class":692},[107,1588,1495],{"class":171},[107,1590,1572],{"class":692},[107,1592,1593],{"class":117}," \"NO\"",[107,1595,1086],{"class":171},[107,1597,1599,1601,1603],{"class":109,"line":1598},88,[107,1600,1517],{"class":692},[107,1602,1559],{"class":117},[107,1604,1605],{"class":599},"  # stricter source never produces a value the load can't take\n",[107,1607,1609,1611],{"class":109,"line":1608},89,[107,1610,978],{"class":692},[107,1612,1613],{"class":117}," \"breaking\"\n",[107,1615,1617],{"class":109,"line":1616},90,[107,1618,616],{"emptyLinePlaceholder":615},[107,1620,1622],{"class":109,"line":1621},91,[107,1623,616],{"emptyLinePlaceholder":615},[107,1625,1627,1629,1632,1635,1637,1640,1642,1645,1647,1649,1651,1653,1655],{"class":109,"line":1626},92,[107,1628,928],{"class":692},[107,1630,1631],{"class":113}," diff_schema",[107,1633,1634],{"class":171},"(baseline: ",[107,1636,1033],{"class":121},[107,1638,1639],{"class":171},", current: ",[107,1641,1033],{"class":121},[107,1643,1644],{"class":171},") -> list[tuple[",[107,1646,937],{"class":121},[107,1648,747],{"class":171},[107,1650,937],{"class":121},[107,1652,747],{"class":171},[107,1654,937],{"class":121},[107,1656,1036],{"class":171},[107,1658,1660],{"class":109,"line":1659},93,[107,1661,1662],{"class":117},"    \"\"\"Return [(severity, table, message)] for every difference.\"\"\"\n",[107,1664,1666,1669,1671],{"class":109,"line":1665},94,[107,1667,1668],{"class":171},"    findings ",[107,1670,964],{"class":692},[107,1672,1673],{"class":171}," []\n",[107,1675,1677,1680,1683,1685,1688,1690,1693,1696,1699,1702],{"class":109,"line":1676},95,[107,1678,1679],{"class":692},"    for",[107,1681,1682],{"class":171}," table ",[107,1684,1113],{"class":692},[107,1686,1687],{"class":121}," sorted",[107,1689,1193],{"class":171},[107,1691,1692],{"class":121},"set",[107,1694,1695],{"class":171},"(baseline) ",[107,1697,1698],{"class":692},"|",[107,1700,1701],{"class":121}," set",[107,1703,1704],{"class":171},"(current)):\n",[107,1706,1708,1711,1713],{"class":109,"line":1707},96,[107,1709,1710],{"class":171},"        old_cols ",[107,1712,964],{"class":692},[107,1714,1715],{"class":171}," baseline.get(table, {})\n",[107,1717,1719,1722,1724],{"class":109,"line":1718},97,[107,1720,1721],{"class":171},"        new_cols ",[107,1723,964],{"class":692},[107,1725,1726],{"class":171}," current.get(table, {})\n",[107,1728,1730],{"class":109,"line":1729},98,[107,1731,616],{"emptyLinePlaceholder":615},[107,1733,1735,1738,1741,1743,1745],{"class":109,"line":1734},99,[107,1736,1737],{"class":692},"        if",[107,1739,1740],{"class":171}," old_cols ",[107,1742,1477],{"class":692},[107,1744,1486],{"class":692},[107,1746,1747],{"class":171}," new_cols:\n",[107,1749,1751,1754,1757,1760,1763],{"class":109,"line":1750},100,[107,1752,1753],{"class":171},"            findings.append((",[107,1755,1756],{"class":117},"\"breaking\"",[107,1758,1759],{"class":171},", table, ",[107,1761,1762],{"class":117},"\"table missing or not visible to this role\"",[107,1764,1765],{"class":171},"))\n",[107,1767,1769],{"class":109,"line":1768},101,[107,1770,1771],{"class":692},"            continue\n",[107,1773,1775,1777,1780,1782,1784],{"class":109,"line":1774},102,[107,1776,1737],{"class":692},[107,1778,1779],{"class":171}," new_cols ",[107,1781,1477],{"class":692},[107,1783,1486],{"class":692},[107,1785,1786],{"class":171}," old_cols:\n",[107,1788,1790,1792,1795,1797,1800],{"class":109,"line":1789},103,[107,1791,1753],{"class":171},[107,1793,1794],{"class":117},"\"additive\"",[107,1796,1759],{"class":171},[107,1798,1799],{"class":117},"\"table not in baseline\"",[107,1801,1765],{"class":171},[107,1803,1805],{"class":109,"line":1804},104,[107,1806,1771],{"class":692},[107,1808,1810],{"class":109,"line":1809},105,[107,1811,616],{"emptyLinePlaceholder":615},[107,1813,1815,1818,1821,1823,1825,1827,1829,1832,1835,1837],{"class":109,"line":1814},106,[107,1816,1817],{"class":692},"        for",[107,1819,1820],{"class":171}," column ",[107,1822,1113],{"class":692},[107,1824,1687],{"class":121},[107,1826,1193],{"class":171},[107,1828,1692],{"class":121},[107,1830,1831],{"class":171},"(old_cols) ",[107,1833,1834],{"class":692},"-",[107,1836,1701],{"class":121},[107,1838,1839],{"class":171},"(new_cols)):\n",[107,1841,1843,1845,1847,1849,1851,1854,1856,1859,1861,1863],{"class":109,"line":1842},107,[107,1844,1753],{"class":171},[107,1846,1756],{"class":117},[107,1848,1759],{"class":171},[107,1850,1212],{"class":692},[107,1852,1853],{"class":117},"\"column removed: ",[107,1855,1217],{"class":121},[107,1857,1858],{"class":171},"column",[107,1860,1223],{"class":121},[107,1862,508],{"class":117},[107,1864,1765],{"class":171},[107,1866,1868,1870,1872,1874,1876,1878,1880,1883,1885,1887],{"class":109,"line":1867},108,[107,1869,1817],{"class":692},[107,1871,1820],{"class":171},[107,1873,1113],{"class":692},[107,1875,1687],{"class":121},[107,1877,1193],{"class":171},[107,1879,1692],{"class":121},[107,1881,1882],{"class":171},"(new_cols) ",[107,1884,1834],{"class":692},[107,1886,1701],{"class":121},[107,1888,1889],{"class":171},"(old_cols)):\n",[107,1891,1893,1895,1897,1899,1901,1904,1906,1908,1910,1912,1914,1917,1920,1923,1925,1928],{"class":109,"line":1892},109,[107,1894,1753],{"class":171},[107,1896,1794],{"class":117},[107,1898,1759],{"class":171},[107,1900,1212],{"class":692},[107,1902,1903],{"class":117},"\"column added: ",[107,1905,1217],{"class":121},[107,1907,1858],{"class":171},[107,1909,1223],{"class":121},[107,1911,741],{"class":117},[107,1913,1217],{"class":121},[107,1915,1916],{"class":171},"new_cols[column][",[107,1918,1919],{"class":117},"'data_type'",[107,1921,1922],{"class":171},"]",[107,1924,1223],{"class":121},[107,1926,1927],{"class":117},")\"",[107,1929,1765],{"class":171},[107,1931,1933,1935,1937,1939,1941,1943,1945,1947,1950,1952],{"class":109,"line":1932},110,[107,1934,1817],{"class":692},[107,1936,1820],{"class":171},[107,1938,1113],{"class":692},[107,1940,1687],{"class":121},[107,1942,1193],{"class":171},[107,1944,1692],{"class":121},[107,1946,1831],{"class":171},[107,1948,1949],{"class":692},"&",[107,1951,1701],{"class":121},[107,1953,1839],{"class":171},[107,1955,1957,1960,1962,1965,1968,1971,1974,1976,1978],{"class":109,"line":1956},111,[107,1958,1959],{"class":171},"            old_type ",[107,1961,964],{"class":692},[107,1963,1964],{"class":121}," tuple",[107,1966,1967],{"class":171},"(old_cols[column].get(a) ",[107,1969,1970],{"class":692},"for",[107,1972,1973],{"class":171}," a ",[107,1975,1113],{"class":692},[107,1977,840],{"class":121},[107,1979,768],{"class":171},[107,1981,1983,1986,1988,1990,1993,1995,1997,1999,2001],{"class":109,"line":1982},112,[107,1984,1985],{"class":171},"            new_type ",[107,1987,964],{"class":692},[107,1989,1964],{"class":121},[107,1991,1992],{"class":171},"(new_cols[column].get(a) ",[107,1994,1970],{"class":692},[107,1996,1973],{"class":171},[107,1998,1113],{"class":692},[107,2000,840],{"class":121},[107,2002,768],{"class":171},[107,2004,2006,2009,2012,2015],{"class":109,"line":2005},113,[107,2007,2008],{"class":692},"            if",[107,2010,2011],{"class":171}," old_type ",[107,2013,2014],{"class":692},"!=",[107,2016,2017],{"class":171}," new_type:\n",[107,2019,2021],{"class":109,"line":2020},114,[107,2022,2023],{"class":599},"                # One finding per retyped column; precision\u002Flength differences follow from it.\n",[107,2025,2027,2030,2032],{"class":109,"line":2026},115,[107,2028,2029],{"class":171},"                old_label, new_label ",[107,2031,964],{"class":692},[107,2033,2034],{"class":171}," type_label(old_cols[column]), type_label(new_cols[column])\n",[107,2036,2038,2041,2043,2045,2047,2049,2051,2053,2055,2058,2060,2063,2065,2068,2070,2073,2075,2077],{"class":109,"line":2037},116,[107,2039,2040],{"class":171},"                findings.append((",[107,2042,1756],{"class":117},[107,2044,1759],{"class":171},[107,2046,1212],{"class":692},[107,2048,508],{"class":117},[107,2050,1217],{"class":121},[107,2052,1858],{"class":171},[107,2054,1223],{"class":121},[107,2056,2057],{"class":117},": type ",[107,2059,1217],{"class":121},[107,2061,2062],{"class":171},"old_label",[107,2064,1223],{"class":121},[107,2066,2067],{"class":117}," -> ",[107,2069,1217],{"class":121},[107,2071,2072],{"class":171},"new_label",[107,2074,1223],{"class":121},[107,2076,508],{"class":117},[107,2078,1765],{"class":171},[107,2080,2082],{"class":109,"line":2081},117,[107,2083,2084],{"class":692},"                continue\n",[107,2086,2088,2090,2092,2094,2097],{"class":109,"line":2087},118,[107,2089,1107],{"class":692},[107,2091,1469],{"class":171},[107,2093,1113],{"class":692},[107,2095,2096],{"class":121}," DETAIL_ATTRIBUTES",[107,2098,1086],{"class":171},[107,2100,2102,2105,2107],{"class":109,"line":2101},119,[107,2103,2104],{"class":171},"                old ",[107,2106,964],{"class":692},[107,2108,2109],{"class":171}," old_cols[column].get(attribute)\n",[107,2111,2113,2116,2118],{"class":109,"line":2112},120,[107,2114,2115],{"class":171},"                new ",[107,2117,964],{"class":692},[107,2119,2120],{"class":171}," new_cols[column].get(attribute)\n",[107,2122,2124,2127,2129,2131],{"class":109,"line":2123},121,[107,2125,2126],{"class":692},"                if",[107,2128,1480],{"class":171},[107,2130,2014],{"class":692},[107,2132,2133],{"class":171}," new:\n",[107,2135,2137,2140,2142],{"class":109,"line":2136},122,[107,2138,2139],{"class":171},"                    severity ",[107,2141,964],{"class":692},[107,2143,2144],{"class":171}," classify(attribute, old, new)\n",[107,2146,2148,2151,2153,2155,2157,2159,2161,2163,2165,2168,2170,2173,2176,2178,2180,2182,2185,2187,2189],{"class":109,"line":2147},123,[107,2149,2150],{"class":171},"                    findings.append((severity, table, ",[107,2152,1212],{"class":692},[107,2154,508],{"class":117},[107,2156,1217],{"class":121},[107,2158,1858],{"class":171},[107,2160,1223],{"class":121},[107,2162,195],{"class":117},[107,2164,1217],{"class":121},[107,2166,2167],{"class":171},"attribute",[107,2169,1223],{"class":121},[107,2171,2172],{"class":121}," {",[107,2174,2175],{"class":171},"old",[107,2177,1223],{"class":121},[107,2179,2067],{"class":117},[107,2181,1217],{"class":121},[107,2183,2184],{"class":171},"new",[107,2186,1223],{"class":121},[107,2188,508],{"class":117},[107,2190,1765],{"class":171},[107,2192,2194,2196],{"class":109,"line":2193},124,[107,2195,978],{"class":692},[107,2197,2198],{"class":171}," findings\n",[107,2200,2202],{"class":109,"line":2201},125,[107,2203,616],{"emptyLinePlaceholder":615},[107,2205,2207],{"class":109,"line":2206},126,[107,2208,616],{"emptyLinePlaceholder":615},[107,2210,2212,2214,2217,2220,2223],{"class":109,"line":2211},127,[107,2213,928],{"class":692},[107,2215,2216],{"class":113}," main",[107,2218,2219],{"class":171},"() -> ",[107,2221,2222],{"class":121},"int",[107,2224,1086],{"class":171},[107,2226,2228,2231,2233,2236,2240,2242,2245],{"class":109,"line":2227},128,[107,2229,2230],{"class":171},"    parser ",[107,2232,964],{"class":692},[107,2234,2235],{"class":171}," argparse.ArgumentParser(",[107,2237,2239],{"class":2238},"s9osk","description",[107,2241,964],{"class":692},[107,2243,2244],{"class":117},"\"Detect PostgreSQL schema drift before an ETL run.\"",[107,2246,768],{"class":171},[107,2248,2250,2253,2256,2258,2261,2263,2266,2268,2271,2273,2276],{"class":109,"line":2249},129,[107,2251,2252],{"class":171},"    parser.add_argument(",[107,2254,2255],{"class":117},"\"--dsn\"",[107,2257,747],{"class":171},[107,2259,2260],{"class":2238},"required",[107,2262,964],{"class":692},[107,2264,2265],{"class":121},"True",[107,2267,747],{"class":171},[107,2269,2270],{"class":2238},"help",[107,2272,964],{"class":692},[107,2274,2275],{"class":117},"\"libpq connection string or URI\"",[107,2277,768],{"class":171},[107,2279,2281,2283,2286,2288,2290,2292,2294,2296,2298,2300,2303],{"class":109,"line":2280},130,[107,2282,2252],{"class":171},[107,2284,2285],{"class":117},"\"--tables\"",[107,2287,747],{"class":171},[107,2289,2260],{"class":2238},[107,2291,964],{"class":692},[107,2293,2265],{"class":121},[107,2295,747],{"class":171},[107,2297,2270],{"class":2238},[107,2299,964],{"class":692},[107,2301,2302],{"class":117},"\"Comma-separated tables, optionally schema-qualified\"",[107,2304,768],{"class":171},[107,2306,2308,2310,2313,2315,2318,2320,2323,2325,2327,2329,2332],{"class":109,"line":2307},131,[107,2309,2252],{"class":171},[107,2311,2312],{"class":117},"\"--snapshot\"",[107,2314,747],{"class":171},[107,2316,2317],{"class":2238},"action",[107,2319,964],{"class":692},[107,2321,2322],{"class":117},"\"store_true\"",[107,2324,747],{"class":171},[107,2326,2270],{"class":2238},[107,2328,964],{"class":692},[107,2330,2331],{"class":117},"\"Write a baseline instead of checking\"",[107,2333,768],{"class":171},[107,2335,2337,2339,2342,2344,2347,2349,2352,2354,2356,2358,2361],{"class":109,"line":2336},132,[107,2338,2252],{"class":171},[107,2340,2341],{"class":117},"\"--output\"",[107,2343,747],{"class":171},[107,2345,2346],{"class":2238},"default",[107,2348,964],{"class":692},[107,2350,2351],{"class":117},"\"schema_baseline.json\"",[107,2353,747],{"class":171},[107,2355,2270],{"class":2238},[107,2357,964],{"class":692},[107,2359,2360],{"class":117},"\"Baseline path to write (--snapshot)\"",[107,2362,768],{"class":171},[107,2364,2366,2368,2371,2373,2375,2377,2380],{"class":109,"line":2365},133,[107,2367,2252],{"class":171},[107,2369,2370],{"class":117},"\"--baseline\"",[107,2372,747],{"class":171},[107,2374,2270],{"class":2238},[107,2376,964],{"class":692},[107,2378,2379],{"class":117},"\"Baseline path to compare against\"",[107,2381,768],{"class":171},[107,2383,2385,2387,2390,2392,2394,2396,2398,2400,2402,2404,2407],{"class":109,"line":2384},134,[107,2386,2252],{"class":171},[107,2388,2389],{"class":117},"\"--allow-additive\"",[107,2391,747],{"class":171},[107,2393,2317],{"class":2238},[107,2395,964],{"class":692},[107,2397,2322],{"class":117},[107,2399,747],{"class":171},[107,2401,2270],{"class":2238},[107,2403,964],{"class":692},[107,2405,2406],{"class":117},"\"Exit 0 when all drift is additive\"",[107,2408,768],{"class":171},[107,2410,2412,2415,2417],{"class":109,"line":2411},135,[107,2413,2414],{"class":171},"    args ",[107,2416,964],{"class":692},[107,2418,2419],{"class":171}," parser.parse_args()\n",[107,2421,2423],{"class":109,"line":2422},136,[107,2424,616],{"emptyLinePlaceholder":615},[107,2426,2428,2431,2433,2436,2438,2441,2443,2446,2449,2451,2453],{"class":109,"line":2427},137,[107,2429,2430],{"class":171},"    tables ",[107,2432,964],{"class":692},[107,2434,2435],{"class":171}," [t.strip() ",[107,2437,1970],{"class":692},[107,2439,2440],{"class":171}," t ",[107,2442,1113],{"class":692},[107,2444,2445],{"class":171}," args.tables.split(",[107,2447,2448],{"class":117},"\",\"",[107,2450,1316],{"class":171},[107,2452,1356],{"class":692},[107,2454,2455],{"class":171}," t.strip()]\n",[107,2457,2459,2461],{"class":109,"line":2458},138,[107,2460,1083],{"class":692},[107,2462,1086],{"class":171},[107,2464,2466,2469,2471],{"class":109,"line":2465},139,[107,2467,2468],{"class":171},"        current ",[107,2470,964],{"class":692},[107,2472,2473],{"class":171}," fetch_schema(args.dsn, tables)\n",[107,2475,2477,2480,2483,2485],{"class":109,"line":2476},140,[107,2478,2479],{"class":692},"    except",[107,2481,2482],{"class":171}," psycopg2.Error ",[107,2484,1098],{"class":692},[107,2486,2487],{"class":171}," error:\n",[107,2489,2491,2494,2496,2498,2501,2503,2506,2508,2510,2512,2515,2517],{"class":109,"line":2490},141,[107,2492,2493],{"class":121},"        print",[107,2495,1193],{"class":171},[107,2497,1212],{"class":692},[107,2499,2500],{"class":117},"\"error: could not read schema: ",[107,2502,1217],{"class":121},[107,2504,2505],{"class":171},"error",[107,2507,1223],{"class":121},[107,2509,508],{"class":117},[107,2511,747],{"class":171},[107,2513,2514],{"class":2238},"file",[107,2516,964],{"class":692},[107,2518,2519],{"class":171},"sys.stderr)\n",[107,2521,2523,2525],{"class":109,"line":2522},142,[107,2524,1517],{"class":692},[107,2526,2527],{"class":121}," 2\n",[107,2529,2531],{"class":109,"line":2530},143,[107,2532,616],{"emptyLinePlaceholder":615},[107,2534,2536,2538],{"class":109,"line":2535},144,[107,2537,1466],{"class":692},[107,2539,2540],{"class":171}," args.snapshot:\n",[107,2542,2544,2547,2549,2552,2554,2557,2559,2562,2564,2566],{"class":109,"line":2543},145,[107,2545,2546],{"class":171},"        empty ",[107,2548,964],{"class":692},[107,2550,2551],{"class":171}," [t ",[107,2553,1970],{"class":692},[107,2555,2556],{"class":171}," t, cols ",[107,2558,1113],{"class":692},[107,2560,2561],{"class":171}," current.items() ",[107,2563,1356],{"class":692},[107,2565,1486],{"class":692},[107,2567,2568],{"class":171}," cols]\n",[107,2570,2572,2574],{"class":109,"line":2571},146,[107,2573,1737],{"class":692},[107,2575,2576],{"class":171}," empty:\n",[107,2578,2580,2583,2585,2587,2590,2592,2595,2598,2600,2602,2604,2606,2608],{"class":109,"line":2579},147,[107,2581,2582],{"class":121},"            print",[107,2584,1193],{"class":171},[107,2586,1212],{"class":692},[107,2588,2589],{"class":117},"\"error: no visible columns for ",[107,2591,1217],{"class":121},[107,2593,2594],{"class":117},"', '",[107,2596,2597],{"class":171},".join(empty)",[107,2599,1223],{"class":121},[107,2601,508],{"class":117},[107,2603,747],{"class":171},[107,2605,2514],{"class":2238},[107,2607,964],{"class":692},[107,2609,2519],{"class":171},[107,2611,2613,2616],{"class":109,"line":2612},148,[107,2614,2615],{"class":692},"            return",[107,2617,2527],{"class":121},[107,2619,2621,2623,2626,2629,2632,2634,2637,2639,2642,2644,2646],{"class":109,"line":2620},149,[107,2622,1092],{"class":692},[107,2624,2625],{"class":121}," open",[107,2627,2628],{"class":171},"(args.output, ",[107,2630,2631],{"class":117},"\"w\"",[107,2633,747],{"class":171},[107,2635,2636],{"class":2238},"encoding",[107,2638,964],{"class":692},[107,2640,2641],{"class":117},"\"utf-8\"",[107,2643,1316],{"class":171},[107,2645,1098],{"class":692},[107,2647,2648],{"class":171}," handle:\n",[107,2650,2652,2655,2658,2660,2663,2665,2668,2670,2672],{"class":109,"line":2651},150,[107,2653,2654],{"class":171},"            json.dump(current, handle, ",[107,2656,2657],{"class":2238},"indent",[107,2659,964],{"class":692},[107,2661,2662],{"class":121},"2",[107,2664,747],{"class":171},[107,2666,2667],{"class":2238},"sort_keys",[107,2669,964],{"class":692},[107,2671,2265],{"class":121},[107,2673,768],{"class":171},[107,2675,2677,2680,2682,2685,2687],{"class":109,"line":2676},151,[107,2678,2679],{"class":171},"            handle.write(",[107,2681,508],{"class":117},[107,2683,2684],{"class":121},"\\n",[107,2686,508],{"class":117},[107,2688,768],{"class":171},[107,2690,2692,2694,2696,2698,2701,2704,2707,2709,2712,2714,2717,2719,2721],{"class":109,"line":2691},152,[107,2693,2493],{"class":121},[107,2695,1193],{"class":171},[107,2697,1212],{"class":692},[107,2699,2700],{"class":117},"\"Baseline for ",[107,2702,2703],{"class":121},"{len",[107,2705,2706],{"class":171},"(current)",[107,2708,1223],{"class":121},[107,2710,2711],{"class":117}," table(s) written to ",[107,2713,1217],{"class":121},[107,2715,2716],{"class":171},"args.output",[107,2718,1223],{"class":121},[107,2720,508],{"class":117},[107,2722,768],{"class":171},[107,2724,2726,2728],{"class":109,"line":2725},153,[107,2727,1517],{"class":692},[107,2729,2730],{"class":121}," 0\n",[107,2732,2734],{"class":109,"line":2733},154,[107,2735,616],{"emptyLinePlaceholder":615},[107,2737,2739,2741,2743],{"class":109,"line":2738},155,[107,2740,1466],{"class":692},[107,2742,1486],{"class":692},[107,2744,2745],{"class":171}," args.baseline:\n",[107,2747,2749,2751,2753,2756,2758,2760,2762],{"class":109,"line":2748},156,[107,2750,2493],{"class":121},[107,2752,1193],{"class":171},[107,2754,2755],{"class":117},"\"error: --baseline is required unless --snapshot is given\"",[107,2757,747],{"class":171},[107,2759,2514],{"class":2238},[107,2761,964],{"class":692},[107,2763,2519],{"class":171},[107,2765,2767,2769],{"class":109,"line":2766},157,[107,2768,1517],{"class":692},[107,2770,2527],{"class":121},[107,2772,2774],{"class":109,"line":2773},158,[107,2775,616],{"emptyLinePlaceholder":615},[107,2777,2779,2782,2784,2787,2789,2791,2793,2795,2797],{"class":109,"line":2778},159,[107,2780,2781],{"class":692},"    with",[107,2783,2625],{"class":121},[107,2785,2786],{"class":171},"(args.baseline, ",[107,2788,2636],{"class":2238},[107,2790,964],{"class":692},[107,2792,2641],{"class":117},[107,2794,1316],{"class":171},[107,2796,1098],{"class":692},[107,2798,2648],{"class":171},[107,2800,2802,2805,2807],{"class":109,"line":2801},160,[107,2803,2804],{"class":171},"        baseline ",[107,2806,964],{"class":692},[107,2808,2809],{"class":171}," json.load(handle)\n",[107,2811,2813],{"class":109,"line":2812},161,[107,2814,2815],{"class":599},"    # Only compare the tables requested on this run.\n",[107,2817,2819,2822,2824,2827,2829,2831,2833,2836,2838,2840,2842],{"class":109,"line":2818},162,[107,2820,2821],{"class":171},"    baseline ",[107,2823,964],{"class":692},[107,2825,2826],{"class":171}," {t: cols ",[107,2828,1970],{"class":692},[107,2830,2556],{"class":171},[107,2832,1113],{"class":692},[107,2834,2835],{"class":171}," baseline.items() ",[107,2837,1356],{"class":692},[107,2839,2440],{"class":171},[107,2841,1113],{"class":692},[107,2843,2844],{"class":171}," current}\n",[107,2846,2848],{"class":109,"line":2847},163,[107,2849,616],{"emptyLinePlaceholder":615},[107,2851,2853,2855,2857],{"class":109,"line":2852},164,[107,2854,1668],{"class":171},[107,2856,964],{"class":692},[107,2858,2859],{"class":171}," diff_schema(baseline, current)\n",[107,2861,2863,2865,2867],{"class":109,"line":2862},165,[107,2864,1466],{"class":692},[107,2866,1486],{"class":692},[107,2868,2869],{"class":171}," findings:\n",[107,2871,2873,2875,2877,2880],{"class":109,"line":2872},166,[107,2874,2493],{"class":121},[107,2876,1193],{"class":171},[107,2878,2879],{"class":117},"\"No schema drift detected.\"",[107,2881,768],{"class":171},[107,2883,2885,2887],{"class":109,"line":2884},167,[107,2886,1517],{"class":692},[107,2888,2730],{"class":121},[107,2890,2892],{"class":109,"line":2891},168,[107,2893,616],{"emptyLinePlaceholder":615},[107,2895,2897,2899,2902,2904],{"class":109,"line":2896},169,[107,2898,1679],{"class":692},[107,2900,2901],{"class":171}," severity, table, message ",[107,2903,1113],{"class":692},[107,2905,2869],{"class":171},[107,2907,2909,2911,2913,2915,2917,2919,2922,2924,2926,2928,2930,2932,2934,2937,2939,2941],{"class":109,"line":2908},170,[107,2910,2493],{"class":121},[107,2912,1193],{"class":171},[107,2914,1212],{"class":692},[107,2916,508],{"class":117},[107,2918,1217],{"class":121},[107,2920,2921],{"class":171},"severity.upper()",[107,2923,1223],{"class":121},[107,2925,2172],{"class":121},[107,2927,1230],{"class":171},[107,2929,1223],{"class":121},[107,2931,195],{"class":117},[107,2933,1217],{"class":121},[107,2935,2936],{"class":171},"message",[107,2938,1223],{"class":121},[107,2940,508],{"class":117},[107,2942,768],{"class":171},[107,2944,2946],{"class":109,"line":2945},171,[107,2947,616],{"emptyLinePlaceholder":615},[107,2949,2951,2954,2956,2959,2961,2964,2967,2970,2972,2975,2977,2980,2982,2984,2986,2989],{"class":109,"line":2950},172,[107,2952,2953],{"class":171},"    breaking ",[107,2955,964],{"class":692},[107,2957,2958],{"class":121}," sum",[107,2960,1193],{"class":171},[107,2962,2963],{"class":121},"1",[107,2965,2966],{"class":692}," for",[107,2968,2969],{"class":171}," f ",[107,2971,1113],{"class":692},[107,2973,2974],{"class":171}," findings ",[107,2976,1356],{"class":692},[107,2978,2979],{"class":171}," f[",[107,2981,438],{"class":121},[107,2983,1237],{"class":171},[107,2985,1572],{"class":692},[107,2987,2988],{"class":117}," \"breaking\"",[107,2990,768],{"class":171},[107,2992,2994,2997,2999,3002,3005,3007],{"class":109,"line":2993},173,[107,2995,2996],{"class":171},"    additive ",[107,2998,964],{"class":692},[107,3000,3001],{"class":121}," len",[107,3003,3004],{"class":171},"(findings) ",[107,3006,1834],{"class":692},[107,3008,3009],{"class":171}," breaking\n",[107,3011,3013,3016,3018,3021,3024,3027,3029,3031,3033,3036,3038],{"class":109,"line":3012},174,[107,3014,3015],{"class":171},"    status ",[107,3017,964],{"class":692},[107,3019,3020],{"class":121}," 0",[107,3022,3023],{"class":692}," if",[107,3025,3026],{"class":171}," breaking ",[107,3028,1572],{"class":692},[107,3030,3020],{"class":121},[107,3032,1492],{"class":692},[107,3034,3035],{"class":171}," args.allow_additive ",[107,3037,1376],{"class":692},[107,3039,3040],{"class":121}," 1\n",[107,3042,3044,3047,3049,3051,3053,3055,3057,3059,3062,3064,3066,3068,3071,3073,3076,3078,3081],{"class":109,"line":3043},175,[107,3045,3046],{"class":121},"    print",[107,3048,1193],{"class":171},[107,3050,1212],{"class":692},[107,3052,508],{"class":117},[107,3054,1217],{"class":121},[107,3056,29],{"class":171},[107,3058,1223],{"class":121},[107,3060,3061],{"class":117}," breaking, ",[107,3063,1217],{"class":121},[107,3065,38],{"class":171},[107,3067,1223],{"class":121},[107,3069,3070],{"class":117}," additive change(s). Exit ",[107,3072,1217],{"class":121},[107,3074,3075],{"class":171},"status",[107,3077,1223],{"class":121},[107,3079,3080],{"class":117},".\"",[107,3082,768],{"class":171},[107,3084,3086,3088],{"class":109,"line":3085},176,[107,3087,978],{"class":692},[107,3089,3090],{"class":171}," status\n",[107,3092,3094],{"class":109,"line":3093},177,[107,3095,616],{"emptyLinePlaceholder":615},[107,3097,3099],{"class":109,"line":3098},178,[107,3100,616],{"emptyLinePlaceholder":615},[107,3102,3104,3106,3109,3112,3115],{"class":109,"line":3103},179,[107,3105,1356],{"class":692},[107,3107,3108],{"class":121}," __name__",[107,3110,3111],{"class":692}," ==",[107,3113,3114],{"class":117}," \"__main__\"",[107,3116,1086],{"class":171},[107,3118,3120],{"class":109,"line":3119},180,[107,3121,3122],{"class":171},"    sys.exit(main())\n",[51,3124,3126],{"id":3125},"notes","Notes",[56,3128,3129,3157,3211,3243,3262,3289,3307],{},[59,3130,3131,3138,3139,3141,3142,3145,3146,3149,3150,3152,3153,3156],{},[27,3132,3133,3134,3137],{},"Why ",[15,3135,3136],{},"table_schema"," matters."," Filtering ",[15,3140,77],{}," only on ",[15,3143,3144],{},"table_name"," is a common mistake. With an ",[15,3147,3148],{},"orders"," table in both ",[15,3151,95],{}," and an ",[15,3154,3155],{},"archive"," schema, it merges the two column lists, which can hide drift or invent it. Always filter on both.",[59,3158,3159,3168,3169,3171,3172,3175,3176,3179,3180,3182,3183,3186,3187,3189,3190,3193,3194,3197,3198,3200,3201,3203,3204,3200,3207,3210],{},[27,3160,3133,3161,3164,3165,96],{},[15,3162,3163],{},"udt_name"," as well as ",[15,3166,3167],{},"data_type"," For arrays, ",[15,3170,3167],{}," is just ",[15,3173,3174],{},"ARRAY",", and for enums and other user-defined types it is ",[15,3177,3178],{},"USER-DEFINED",". The real type is in ",[15,3181,3163],{},", and ",[15,3184,3185],{},"udt_schema"," says which schema it lives in, so two enums called ",[15,3188,3075],{}," in different schemas don't compare equal. Without them, changing a column from ",[15,3191,3192],{},"text[]"," to ",[15,3195,3196],{},"integer[]",", or from one enum to another, doesn't show up. For a domain, ",[15,3199,3167],{}," and ",[15,3202,3163],{}," report the underlying type, so the script also compares ",[15,3205,3206],{},"domain_schema",[15,3208,3209],{},"domain_name",": moving a column between two domains over the same base type is a type change too.",[59,3212,3213,3216,3217,3220,3221,3224,3225,3228,3229,3231,3232,3235,3236,3239,3240,96],{},[27,3214,3215],{},"Precision beyond length."," ",[15,3218,3219],{},"datetime_precision"," catches ",[15,3222,3223],{},"timestamp(6)"," becoming ",[15,3226,3227],{},"timestamp(0)"," (same ",[15,3230,3163],{},", fewer fractional digits), and ",[15,3233,3234],{},"interval_type"," catches an ",[15,3237,3238],{},"interval"," column restricted to, say, ",[15,3241,3242],{},"DAY TO SECOND",[59,3244,3245,3216,3248,3251,3252,3200,3255,3257,3258,3261],{},[27,3246,3247],{},"Length is where silent truncation lives.",[15,3249,3250],{},"character_maximum_length"," is null for unbounded ",[15,3253,3254],{},"varchar",[15,3256,572],{},", and set for ",[15,3259,3260],{},"varchar(n)",". A source column widened from 64 to 255 is harmless to read, but a destination still declared at 64 will reject the longer values. That's why the script flags it: the next step is to check the destination, not to suppress the warning.",[59,3263,3264,3216,3267,3270,3271,3193,3274,3277,3278,3281,3282,3284,3285,3288],{},[27,3265,3266],{},"Nullability direction.",[15,3268,3269],{},"is_nullable"," going from ",[15,3272,3273],{},"NO",[15,3275,3276],{},"YES"," is breaking because the source can now send nulls into a destination column declared ",[15,3279,3280],{},"NOT NULL",", or into metrics that assumed a value. The reverse is harmless for the load. Keep in mind that Postgres reports ",[15,3283,3276],{}," whenever a column is ",[32,3286,3287],{},"possibly"," nullable.",[59,3290,3291,3294,3295,3298,3299,3302,3303,3306],{},[27,3292,3293],{},"What it doesn't check."," Defaults, constraints, indexes, and whether a column's ",[32,3296,3297],{},"meaning"," changed. A status column that gains a new value passes this check; data tests (dbt's ",[15,3300,3301],{},"accepted_values",", or a plain ",[15,3304,3305],{},"SELECT DISTINCT"," assertion) cover that.",[59,3308,3309,3312,3313,3315],{},[27,3310,3311],{},"Treat the baseline like a migration."," Update it deliberately, in the same commit as the pipeline change that expects the new shape, not as a reflex when the check goes red. Re-running ",[15,3314,84],{}," to make a failing check pass throws away exactly the information the check exists to surface.",[51,3317,3319],{"id":3318},"references","References",[56,3321,3322,3330,3337,3344],{},[59,3323,3324],{},[41,3325,3329],{"href":3326,"rel":3327},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Finfoschema-columns.html",[3328],"nofollow","PostgreSQL: information_schema.columns",[59,3331,3332],{},[41,3333,3336],{"href":3334,"rel":3335},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Finformation-schema.html",[3328],"PostgreSQL: the information schema",[59,3338,3339],{},[41,3340,3343],{"href":3341,"rel":3342},"https:\u002F\u002Fwww.psycopg.org\u002Fdocs\u002Fusage.html",[3328],"psycopg2: basic module usage and connection context managers",[59,3345,3346],{},[41,3347,3350],{"href":3348,"rel":3349},"https:\u002F\u002Fdocs.getdbt.com\u002Fdocs\u002Fbuild\u002Fdata-tests",[3328],"dbt: data tests (unique, not_null, accepted_values, relationships)",[3352,3353,3354],"style",{},"html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}",{"title":103,"searchDepth":128,"depth":128,"links":3356},[3357,3358,3359,3360,3361],{"id":53,"depth":128,"text":54},{"id":88,"depth":128,"text":89},{"id":587,"depth":128,"text":588},{"id":3125,"depth":128,"text":3126},{"id":3318,"depth":128,"text":3319},"techcolumnist",[3364,3365],"scripts","engineering","2026-01-21T14:00:00Z","A Python pre-flight check that snapshots Postgres column definitions from information_schema and fails the ETL run when a column is dropped, retyped, or resized.","md",false,null,{},"\u002Fblog\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run",{"title":6,"description":3367},[3362],"blog\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run",[114,3377,3378,3379],"data-warehouse","etl","sql","\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run\u002F","TmTydnKmQD8eGHOhZoDRLlMRbPaDKD8OaT3HVxFl9D0",{"title":3383,"description":3384,"date":3385,"url":3386,"categories":3387,"tags":3388,"image":3370,"readingTime":139,"canonical":3362,"sites":3392,"series":3370,"seriesOrder":3370},"Python: Discovery – Crawling DNS Zones for Orphaned Records","A dnspython script that transfers a zone with AXFR, probes every A\u002FAAAA record for signs of life, flags dangling CNAMEs, and reports orphan cleanup candidates.","2026-01-28T14:00:00Z","\u002F2026\u002F01\u002F28\u002Fpython-discovery-crawling-dns-zones-for-orphaned-records\u002F",[3364,3365],[114,3389,3390,3391],"discovery","network","windows",[3362],{"title":3394,"description":3395,"date":3396,"url":3397,"categories":3398,"tags":3399,"image":3370,"readingTime":150,"canonical":3362,"sites":3401,"series":3370,"seriesOrder":3370},"Python: gcloud – Audit IAM Bindings Across a GCP Organization","A Python script that pulls the IAM bindings across a GCP organization through Cloud Asset Inventory and flags basic roles, public access, and external identities.","2026-01-14T14:00:00Z","\u002F2026\u002F01\u002F14\u002Fpython-gcloud-audit-iam-bindings-across-a-gcp-organization\u002F",[3364,3365],[114,3400],"gcloud",[3362],[3403,3410,3419],{"title":3404,"description":3405,"date":3406,"url":43,"categories":3407,"tags":3408,"image":3370,"readingTime":204,"canonical":3362,"sites":3409,"series":3370,"seriesOrder":3370},"Python: Data Warehouse – Incremental Loads from Postgres to BigQuery","A Python script that copies new and changed Postgres rows into BigQuery with one load job and a MERGE, advancing the watermark in the same transaction.","2025-12-17T14:00:00Z",[3364,3365],[114,3377,3378,3400,3379],[3362],{"title":3411,"description":3412,"date":3413,"url":3414,"categories":3415,"tags":3416,"image":3370,"readingTime":204,"canonical":3362,"sites":3418,"series":3370,"seriesOrder":3370},"Node.js: Data Warehouse – Streaming ETL Rows Without Buffering Them","A Node.js script that streams Postgres rows through a cursor, a transform and COPY FROM STDIN with backpressure, so memory tracks batch size, not table size.","2026-02-25T14:00:00Z","\u002F2026\u002F02\u002F25\u002Fnode-js-data-warehouse-streaming-etl-rows-without-loading-them-all-in-memory\u002F",[3364,3365],[3417,3377,3378,3379],"nodejs",[3362],{"title":3420,"description":3421,"date":3422,"url":3423,"categories":3424,"tags":3426,"image":3370,"readingTime":242,"canonical":3362,"sites":3427,"series":3370,"seriesOrder":3370},"Data Warehouse: When to Stop Querying Production and Build a Warehouse","How to measure reporting load on a production Postgres database, the cheap fixes to try first, and the signals that mean a separate warehouse is finally justified.","2026-05-27T14:00:00Z","\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse\u002F",[3365,3425],"strategy",[3377,3379,114],[3362],{"doc":3370,"posts":3429},[],1790052513410]