[{"data":1,"prerenderedAt":3356},["ShallowReactive",2],{"post:\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team\u002F":3},{"post":4,"newer":3320,"older":3327,"related":3336,"series":3354},{"id":5,"title":6,"body":7,"canonical":3301,"categories":3302,"date":3305,"description":3306,"extension":3307,"featured":3308,"hero":3309,"image":3309,"meta":3310,"navigation":199,"path":3311,"readingTime":226,"seo":3312,"series":3309,"seriesOrder":3309,"sites":3313,"source":3309,"stem":3314,"tags":3315,"updated":3309,"url":3318,"__hash__":3319},"blog\u002Fblog\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team.md","Data Warehouse: Building a Reporting Layer Without a BI Team",{"type":8,"value":9,"toc":3293},"minimark",[10,14,17,22,25,28,32,35,38,124,477,486,490,493,496,521,557,748,2039,2042,2251,2263,2458,2462,2465,2468,2609,2612,2920,2927,3079,3107,3186,3193,3213,3217,3225,3229,3289],[11,12,13],"p",{},"Every infrastructure team eventually piles up data it can't answer questions with. Ticket volumes live in the PSA, uptime lives in the monitoring platform, change records live in a spreadsheet someone started three years ago and never stopped using. Leadership asks a reasonable question, like \"which client's environment generates the most after-hours tickets,\" and the honest answer is that nobody can produce it without a day of manual exporting and VLOOKUPs. A BI team would fix this properly. Most IT organizations I've worked with don't have one and won't get one budgeted anytime soon. The reporting layer still has to get built, just by whoever's already in the room.",[11,15,16],{},"This post is the build I'd reach for in that situation, with the actual SQL and the loader script. Everything runs on one PostgreSQL database and a scheduler you already have.",[18,19,21],"h2",{"id":20},"start-with-one-source-of-truth-not-five","Start with one source of truth, not five",[11,23,24],{},"The instinct when you finally get permission to build something is to connect everything at once (the PSA, the RMM, the monitoring tool, the identity provider) and design a grand unified model. That's how these projects die before they ship anything. I've had better luck picking the single dataset that answers the most urgent recurring question, building a small, ugly, correct pipeline for just that, and shipping a report people actually use within the first month. Momentum and trust come from something working, not from an architecture diagram.",[11,26,27],{},"For most IT operations teams that first dataset is tickets. It answers volume, response and resolution time, and backlog questions, and every PSA has some kind of API for it.",[18,29,31],{"id":30},"pick-boring-tools-on-purpose","Pick boring tools on purpose",[11,33,34],{},"Without a BI team, there's no one to maintain a Snowflake instance with a dbt pipeline feeding a Looker deployment. That's fine, because that stack solves a scale problem a shop this size doesn't have yet. A single Postgres database (or a managed one: Cloud SQL, Azure Database for PostgreSQL, RDS), a few scheduled scripts doing extract-and-load, and a reporting tool the organization already licenses will carry an IT operations reporting need further than the fancy version. It's something the same one or two people who built it can actually keep running.",[11,36,37],{},"The one structural decision worth making up front is three schemas with three roles, so nobody has to remember which tables are safe to point a dashboard at:",[39,40,41,60],"table",{},[42,43,44],"thead",{},[45,46,47,51,54,57],"tr",{},[48,49,50],"th",{},"Schema",[48,52,53],{},"Holds",[48,55,56],{},"Written by",[48,58,59],{},"Read by",[61,62,63,85,105],"tbody",{},[45,64,65,72,75,80],{},[66,67,68],"td",{},[69,70,71],"code",{},"raw",[66,73,74],{},"API responses as landed, one row per source record, JSON payload intact",[66,76,77],{},[69,78,79],{},"etl_loader",[66,81,82],{},[69,83,84],{},"transformer",[45,86,87,92,97,101],{},[66,88,89],{},[69,90,91],{},"staging",[66,93,94,95],{},"Views that pull typed, cleaned columns out of ",[69,96,71],{},[66,98,99],{},[69,100,84],{},[66,102,103],{},[69,104,84],{},[45,106,107,112,115,119],{},[66,108,109],{},[69,110,111],{},"marts",[66,113,114],{},"Metric views and tables the dashboards use",[66,116,117],{},[69,118,84],{},[66,120,121],{},[69,122,123],{},"report_reader",[125,126,131],"pre",{"className":127,"code":128,"language":129,"meta":130,"style":130},"language-sql shiki shiki-themes github-dark","CREATE SCHEMA IF NOT EXISTS raw;\nCREATE SCHEMA IF NOT EXISTS staging;\nCREATE SCHEMA IF NOT EXISTS marts;\n\nCREATE ROLE etl_loader    LOGIN PASSWORD '\u003Cpassword>';\nCREATE ROLE transformer   LOGIN PASSWORD '\u003Cpassword>';\nCREATE ROLE report_reader LOGIN PASSWORD '\u003Cpassword>';\n\n-- loader owns raw\nGRANT USAGE, CREATE ON SCHEMA raw TO etl_loader;\n\n-- transformer reads raw, builds staging and marts\nGRANT USAGE ON SCHEMA raw TO transformer;\nALTER DEFAULT PRIVILEGES FOR ROLE etl_loader IN SCHEMA raw GRANT SELECT ON TABLES TO transformer;\nGRANT USAGE, CREATE ON SCHEMA staging, marts TO transformer;\n\n-- dashboards see marts only, including views created later\nGRANT USAGE ON SCHEMA marts TO report_reader;\nALTER DEFAULT PRIVILEGES FOR ROLE transformer IN SCHEMA marts GRANT SELECT ON TABLES TO report_reader;\nALTER ROLE report_reader SET statement_timeout = '60s';\n","sql","",[69,132,133,162,178,194,201,224,242,260,265,272,296,301,307,327,370,390,395,401,420,454],{"__ignoreMap":130},[134,135,138,142,145,149,152,155,158],"span",{"class":136,"line":137},"line",1,[134,139,141],{"class":140},"snl16","CREATE",[134,143,144],{"class":140}," SCHEMA",[134,146,148],{"class":147},"svObZ"," IF",[134,150,151],{"class":140}," NOT",[134,153,154],{"class":140}," EXISTS",[134,156,157],{"class":140}," raw",[134,159,161],{"class":160},"s95oV",";\n",[134,163,165,167,169,171,173,175],{"class":136,"line":164},2,[134,166,141],{"class":140},[134,168,144],{"class":140},[134,170,148],{"class":147},[134,172,151],{"class":140},[134,174,154],{"class":140},[134,176,177],{"class":160}," staging;\n",[134,179,181,183,185,187,189,191],{"class":136,"line":180},3,[134,182,141],{"class":140},[134,184,144],{"class":140},[134,186,148],{"class":147},[134,188,151],{"class":140},[134,190,154],{"class":140},[134,192,193],{"class":160}," marts;\n",[134,195,197],{"class":136,"line":196},4,[134,198,200],{"emptyLinePlaceholder":199},true,"\n",[134,202,204,206,209,212,215,218,222],{"class":136,"line":203},5,[134,205,141],{"class":140},[134,207,208],{"class":140}," ROLE",[134,210,211],{"class":160}," etl_loader    ",[134,213,214],{"class":140},"LOGIN",[134,216,217],{"class":140}," PASSWORD",[134,219,221],{"class":220},"sU2Wk"," '\u003Cpassword>'",[134,223,161],{"class":160},[134,225,227,229,231,234,236,238,240],{"class":136,"line":226},6,[134,228,141],{"class":140},[134,230,208],{"class":140},[134,232,233],{"class":160}," transformer   ",[134,235,214],{"class":140},[134,237,217],{"class":140},[134,239,221],{"class":220},[134,241,161],{"class":160},[134,243,245,247,249,252,254,256,258],{"class":136,"line":244},7,[134,246,141],{"class":140},[134,248,208],{"class":140},[134,250,251],{"class":160}," report_reader ",[134,253,214],{"class":140},[134,255,217],{"class":140},[134,257,221],{"class":220},[134,259,161],{"class":160},[134,261,263],{"class":136,"line":262},8,[134,264,200],{"emptyLinePlaceholder":199},[134,266,268],{"class":136,"line":267},9,[134,269,271],{"class":270},"sAwPA","-- loader owns raw\n",[134,273,275,278,281,283,286,288,290,293],{"class":136,"line":274},10,[134,276,277],{"class":140},"GRANT",[134,279,280],{"class":160}," USAGE, ",[134,282,141],{"class":140},[134,284,285],{"class":140}," ON",[134,287,144],{"class":140},[134,289,157],{"class":140},[134,291,292],{"class":140}," TO",[134,294,295],{"class":160}," etl_loader;\n",[134,297,299],{"class":136,"line":298},11,[134,300,200],{"emptyLinePlaceholder":199},[134,302,304],{"class":136,"line":303},12,[134,305,306],{"class":270},"-- transformer reads raw, builds staging and marts\n",[134,308,310,312,315,318,320,322,324],{"class":136,"line":309},13,[134,311,277],{"class":140},[134,313,314],{"class":160}," USAGE ",[134,316,317],{"class":140},"ON",[134,319,144],{"class":140},[134,321,157],{"class":140},[134,323,292],{"class":140},[134,325,326],{"class":160}," transformer;\n",[134,328,330,333,336,339,342,344,347,350,352,354,357,360,362,365,368],{"class":136,"line":329},14,[134,331,332],{"class":140},"ALTER",[134,334,335],{"class":140}," DEFAULT",[134,337,338],{"class":160}," PRIVILEGES ",[134,340,341],{"class":140},"FOR",[134,343,208],{"class":140},[134,345,346],{"class":160}," etl_loader ",[134,348,349],{"class":140},"IN",[134,351,144],{"class":140},[134,353,157],{"class":140},[134,355,356],{"class":140}," GRANT",[134,358,359],{"class":140}," SELECT",[134,361,285],{"class":140},[134,363,364],{"class":160}," TABLES ",[134,366,367],{"class":140},"TO",[134,369,326],{"class":160},[134,371,373,375,377,379,381,383,386,388],{"class":136,"line":372},15,[134,374,277],{"class":140},[134,376,280],{"class":160},[134,378,141],{"class":140},[134,380,285],{"class":140},[134,382,144],{"class":140},[134,384,385],{"class":160}," staging, marts ",[134,387,367],{"class":140},[134,389,326],{"class":160},[134,391,393],{"class":136,"line":392},16,[134,394,200],{"emptyLinePlaceholder":199},[134,396,398],{"class":136,"line":397},17,[134,399,400],{"class":270},"-- dashboards see marts only, including views created later\n",[134,402,404,406,408,410,412,415,417],{"class":136,"line":403},18,[134,405,277],{"class":140},[134,407,314],{"class":160},[134,409,317],{"class":140},[134,411,144],{"class":140},[134,413,414],{"class":160}," marts ",[134,416,367],{"class":140},[134,418,419],{"class":160}," report_reader;\n",[134,421,423,425,427,429,431,433,436,438,440,442,444,446,448,450,452],{"class":136,"line":422},19,[134,424,332],{"class":140},[134,426,335],{"class":140},[134,428,338],{"class":160},[134,430,341],{"class":140},[134,432,208],{"class":140},[134,434,435],{"class":160}," transformer ",[134,437,349],{"class":140},[134,439,144],{"class":140},[134,441,414],{"class":160},[134,443,277],{"class":140},[134,445,359],{"class":140},[134,447,285],{"class":140},[134,449,364],{"class":160},[134,451,367],{"class":140},[134,453,419],{"class":160},[134,455,457,459,461,463,466,469,472,475],{"class":136,"line":456},20,[134,458,332],{"class":140},[134,460,208],{"class":140},[134,462,251],{"class":160},[134,464,465],{"class":140},"SET",[134,467,468],{"class":160}," statement_timeout ",[134,470,471],{"class":140},"=",[134,473,474],{"class":220}," '60s'",[134,476,161],{"class":160},[11,478,479,482,483,485],{},[69,480,481],{},"ALTER DEFAULT PRIVILEGES ... FOR ROLE"," applies to objects that role creates in the future. Without it, every new view is invisible to the dashboard account until someone remembers to ",[69,484,277],{}," it, which is exactly the kind of tribal knowledge that doesn't survive a handoff. Default privileges don't reach back to existing objects, so grant those once by hand.",[18,487,489],{"id":488},"the-extract-step-is-where-most-of-the-effort-goes","The extract step is where most of the effort goes",[11,491,492],{},"Nobody warns you that the actual warehouse part, a database with some fact tables, is the easy part. The hard part is getting clean, consistent data out of tools that were never designed to be queried by anything other than their own UI. PSA and RMM APIs paginate inconsistently, rate-limit without clear documentation, and return null in places their own UI quietly fills in a default. Budget twice the time you think you need for extraction and normalization, and assume every source has at least one field that means something different than its name suggests.",[11,494,495],{},"Two habits make that manageable:",[497,498,499,515],"ol",{},[500,501,502,506,507,510,511,514],"li",{},[503,504,505],"strong",{},"Land the raw payload untouched."," Store each record's full JSON in a ",[69,508,509],{},"jsonb"," column and pull typed columns out later in a view. When you discover the ",[69,512,513],{},"closed_date"," field means something odd, you fix a view, not a reload of two years of history.",[500,516,517,520],{},[503,518,519],{},"Upsert on the source ID."," Re-running the loader, or overlapping two runs, should never create duplicates.",[11,522,523,524,527,528,531,532,535,536,539,540,544,545,548,549,552,553,556],{},"The loader below does both. It's written against a generic REST API that returns a JSON array per page and supports an ",[69,525,526],{},"updated since"," filter, and it follows an RFC 8288 ",[69,529,530],{},"Link: \u003C...>; rel=\"next\""," header for pagination. That's the part you adapt per vendor. It honors ",[69,533,534],{},"429 Too Many Requests"," with a ",[69,537,538],{},"Retry-After"," header given in seconds, falling back to exponential backoff otherwise (RFC 6585 says servers ",[541,542,543],"em",{},"may"," send one), and it writes a row to ",[69,546,547],{},"raw.etl_runs"," so the pipeline's health is queryable. It needs Python 3.10+, ",[69,550,551],{},"requests",", and ",[69,554,555],{},"psycopg2-binary",".",[125,558,560],{"className":127,"code":559,"language":129,"meta":130,"style":130},"CREATE TABLE IF NOT EXISTS raw.psa_tickets (\n    source_id  text PRIMARY KEY,\n    payload    jsonb NOT NULL,\n    loaded_at  timestamptz NOT NULL DEFAULT now()\n);\n\nCREATE TABLE IF NOT EXISTS raw.etl_runs (\n    run_id      bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,\n    source      text NOT NULL,\n    started_at  timestamptz NOT NULL,\n    finished_at timestamptz,\n    status      text NOT NULL,\n    rows_loaded integer,\n    error       text\n);\n",[69,561,562,586,600,610,629,634,638,659,683,694,705,714,726,736,744],{"__ignoreMap":130},[134,563,564,566,569,571,573,575,578,580,583],{"class":136,"line":137},[134,565,141],{"class":140},[134,567,568],{"class":140}," TABLE",[134,570,148],{"class":147},[134,572,151],{"class":140},[134,574,154],{"class":140},[134,576,157],{"class":577},"sDLfK",[134,579,556],{"class":160},[134,581,582],{"class":577},"psa_tickets",[134,584,585],{"class":160}," (\n",[134,587,588,591,594,597],{"class":136,"line":164},[134,589,590],{"class":160},"    source_id  ",[134,592,593],{"class":140},"text",[134,595,596],{"class":140}," PRIMARY KEY",[134,598,599],{"class":160},",\n",[134,601,602,605,608],{"class":136,"line":180},[134,603,604],{"class":160},"    payload    jsonb ",[134,606,607],{"class":140},"NOT NULL",[134,609,599],{"class":160},[134,611,612,615,618,621,623,626],{"class":136,"line":196},[134,613,614],{"class":160},"    loaded_at  ",[134,616,617],{"class":140},"timestamptz",[134,619,620],{"class":140}," NOT NULL",[134,622,335],{"class":140},[134,624,625],{"class":140}," now",[134,627,628],{"class":160},"()\n",[134,630,631],{"class":136,"line":203},[134,632,633],{"class":160},");\n",[134,635,636],{"class":136,"line":226},[134,637,200],{"emptyLinePlaceholder":199},[134,639,640,642,644,646,648,650,652,654,657],{"class":136,"line":244},[134,641,141],{"class":140},[134,643,568],{"class":140},[134,645,148],{"class":147},[134,647,151],{"class":140},[134,649,154],{"class":140},[134,651,157],{"class":577},[134,653,556],{"class":160},[134,655,656],{"class":577},"etl_runs",[134,658,585],{"class":160},[134,660,661,664,667,670,673,676,679,681],{"class":136,"line":262},[134,662,663],{"class":160},"    run_id      ",[134,665,666],{"class":140},"bigint",[134,668,669],{"class":140}," GENERATED",[134,671,672],{"class":140}," ALWAYS",[134,674,675],{"class":140}," AS",[134,677,678],{"class":140}," IDENTITY",[134,680,596],{"class":140},[134,682,599],{"class":160},[134,684,685,688,690,692],{"class":136,"line":267},[134,686,687],{"class":160},"    source      ",[134,689,593],{"class":140},[134,691,620],{"class":140},[134,693,599],{"class":160},[134,695,696,699,701,703],{"class":136,"line":274},[134,697,698],{"class":160},"    started_at  ",[134,700,617],{"class":140},[134,702,620],{"class":140},[134,704,599],{"class":160},[134,706,707,710,712],{"class":136,"line":298},[134,708,709],{"class":160},"    finished_at ",[134,711,617],{"class":140},[134,713,599],{"class":160},[134,715,716,719,722,724],{"class":136,"line":303},[134,717,718],{"class":140},"    status",[134,720,721],{"class":140},"      text",[134,723,620],{"class":140},[134,725,599],{"class":160},[134,727,728,731,734],{"class":136,"line":309},[134,729,730],{"class":160},"    rows_loaded ",[134,732,733],{"class":140},"integer",[134,735,599],{"class":160},[134,737,738,741],{"class":136,"line":329},[134,739,740],{"class":160},"    error       ",[134,742,743],{"class":140},"text\n",[134,745,746],{"class":136,"line":372},[134,747,633],{"class":160},[125,749,753],{"className":750,"code":751,"language":752,"meta":130,"style":130},"language-python shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"\nload_api_to_raw.py\n\nPulls records changed since the last successful run from a paginated REST API\nand upserts them into a raw jsonb landing table in PostgreSQL. Records every\nrun (success or failure) in raw.etl_runs.\n\nUsage:\n    PSA_TOKEN=... python load_api_to_raw.py --dsn \u003Cdsn> --source psa_tickets \\\n        --url https:\u002F\u002Fpsa.example.com\u002Fapi\u002Fv1\u002Ftickets --since-param updatedSince\nReads:  the API at --url (bearer token from PSA_TOKEN); raw.etl_runs.\nWrites: raw.\u003Csource> (upsert on source_id); one row in raw.etl_runs.\n\"\"\"\n\nimport argparse\nimport datetime as dt\nimport os\nimport sys\nimport time\nfrom urllib.parse import urljoin\n\nimport psycopg2\nimport requests\nfrom psycopg2 import sql\nfrom psycopg2.extras import Json, execute_values\n\nMAX_RETRIES = 5\n\n\ndef last_success(cur, source):\n    cur.execute(\n        \"SELECT max(started_at) FROM raw.etl_runs WHERE source = %s AND status = 'success'\",\n        (source,),\n    )\n    return cur.fetchone()[0]\n\n\ndef fetch_pages(session, url, params):\n    \"\"\"Yield one page (a list of records) at a time, following Link rel=next.\"\"\"\n    while url:\n        for attempt in range(MAX_RETRIES):\n            response = session.get(url, params=params, timeout=60)\n            if response.status_code != 429:\n                break\n            retry_after = response.headers.get(\"Retry-After\", \"\")\n            time.sleep(int(retry_after) if retry_after.isdigit() else 2 ** attempt)\n        response.raise_for_status()\n        yield response.json()\n        next_url = response.links.get(\"next\", {}).get(\"url\")\n        url = urljoin(response.url, next_url) if next_url else None  # RFC 8288 allows relative links\n        params = None  # the next link already carries the query string\n\n\ndef main() -> int:\n    parser = argparse.ArgumentParser(description=\"Load changed API records into a raw jsonb table.\")\n    parser.add_argument(\"--dsn\", required=True)\n    parser.add_argument(\"--source\", required=True, help=\"Raw table name, e.g. psa_tickets\")\n    parser.add_argument(\"--url\", required=True)\n    parser.add_argument(\"--since-param\", default=\"updatedSince\")\n    parser.add_argument(\"--id-field\", default=\"id\")\n    parser.add_argument(\"--overlap-minutes\", type=int, default=15)\n    args = parser.parse_args()\n\n    session = requests.Session()\n    session.headers[\"Authorization\"] = f\"Bearer {os.environ['PSA_TOKEN']}\"\n    started = dt.datetime.now(dt.timezone.utc)\n    upsert = sql.SQL(\n        \"INSERT INTO {table} (source_id, payload) VALUES %s \"\n        \"ON CONFLICT (source_id) DO UPDATE SET payload = EXCLUDED.payload, loaded_at = now()\"\n    ).format(table=sql.Identifier(\"raw\", args.source))\n\n    conn = psycopg2.connect(args.dsn)\n    conn.autocommit = True\n    rows = 0\n    try:\n        with conn.cursor() as cur:\n            since = last_success(cur, args.source)\n            params = {}\n            if since:\n                params[args.since_param] = (since - dt.timedelta(minutes=args.overlap_minutes)).isoformat()\n            for page in fetch_pages(session, args.url, params):\n                values = [(str(record[args.id_field]), Json(record)) for record in page]\n                if values:\n                    execute_values(cur, upsert.as_string(conn), values, page_size=500)\n                    rows += len(values)\n            cur.execute(\n                \"INSERT INTO raw.etl_runs (source, started_at, finished_at, status, rows_loaded) \"\n                \"VALUES (%s, %s, now(), 'success', %s)\",\n                (args.source, started, rows),\n            )\n        print(f\"{args.source}: {rows} records upserted\")\n        return 0\n    except Exception as error:\n        with conn.cursor() as cur:\n            cur.execute(\n                \"INSERT INTO raw.etl_runs (source, started_at, finished_at, status, rows_loaded, error) \"\n                \"VALUES (%s, %s, now(), 'failed', %s, %s)\",\n                (args.source, started, rows, str(error)[:2000]),\n            )\n        print(f\"{args.source}: failed after {rows} records: {error}\", file=sys.stderr)\n        return 1\n    finally:\n        conn.close()\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n","python",[69,754,755,760,765,770,774,779,784,789,793,798,806,811,816,821,825,829,837,850,857,864,871,885,890,898,906,919,932,937,949,954,959,971,977,991,997,1003,1018,1023,1028,1039,1045,1054,1077,1108,1126,1132,1154,1184,1190,1199,1221,1245,1258,1263,1268,1283,1304,1325,1353,1371,1391,1410,1438,1449,1454,1465,1503,1514,1525,1542,1548,1567,1572,1583,1594,1605,1613,1627,1638,1649,1657,1682,1696,1724,1733,1749,1764,1770,1776,1798,1804,1810,1846,1854,1869,1880,1885,1891,1915,1932,1937,1985,1993,2001,2007,2012,2017,2033],{"__ignoreMap":130},[134,756,757],{"class":136,"line":137},[134,758,759],{"class":270},"#!\u002Fusr\u002Fbin\u002Fenv python3\n",[134,761,762],{"class":136,"line":164},[134,763,764],{"class":220},"\"\"\"\n",[134,766,767],{"class":136,"line":180},[134,768,769],{"class":220},"load_api_to_raw.py\n",[134,771,772],{"class":136,"line":196},[134,773,200],{"emptyLinePlaceholder":199},[134,775,776],{"class":136,"line":203},[134,777,778],{"class":220},"Pulls records changed since the last successful run from a paginated REST API\n",[134,780,781],{"class":136,"line":226},[134,782,783],{"class":220},"and upserts them into a raw jsonb landing table in PostgreSQL. Records every\n",[134,785,786],{"class":136,"line":244},[134,787,788],{"class":220},"run (success or failure) in raw.etl_runs.\n",[134,790,791],{"class":136,"line":262},[134,792,200],{"emptyLinePlaceholder":199},[134,794,795],{"class":136,"line":267},[134,796,797],{"class":220},"Usage:\n",[134,799,800,803],{"class":136,"line":274},[134,801,802],{"class":220},"    PSA_TOKEN=... python load_api_to_raw.py --dsn \u003Cdsn> --source psa_tickets ",[134,804,805],{"class":577},"\\\n",[134,807,808],{"class":136,"line":298},[134,809,810],{"class":220},"        --url https:\u002F\u002Fpsa.example.com\u002Fapi\u002Fv1\u002Ftickets --since-param updatedSince\n",[134,812,813],{"class":136,"line":303},[134,814,815],{"class":220},"Reads:  the API at --url (bearer token from PSA_TOKEN); raw.etl_runs.\n",[134,817,818],{"class":136,"line":309},[134,819,820],{"class":220},"Writes: raw.\u003Csource> (upsert on source_id); one row in raw.etl_runs.\n",[134,822,823],{"class":136,"line":329},[134,824,764],{"class":220},[134,826,827],{"class":136,"line":372},[134,828,200],{"emptyLinePlaceholder":199},[134,830,831,834],{"class":136,"line":392},[134,832,833],{"class":140},"import",[134,835,836],{"class":160}," argparse\n",[134,838,839,841,844,847],{"class":136,"line":397},[134,840,833],{"class":140},[134,842,843],{"class":160}," datetime ",[134,845,846],{"class":140},"as",[134,848,849],{"class":160}," dt\n",[134,851,852,854],{"class":136,"line":403},[134,853,833],{"class":140},[134,855,856],{"class":160}," os\n",[134,858,859,861],{"class":136,"line":422},[134,860,833],{"class":140},[134,862,863],{"class":160}," sys\n",[134,865,866,868],{"class":136,"line":456},[134,867,833],{"class":140},[134,869,870],{"class":160}," time\n",[134,872,874,877,880,882],{"class":136,"line":873},21,[134,875,876],{"class":140},"from",[134,878,879],{"class":160}," urllib.parse ",[134,881,833],{"class":140},[134,883,884],{"class":160}," urljoin\n",[134,886,888],{"class":136,"line":887},22,[134,889,200],{"emptyLinePlaceholder":199},[134,891,893,895],{"class":136,"line":892},23,[134,894,833],{"class":140},[134,896,897],{"class":160}," psycopg2\n",[134,899,901,903],{"class":136,"line":900},24,[134,902,833],{"class":140},[134,904,905],{"class":160}," requests\n",[134,907,909,911,914,916],{"class":136,"line":908},25,[134,910,876],{"class":140},[134,912,913],{"class":160}," psycopg2 ",[134,915,833],{"class":140},[134,917,918],{"class":160}," sql\n",[134,920,922,924,927,929],{"class":136,"line":921},26,[134,923,876],{"class":140},[134,925,926],{"class":160}," psycopg2.extras ",[134,928,833],{"class":140},[134,930,931],{"class":160}," Json, execute_values\n",[134,933,935],{"class":136,"line":934},27,[134,936,200],{"emptyLinePlaceholder":199},[134,938,940,943,946],{"class":136,"line":939},28,[134,941,942],{"class":577},"MAX_RETRIES",[134,944,945],{"class":140}," =",[134,947,948],{"class":577}," 5\n",[134,950,952],{"class":136,"line":951},29,[134,953,200],{"emptyLinePlaceholder":199},[134,955,957],{"class":136,"line":956},30,[134,958,200],{"emptyLinePlaceholder":199},[134,960,962,965,968],{"class":136,"line":961},31,[134,963,964],{"class":140},"def",[134,966,967],{"class":147}," last_success",[134,969,970],{"class":160},"(cur, source):\n",[134,972,974],{"class":136,"line":973},32,[134,975,976],{"class":160},"    cur.execute(\n",[134,978,980,983,986,989],{"class":136,"line":979},33,[134,981,982],{"class":220},"        \"SELECT max(started_at) FROM raw.etl_runs WHERE source = ",[134,984,985],{"class":577},"%s",[134,987,988],{"class":220}," AND status = 'success'\"",[134,990,599],{"class":160},[134,992,994],{"class":136,"line":993},34,[134,995,996],{"class":160},"        (source,),\n",[134,998,1000],{"class":136,"line":999},35,[134,1001,1002],{"class":160},"    )\n",[134,1004,1006,1009,1012,1015],{"class":136,"line":1005},36,[134,1007,1008],{"class":140},"    return",[134,1010,1011],{"class":160}," cur.fetchone()[",[134,1013,1014],{"class":577},"0",[134,1016,1017],{"class":160},"]\n",[134,1019,1021],{"class":136,"line":1020},37,[134,1022,200],{"emptyLinePlaceholder":199},[134,1024,1026],{"class":136,"line":1025},38,[134,1027,200],{"emptyLinePlaceholder":199},[134,1029,1031,1033,1036],{"class":136,"line":1030},39,[134,1032,964],{"class":140},[134,1034,1035],{"class":147}," fetch_pages",[134,1037,1038],{"class":160},"(session, url, params):\n",[134,1040,1042],{"class":136,"line":1041},40,[134,1043,1044],{"class":220},"    \"\"\"Yield one page (a list of records) at a time, following Link rel=next.\"\"\"\n",[134,1046,1048,1051],{"class":136,"line":1047},41,[134,1049,1050],{"class":140},"    while",[134,1052,1053],{"class":160}," url:\n",[134,1055,1057,1060,1063,1066,1069,1072,1074],{"class":136,"line":1056},42,[134,1058,1059],{"class":140},"        for",[134,1061,1062],{"class":160}," attempt ",[134,1064,1065],{"class":140},"in",[134,1067,1068],{"class":577}," range",[134,1070,1071],{"class":160},"(",[134,1073,942],{"class":577},[134,1075,1076],{"class":160},"):\n",[134,1078,1080,1083,1085,1088,1092,1094,1097,1100,1102,1105],{"class":136,"line":1079},43,[134,1081,1082],{"class":160},"            response ",[134,1084,471],{"class":140},[134,1086,1087],{"class":160}," session.get(url, ",[134,1089,1091],{"class":1090},"s9osk","params",[134,1093,471],{"class":140},[134,1095,1096],{"class":160},"params, ",[134,1098,1099],{"class":1090},"timeout",[134,1101,471],{"class":140},[134,1103,1104],{"class":577},"60",[134,1106,1107],{"class":160},")\n",[134,1109,1111,1114,1117,1120,1123],{"class":136,"line":1110},44,[134,1112,1113],{"class":140},"            if",[134,1115,1116],{"class":160}," response.status_code ",[134,1118,1119],{"class":140},"!=",[134,1121,1122],{"class":577}," 429",[134,1124,1125],{"class":160},":\n",[134,1127,1129],{"class":136,"line":1128},45,[134,1130,1131],{"class":140},"                break\n",[134,1133,1135,1138,1140,1143,1146,1149,1152],{"class":136,"line":1134},46,[134,1136,1137],{"class":160},"            retry_after ",[134,1139,471],{"class":140},[134,1141,1142],{"class":160}," response.headers.get(",[134,1144,1145],{"class":220},"\"Retry-After\"",[134,1147,1148],{"class":160},", ",[134,1150,1151],{"class":220},"\"\"",[134,1153,1107],{"class":160},[134,1155,1157,1160,1163,1166,1169,1172,1175,1178,1181],{"class":136,"line":1156},47,[134,1158,1159],{"class":160},"            time.sleep(",[134,1161,1162],{"class":577},"int",[134,1164,1165],{"class":160},"(retry_after) ",[134,1167,1168],{"class":140},"if",[134,1170,1171],{"class":160}," retry_after.isdigit() ",[134,1173,1174],{"class":140},"else",[134,1176,1177],{"class":577}," 2",[134,1179,1180],{"class":140}," **",[134,1182,1183],{"class":160}," attempt)\n",[134,1185,1187],{"class":136,"line":1186},48,[134,1188,1189],{"class":160},"        response.raise_for_status()\n",[134,1191,1193,1196],{"class":136,"line":1192},49,[134,1194,1195],{"class":140},"        yield",[134,1197,1198],{"class":160}," response.json()\n",[134,1200,1202,1205,1207,1210,1213,1216,1219],{"class":136,"line":1201},50,[134,1203,1204],{"class":160},"        next_url ",[134,1206,471],{"class":140},[134,1208,1209],{"class":160}," response.links.get(",[134,1211,1212],{"class":220},"\"next\"",[134,1214,1215],{"class":160},", {}).get(",[134,1217,1218],{"class":220},"\"url\"",[134,1220,1107],{"class":160},[134,1222,1224,1227,1229,1232,1234,1237,1239,1242],{"class":136,"line":1223},51,[134,1225,1226],{"class":160},"        url ",[134,1228,471],{"class":140},[134,1230,1231],{"class":160}," urljoin(response.url, next_url) ",[134,1233,1168],{"class":140},[134,1235,1236],{"class":160}," next_url ",[134,1238,1174],{"class":140},[134,1240,1241],{"class":577}," None",[134,1243,1244],{"class":270},"  # RFC 8288 allows relative links\n",[134,1246,1248,1251,1253,1255],{"class":136,"line":1247},52,[134,1249,1250],{"class":160},"        params ",[134,1252,471],{"class":140},[134,1254,1241],{"class":577},[134,1256,1257],{"class":270},"  # the next link already carries the query string\n",[134,1259,1261],{"class":136,"line":1260},53,[134,1262,200],{"emptyLinePlaceholder":199},[134,1264,1266],{"class":136,"line":1265},54,[134,1267,200],{"emptyLinePlaceholder":199},[134,1269,1271,1273,1276,1279,1281],{"class":136,"line":1270},55,[134,1272,964],{"class":140},[134,1274,1275],{"class":147}," main",[134,1277,1278],{"class":160},"() -> ",[134,1280,1162],{"class":577},[134,1282,1125],{"class":160},[134,1284,1286,1289,1291,1294,1297,1299,1302],{"class":136,"line":1285},56,[134,1287,1288],{"class":160},"    parser ",[134,1290,471],{"class":140},[134,1292,1293],{"class":160}," argparse.ArgumentParser(",[134,1295,1296],{"class":1090},"description",[134,1298,471],{"class":140},[134,1300,1301],{"class":220},"\"Load changed API records into a raw jsonb table.\"",[134,1303,1107],{"class":160},[134,1305,1307,1310,1313,1315,1318,1320,1323],{"class":136,"line":1306},57,[134,1308,1309],{"class":160},"    parser.add_argument(",[134,1311,1312],{"class":220},"\"--dsn\"",[134,1314,1148],{"class":160},[134,1316,1317],{"class":1090},"required",[134,1319,471],{"class":140},[134,1321,1322],{"class":577},"True",[134,1324,1107],{"class":160},[134,1326,1328,1330,1333,1335,1337,1339,1341,1343,1346,1348,1351],{"class":136,"line":1327},58,[134,1329,1309],{"class":160},[134,1331,1332],{"class":220},"\"--source\"",[134,1334,1148],{"class":160},[134,1336,1317],{"class":1090},[134,1338,471],{"class":140},[134,1340,1322],{"class":577},[134,1342,1148],{"class":160},[134,1344,1345],{"class":1090},"help",[134,1347,471],{"class":140},[134,1349,1350],{"class":220},"\"Raw table name, e.g. psa_tickets\"",[134,1352,1107],{"class":160},[134,1354,1356,1358,1361,1363,1365,1367,1369],{"class":136,"line":1355},59,[134,1357,1309],{"class":160},[134,1359,1360],{"class":220},"\"--url\"",[134,1362,1148],{"class":160},[134,1364,1317],{"class":1090},[134,1366,471],{"class":140},[134,1368,1322],{"class":577},[134,1370,1107],{"class":160},[134,1372,1374,1376,1379,1381,1384,1386,1389],{"class":136,"line":1373},60,[134,1375,1309],{"class":160},[134,1377,1378],{"class":220},"\"--since-param\"",[134,1380,1148],{"class":160},[134,1382,1383],{"class":1090},"default",[134,1385,471],{"class":140},[134,1387,1388],{"class":220},"\"updatedSince\"",[134,1390,1107],{"class":160},[134,1392,1394,1396,1399,1401,1403,1405,1408],{"class":136,"line":1393},61,[134,1395,1309],{"class":160},[134,1397,1398],{"class":220},"\"--id-field\"",[134,1400,1148],{"class":160},[134,1402,1383],{"class":1090},[134,1404,471],{"class":140},[134,1406,1407],{"class":220},"\"id\"",[134,1409,1107],{"class":160},[134,1411,1413,1415,1418,1420,1423,1425,1427,1429,1431,1433,1436],{"class":136,"line":1412},62,[134,1414,1309],{"class":160},[134,1416,1417],{"class":220},"\"--overlap-minutes\"",[134,1419,1148],{"class":160},[134,1421,1422],{"class":1090},"type",[134,1424,471],{"class":140},[134,1426,1162],{"class":577},[134,1428,1148],{"class":160},[134,1430,1383],{"class":1090},[134,1432,471],{"class":140},[134,1434,1435],{"class":577},"15",[134,1437,1107],{"class":160},[134,1439,1441,1444,1446],{"class":136,"line":1440},63,[134,1442,1443],{"class":160},"    args ",[134,1445,471],{"class":140},[134,1447,1448],{"class":160}," parser.parse_args()\n",[134,1450,1452],{"class":136,"line":1451},64,[134,1453,200],{"emptyLinePlaceholder":199},[134,1455,1457,1460,1462],{"class":136,"line":1456},65,[134,1458,1459],{"class":160},"    session ",[134,1461,471],{"class":140},[134,1463,1464],{"class":160}," requests.Session()\n",[134,1466,1468,1471,1474,1477,1479,1482,1485,1488,1491,1494,1497,1500],{"class":136,"line":1467},66,[134,1469,1470],{"class":160},"    session.headers[",[134,1472,1473],{"class":220},"\"Authorization\"",[134,1475,1476],{"class":160},"] ",[134,1478,471],{"class":140},[134,1480,1481],{"class":140}," f",[134,1483,1484],{"class":220},"\"Bearer ",[134,1486,1487],{"class":577},"{",[134,1489,1490],{"class":160},"os.environ[",[134,1492,1493],{"class":220},"'PSA_TOKEN'",[134,1495,1496],{"class":160},"]",[134,1498,1499],{"class":577},"}",[134,1501,1502],{"class":220},"\"\n",[134,1504,1506,1509,1511],{"class":136,"line":1505},67,[134,1507,1508],{"class":160},"    started ",[134,1510,471],{"class":140},[134,1512,1513],{"class":160}," dt.datetime.now(dt.timezone.utc)\n",[134,1515,1517,1520,1522],{"class":136,"line":1516},68,[134,1518,1519],{"class":160},"    upsert ",[134,1521,471],{"class":140},[134,1523,1524],{"class":160}," sql.SQL(\n",[134,1526,1528,1531,1534,1537,1539],{"class":136,"line":1527},69,[134,1529,1530],{"class":220},"        \"INSERT INTO ",[134,1532,1533],{"class":577},"{table}",[134,1535,1536],{"class":220}," (source_id, payload) VALUES ",[134,1538,985],{"class":577},[134,1540,1541],{"class":220}," \"\n",[134,1543,1545],{"class":136,"line":1544},70,[134,1546,1547],{"class":220},"        \"ON CONFLICT (source_id) DO UPDATE SET payload = EXCLUDED.payload, loaded_at = now()\"\n",[134,1549,1551,1554,1556,1558,1561,1564],{"class":136,"line":1550},71,[134,1552,1553],{"class":160},"    ).format(",[134,1555,39],{"class":1090},[134,1557,471],{"class":140},[134,1559,1560],{"class":160},"sql.Identifier(",[134,1562,1563],{"class":220},"\"raw\"",[134,1565,1566],{"class":160},", args.source))\n",[134,1568,1570],{"class":136,"line":1569},72,[134,1571,200],{"emptyLinePlaceholder":199},[134,1573,1575,1578,1580],{"class":136,"line":1574},73,[134,1576,1577],{"class":160},"    conn ",[134,1579,471],{"class":140},[134,1581,1582],{"class":160}," psycopg2.connect(args.dsn)\n",[134,1584,1586,1589,1591],{"class":136,"line":1585},74,[134,1587,1588],{"class":160},"    conn.autocommit ",[134,1590,471],{"class":140},[134,1592,1593],{"class":577}," True\n",[134,1595,1597,1600,1602],{"class":136,"line":1596},75,[134,1598,1599],{"class":160},"    rows ",[134,1601,471],{"class":140},[134,1603,1604],{"class":577}," 0\n",[134,1606,1608,1611],{"class":136,"line":1607},76,[134,1609,1610],{"class":140},"    try",[134,1612,1125],{"class":160},[134,1614,1616,1619,1622,1624],{"class":136,"line":1615},77,[134,1617,1618],{"class":140},"        with",[134,1620,1621],{"class":160}," conn.cursor() ",[134,1623,846],{"class":140},[134,1625,1626],{"class":160}," cur:\n",[134,1628,1630,1633,1635],{"class":136,"line":1629},78,[134,1631,1632],{"class":160},"            since ",[134,1634,471],{"class":140},[134,1636,1637],{"class":160}," last_success(cur, args.source)\n",[134,1639,1641,1644,1646],{"class":136,"line":1640},79,[134,1642,1643],{"class":160},"            params ",[134,1645,471],{"class":140},[134,1647,1648],{"class":160}," {}\n",[134,1650,1652,1654],{"class":136,"line":1651},80,[134,1653,1113],{"class":140},[134,1655,1656],{"class":160}," since:\n",[134,1658,1660,1663,1665,1668,1671,1674,1677,1679],{"class":136,"line":1659},81,[134,1661,1662],{"class":160},"                params[args.since_param] ",[134,1664,471],{"class":140},[134,1666,1667],{"class":160}," (since ",[134,1669,1670],{"class":140},"-",[134,1672,1673],{"class":160}," dt.timedelta(",[134,1675,1676],{"class":1090},"minutes",[134,1678,471],{"class":140},[134,1680,1681],{"class":160},"args.overlap_minutes)).isoformat()\n",[134,1683,1685,1688,1691,1693],{"class":136,"line":1684},82,[134,1686,1687],{"class":140},"            for",[134,1689,1690],{"class":160}," page ",[134,1692,1065],{"class":140},[134,1694,1695],{"class":160}," fetch_pages(session, args.url, params):\n",[134,1697,1699,1702,1704,1707,1710,1713,1716,1719,1721],{"class":136,"line":1698},83,[134,1700,1701],{"class":160},"                values ",[134,1703,471],{"class":140},[134,1705,1706],{"class":160}," [(",[134,1708,1709],{"class":577},"str",[134,1711,1712],{"class":160},"(record[args.id_field]), Json(record)) ",[134,1714,1715],{"class":140},"for",[134,1717,1718],{"class":160}," record ",[134,1720,1065],{"class":140},[134,1722,1723],{"class":160}," page]\n",[134,1725,1727,1730],{"class":136,"line":1726},84,[134,1728,1729],{"class":140},"                if",[134,1731,1732],{"class":160}," values:\n",[134,1734,1736,1739,1742,1744,1747],{"class":136,"line":1735},85,[134,1737,1738],{"class":160},"                    execute_values(cur, upsert.as_string(conn), values, ",[134,1740,1741],{"class":1090},"page_size",[134,1743,471],{"class":140},[134,1745,1746],{"class":577},"500",[134,1748,1107],{"class":160},[134,1750,1752,1755,1758,1761],{"class":136,"line":1751},86,[134,1753,1754],{"class":160},"                    rows ",[134,1756,1757],{"class":140},"+=",[134,1759,1760],{"class":577}," len",[134,1762,1763],{"class":160},"(values)\n",[134,1765,1767],{"class":136,"line":1766},87,[134,1768,1769],{"class":160},"            cur.execute(\n",[134,1771,1773],{"class":136,"line":1772},88,[134,1774,1775],{"class":220},"                \"INSERT INTO raw.etl_runs (source, started_at, finished_at, status, rows_loaded) \"\n",[134,1777,1779,1782,1784,1786,1788,1791,1793,1796],{"class":136,"line":1778},89,[134,1780,1781],{"class":220},"                \"VALUES (",[134,1783,985],{"class":577},[134,1785,1148],{"class":220},[134,1787,985],{"class":577},[134,1789,1790],{"class":220},", now(), 'success', ",[134,1792,985],{"class":577},[134,1794,1795],{"class":220},")\"",[134,1797,599],{"class":160},[134,1799,1801],{"class":136,"line":1800},90,[134,1802,1803],{"class":160},"                (args.source, started, rows),\n",[134,1805,1807],{"class":136,"line":1806},91,[134,1808,1809],{"class":160},"            )\n",[134,1811,1813,1816,1818,1821,1824,1826,1829,1831,1834,1836,1839,1841,1844],{"class":136,"line":1812},92,[134,1814,1815],{"class":577},"        print",[134,1817,1071],{"class":160},[134,1819,1820],{"class":140},"f",[134,1822,1823],{"class":220},"\"",[134,1825,1487],{"class":577},[134,1827,1828],{"class":160},"args.source",[134,1830,1499],{"class":577},[134,1832,1833],{"class":220},": ",[134,1835,1487],{"class":577},[134,1837,1838],{"class":160},"rows",[134,1840,1499],{"class":577},[134,1842,1843],{"class":220}," records upserted\"",[134,1845,1107],{"class":160},[134,1847,1849,1852],{"class":136,"line":1848},93,[134,1850,1851],{"class":140},"        return",[134,1853,1604],{"class":577},[134,1855,1857,1860,1863,1866],{"class":136,"line":1856},94,[134,1858,1859],{"class":140},"    except",[134,1861,1862],{"class":577}," Exception",[134,1864,1865],{"class":140}," as",[134,1867,1868],{"class":160}," error:\n",[134,1870,1872,1874,1876,1878],{"class":136,"line":1871},95,[134,1873,1618],{"class":140},[134,1875,1621],{"class":160},[134,1877,846],{"class":140},[134,1879,1626],{"class":160},[134,1881,1883],{"class":136,"line":1882},96,[134,1884,1769],{"class":160},[134,1886,1888],{"class":136,"line":1887},97,[134,1889,1890],{"class":220},"                \"INSERT INTO raw.etl_runs (source, started_at, finished_at, status, rows_loaded, error) \"\n",[134,1892,1894,1896,1898,1900,1902,1905,1907,1909,1911,1913],{"class":136,"line":1893},98,[134,1895,1781],{"class":220},[134,1897,985],{"class":577},[134,1899,1148],{"class":220},[134,1901,985],{"class":577},[134,1903,1904],{"class":220},", now(), 'failed', ",[134,1906,985],{"class":577},[134,1908,1148],{"class":220},[134,1910,985],{"class":577},[134,1912,1795],{"class":220},[134,1914,599],{"class":160},[134,1916,1918,1921,1923,1926,1929],{"class":136,"line":1917},99,[134,1919,1920],{"class":160},"                (args.source, started, rows, ",[134,1922,1709],{"class":577},[134,1924,1925],{"class":160},"(error)[:",[134,1927,1928],{"class":577},"2000",[134,1930,1931],{"class":160},"]),\n",[134,1933,1935],{"class":136,"line":1934},100,[134,1936,1809],{"class":160},[134,1938,1940,1942,1944,1946,1948,1950,1952,1954,1957,1959,1961,1963,1966,1968,1971,1973,1975,1977,1980,1982],{"class":136,"line":1939},101,[134,1941,1815],{"class":577},[134,1943,1071],{"class":160},[134,1945,1820],{"class":140},[134,1947,1823],{"class":220},[134,1949,1487],{"class":577},[134,1951,1828],{"class":160},[134,1953,1499],{"class":577},[134,1955,1956],{"class":220},": failed after ",[134,1958,1487],{"class":577},[134,1960,1838],{"class":160},[134,1962,1499],{"class":577},[134,1964,1965],{"class":220}," records: ",[134,1967,1487],{"class":577},[134,1969,1970],{"class":160},"error",[134,1972,1499],{"class":577},[134,1974,1823],{"class":220},[134,1976,1148],{"class":160},[134,1978,1979],{"class":1090},"file",[134,1981,471],{"class":140},[134,1983,1984],{"class":160},"sys.stderr)\n",[134,1986,1988,1990],{"class":136,"line":1987},102,[134,1989,1851],{"class":140},[134,1991,1992],{"class":577}," 1\n",[134,1994,1996,1999],{"class":136,"line":1995},103,[134,1997,1998],{"class":140},"    finally",[134,2000,1125],{"class":160},[134,2002,2004],{"class":136,"line":2003},104,[134,2005,2006],{"class":160},"        conn.close()\n",[134,2008,2010],{"class":136,"line":2009},105,[134,2011,200],{"emptyLinePlaceholder":199},[134,2013,2015],{"class":136,"line":2014},106,[134,2016,200],{"emptyLinePlaceholder":199},[134,2018,2020,2022,2025,2028,2031],{"class":136,"line":2019},107,[134,2021,1168],{"class":140},[134,2023,2024],{"class":577}," __name__",[134,2026,2027],{"class":140}," ==",[134,2029,2030],{"class":220}," \"__main__\"",[134,2032,1125],{"class":160},[134,2034,2036],{"class":136,"line":2035},108,[134,2037,2038],{"class":160},"    sys.exit(main())\n",[11,2040,2041],{},"The staging view is where the vendor's field names become yours, and where the \"null means default\" surprises get handled once:",[125,2043,2045],{"className":127,"code":2044,"language":129,"meta":130,"style":130},"CREATE OR REPLACE VIEW staging.stg_tickets AS\nSELECT\n    source_id                                           AS ticket_id,\n    (payload ->> 'companyId')::bigint                   AS client_id,\n    lower(trim(payload ->> 'status'))                   AS status,\n    coalesce(payload ->> 'priority', 'p3')              AS priority,  -- UI shows P3 when unset\n    payload ->> 'queue'                                 AS queue,\n    (payload ->> 'createdAt')::timestamptz              AS created_at,\n    (payload ->> 'firstResponseAt')::timestamptz        AS first_response_at,\n    (payload ->> 'resolvedAt')::timestamptz             AS resolved_at,\n    loaded_at\nFROM raw.psa_tickets;\n",[69,2046,2047,2064,2069,2080,2102,2130,2158,2176,2195,2214,2233,2238],{"__ignoreMap":130},[134,2048,2049,2052,2055,2058,2061],{"class":136,"line":137},[134,2050,2051],{"class":140},"CREATE OR REPLACE",[134,2053,2054],{"class":140}," VIEW",[134,2056,2057],{"class":147}," staging",[134,2059,2060],{"class":160},".stg_tickets ",[134,2062,2063],{"class":140},"AS\n",[134,2065,2066],{"class":136,"line":164},[134,2067,2068],{"class":140},"SELECT\n",[134,2070,2071,2074,2077],{"class":136,"line":180},[134,2072,2073],{"class":160},"    source_id                                           ",[134,2075,2076],{"class":140},"AS",[134,2078,2079],{"class":160}," ticket_id,\n",[134,2081,2082,2085,2088,2091,2094,2096,2099],{"class":136,"line":196},[134,2083,2084],{"class":160},"    (payload ",[134,2086,2087],{"class":140},"->>",[134,2089,2090],{"class":220}," 'companyId'",[134,2092,2093],{"class":160},")::",[134,2095,666],{"class":140},[134,2097,2098],{"class":140},"                   AS",[134,2100,2101],{"class":160}," client_id,\n",[134,2103,2104,2107,2109,2112,2115,2117,2120,2123,2125,2128],{"class":136,"line":203},[134,2105,2106],{"class":577},"    lower",[134,2108,1071],{"class":160},[134,2110,2111],{"class":577},"trim",[134,2113,2114],{"class":160},"(payload ",[134,2116,2087],{"class":140},[134,2118,2119],{"class":220}," 'status'",[134,2121,2122],{"class":160},"))                   ",[134,2124,2076],{"class":140},[134,2126,2127],{"class":140}," status",[134,2129,599],{"class":160},[134,2131,2132,2135,2137,2139,2142,2144,2147,2150,2152,2155],{"class":136,"line":226},[134,2133,2134],{"class":577},"    coalesce",[134,2136,2114],{"class":160},[134,2138,2087],{"class":140},[134,2140,2141],{"class":220}," 'priority'",[134,2143,1148],{"class":160},[134,2145,2146],{"class":220},"'p3'",[134,2148,2149],{"class":160},")              ",[134,2151,2076],{"class":140},[134,2153,2154],{"class":160}," priority,  ",[134,2156,2157],{"class":270},"-- UI shows P3 when unset\n",[134,2159,2160,2163,2165,2168,2171,2174],{"class":136,"line":244},[134,2161,2162],{"class":160},"    payload ",[134,2164,2087],{"class":140},[134,2166,2167],{"class":220}," 'queue'",[134,2169,2170],{"class":140},"                                 AS",[134,2172,2173],{"class":140}," queue",[134,2175,599],{"class":160},[134,2177,2178,2180,2182,2185,2187,2189,2192],{"class":136,"line":262},[134,2179,2084],{"class":160},[134,2181,2087],{"class":140},[134,2183,2184],{"class":220}," 'createdAt'",[134,2186,2093],{"class":160},[134,2188,617],{"class":140},[134,2190,2191],{"class":140},"              AS",[134,2193,2194],{"class":160}," created_at,\n",[134,2196,2197,2199,2201,2204,2206,2208,2211],{"class":136,"line":267},[134,2198,2084],{"class":160},[134,2200,2087],{"class":140},[134,2202,2203],{"class":220}," 'firstResponseAt'",[134,2205,2093],{"class":160},[134,2207,617],{"class":140},[134,2209,2210],{"class":140},"        AS",[134,2212,2213],{"class":160}," first_response_at,\n",[134,2215,2216,2218,2220,2223,2225,2227,2230],{"class":136,"line":274},[134,2217,2084],{"class":160},[134,2219,2087],{"class":140},[134,2221,2222],{"class":220}," 'resolvedAt'",[134,2224,2093],{"class":160},[134,2226,617],{"class":140},[134,2228,2229],{"class":140},"             AS",[134,2231,2232],{"class":160}," resolved_at,\n",[134,2234,2235],{"class":136,"line":298},[134,2236,2237],{"class":160},"    loaded_at\n",[134,2239,2240,2243,2245,2247,2249],{"class":136,"line":303},[134,2241,2242],{"class":140},"FROM",[134,2244,157],{"class":577},[134,2246,556],{"class":160},[134,2248,582],{"class":577},[134,2250,161],{"class":160},[11,2252,2253,2254,2256,2257,2262],{},"The JSON field names are placeholders for whatever your PSA returns. The pattern is the point. From here, metric views go in ",[69,2255,111],{},". I cover how to keep those consistent in ",[2258,2259,2261],"a",{"href":2260},"\u002F2026\u002F08\u002F19\u002Fdata-warehouse-reusable-sql-models-for-it-operations-metrics\u002F","reusable SQL models for IT operations metrics",". The answer to the after-hours question from the intro ends up being one short query against a staging view, instead of a day in Excel:",[125,2264,2266],{"className":127,"code":2265,"language":129,"meta":130,"style":130},"SELECT client_id,\n       count(*) FILTER (\n           WHERE extract(isodow FROM created_at AT TIME ZONE 'America\u002FChicago') IN (6, 7)\n              OR (created_at AT TIME ZONE 'America\u002FChicago')::time NOT BETWEEN '08:00' AND '17:00'\n       ) AS after_hours_tickets,\n       count(*) AS all_tickets\nFROM staging.stg_tickets\nWHERE created_at >= now() - interval '90 days'\nGROUP BY client_id\nORDER BY after_hours_tickets DESC\nLIMIT 10;\n",[69,2267,2268,2275,2293,2335,2370,2380,2395,2406,2429,2437,2448],{"__ignoreMap":130},[134,2269,2270,2273],{"class":136,"line":137},[134,2271,2272],{"class":140},"SELECT",[134,2274,2101],{"class":160},[134,2276,2277,2280,2282,2285,2288,2291],{"class":136,"line":164},[134,2278,2279],{"class":577},"       count",[134,2281,1071],{"class":160},[134,2283,2284],{"class":140},"*",[134,2286,2287],{"class":160},") ",[134,2289,2290],{"class":140},"FILTER",[134,2292,585],{"class":160},[134,2294,2295,2298,2301,2303,2306,2309,2312,2315,2318,2320,2322,2325,2328,2330,2333],{"class":136,"line":180},[134,2296,2297],{"class":140},"           WHERE",[134,2299,2300],{"class":160}," extract(isodow ",[134,2302,2242],{"class":140},[134,2304,2305],{"class":160}," created_at ",[134,2307,2308],{"class":140},"AT",[134,2310,2311],{"class":140}," TIME",[134,2313,2314],{"class":140}," ZONE",[134,2316,2317],{"class":220}," 'America\u002FChicago'",[134,2319,2287],{"class":160},[134,2321,349],{"class":140},[134,2323,2324],{"class":160}," (",[134,2326,2327],{"class":577},"6",[134,2329,1148],{"class":160},[134,2331,2332],{"class":577},"7",[134,2334,1107],{"class":160},[134,2336,2337,2340,2343,2345,2347,2349,2351,2353,2356,2358,2361,2364,2367],{"class":136,"line":196},[134,2338,2339],{"class":140},"              OR",[134,2341,2342],{"class":160}," (created_at ",[134,2344,2308],{"class":140},[134,2346,2311],{"class":140},[134,2348,2314],{"class":140},[134,2350,2317],{"class":220},[134,2352,2093],{"class":160},[134,2354,2355],{"class":140},"time",[134,2357,151],{"class":140},[134,2359,2360],{"class":140}," BETWEEN",[134,2362,2363],{"class":220}," '08:00'",[134,2365,2366],{"class":140}," AND",[134,2368,2369],{"class":220}," '17:00'\n",[134,2371,2372,2375,2377],{"class":136,"line":203},[134,2373,2374],{"class":160},"       ) ",[134,2376,2076],{"class":140},[134,2378,2379],{"class":160}," after_hours_tickets,\n",[134,2381,2382,2384,2386,2388,2390,2392],{"class":136,"line":226},[134,2383,2279],{"class":577},[134,2385,1071],{"class":160},[134,2387,2284],{"class":140},[134,2389,2287],{"class":160},[134,2391,2076],{"class":140},[134,2393,2394],{"class":160}," all_tickets\n",[134,2396,2397,2399,2401,2403],{"class":136,"line":244},[134,2398,2242],{"class":140},[134,2400,2057],{"class":577},[134,2402,556],{"class":160},[134,2404,2405],{"class":577},"stg_tickets\n",[134,2407,2408,2411,2413,2416,2418,2421,2423,2426],{"class":136,"line":262},[134,2409,2410],{"class":140},"WHERE",[134,2412,2305],{"class":160},[134,2414,2415],{"class":140},">=",[134,2417,625],{"class":140},[134,2419,2420],{"class":160},"() ",[134,2422,1670],{"class":140},[134,2424,2425],{"class":160}," interval ",[134,2427,2428],{"class":220},"'90 days'\n",[134,2430,2431,2434],{"class":136,"line":267},[134,2432,2433],{"class":140},"GROUP BY",[134,2435,2436],{"class":160}," client_id\n",[134,2438,2439,2442,2445],{"class":136,"line":274},[134,2440,2441],{"class":140},"ORDER BY",[134,2443,2444],{"class":160}," after_hours_tickets ",[134,2446,2447],{"class":140},"DESC\n",[134,2449,2450,2453,2456],{"class":136,"line":298},[134,2451,2452],{"class":140},"LIMIT",[134,2454,2455],{"class":577}," 10",[134,2457,161],{"class":160},[18,2459,2461],{"id":2460},"ownership-has-to-survive-the-person-who-built-it","Ownership has to survive the person who built it",[11,2463,2464],{},"The real risk in a no-BI-team reporting layer isn't the initial build. It's the day the person who built it moves teams or leaves. I've inherited more than one of these pipelines as a pile of undocumented scripts running on someone's forgotten scheduled task, silently failing for months because no alert was ever wired up. Every pipeline needs a runbook, a named backup owner even if that person only touches it twice a year, and monitoring on the pipeline itself, not just the dashboards it feeds, so a failure surfaces before three months of reports quietly go stale.",[11,2466,2467],{},"Put the ownership in the database, next to the data, where the next person will actually find it:",[125,2469,2471],{"className":127,"code":2470,"language":129,"meta":130,"style":130},"CREATE TABLE IF NOT EXISTS raw.pipeline_owners (\n    source          text PRIMARY KEY,\n    owner           text NOT NULL,\n    backup_owner    text NOT NULL,\n    runbook_url     text NOT NULL,\n    max_staleness   interval NOT NULL\n);\n\nINSERT INTO raw.pipeline_owners VALUES\n    ('psa_tickets', 'ops-lead@example.com', 'sysadmin2@example.com',\n     'https:\u002F\u002Fwiki.example.com\u002Freporting\u002Fpsa-tickets', interval '2 hours')\nON CONFLICT (source) DO NOTHING;\n",[69,2472,2473,2494,2505,2517,2528,2539,2547,2551,2555,2569,2589,2602],{"__ignoreMap":130},[134,2474,2475,2477,2479,2481,2483,2485,2487,2489,2492],{"class":136,"line":137},[134,2476,141],{"class":140},[134,2478,568],{"class":140},[134,2480,148],{"class":147},[134,2482,151],{"class":140},[134,2484,154],{"class":140},[134,2486,157],{"class":577},[134,2488,556],{"class":160},[134,2490,2491],{"class":577},"pipeline_owners",[134,2493,585],{"class":160},[134,2495,2496,2499,2501,2503],{"class":136,"line":164},[134,2497,2498],{"class":160},"    source          ",[134,2500,593],{"class":140},[134,2502,596],{"class":140},[134,2504,599],{"class":160},[134,2506,2507,2510,2513,2515],{"class":136,"line":180},[134,2508,2509],{"class":140},"    owner",[134,2511,2512],{"class":140},"           text",[134,2514,620],{"class":140},[134,2516,599],{"class":160},[134,2518,2519,2522,2524,2526],{"class":136,"line":196},[134,2520,2521],{"class":160},"    backup_owner    ",[134,2523,593],{"class":140},[134,2525,620],{"class":140},[134,2527,599],{"class":160},[134,2529,2530,2533,2535,2537],{"class":136,"line":203},[134,2531,2532],{"class":160},"    runbook_url     ",[134,2534,593],{"class":140},[134,2536,620],{"class":140},[134,2538,599],{"class":160},[134,2540,2541,2544],{"class":136,"line":226},[134,2542,2543],{"class":160},"    max_staleness   interval ",[134,2545,2546],{"class":140},"NOT NULL\n",[134,2548,2549],{"class":136,"line":244},[134,2550,633],{"class":160},[134,2552,2553],{"class":136,"line":262},[134,2554,200],{"emptyLinePlaceholder":199},[134,2556,2557,2560,2562,2564,2566],{"class":136,"line":267},[134,2558,2559],{"class":140},"INSERT INTO",[134,2561,157],{"class":577},[134,2563,556],{"class":160},[134,2565,2491],{"class":577},[134,2567,2568],{"class":140}," VALUES\n",[134,2570,2571,2574,2577,2579,2582,2584,2587],{"class":136,"line":274},[134,2572,2573],{"class":160},"    (",[134,2575,2576],{"class":220},"'psa_tickets'",[134,2578,1148],{"class":160},[134,2580,2581],{"class":220},"'ops-lead@example.com'",[134,2583,1148],{"class":160},[134,2585,2586],{"class":220},"'sysadmin2@example.com'",[134,2588,599],{"class":160},[134,2590,2591,2594,2597,2600],{"class":136,"line":298},[134,2592,2593],{"class":220},"     'https:\u002F\u002Fwiki.example.com\u002Freporting\u002Fpsa-tickets'",[134,2595,2596],{"class":160},", interval ",[134,2598,2599],{"class":220},"'2 hours'",[134,2601,1107],{"class":160},[134,2603,2604,2606],{"class":136,"line":303},[134,2605,317],{"class":140},[134,2607,2608],{"class":160}," CONFLICT (source) DO NOTHING;\n",[11,2610,2611],{},"Then a freshness view that turns \"is this data current?\" into a query any dashboard can show at the top of the page:",[125,2613,2615],{"className":127,"code":2614,"language":129,"meta":130,"style":130},"CREATE OR REPLACE VIEW marts.pipeline_freshness AS\nSELECT o.source,\n       o.owner,\n       o.backup_owner,\n       o.runbook_url,\n       max(r.finished_at) FILTER (WHERE r.status = 'success')          AS last_success,\n       now() - max(r.finished_at) FILTER (WHERE r.status = 'success')  AS age,\n       coalesce(now() - max(r.finished_at) FILTER (WHERE r.status = 'success') > o.max_staleness, true) AS is_stale\nFROM raw.pipeline_owners o\nLEFT JOIN raw.etl_runs r ON r.source = o.source\nGROUP BY o.source, o.owner, o.backup_owner, o.runbook_url, o.max_staleness;\n",[69,2616,2617,2631,2645,2657,2668,2679,2723,2769,2831,2844,2875],{"__ignoreMap":130},[134,2618,2619,2621,2623,2626,2629],{"class":136,"line":137},[134,2620,2051],{"class":140},[134,2622,2054],{"class":140},[134,2624,2625],{"class":147}," marts",[134,2627,2628],{"class":160},".pipeline_freshness ",[134,2630,2063],{"class":140},[134,2632,2633,2635,2638,2640,2643],{"class":136,"line":164},[134,2634,2272],{"class":140},[134,2636,2637],{"class":577}," o",[134,2639,556],{"class":160},[134,2641,2642],{"class":577},"source",[134,2644,599],{"class":160},[134,2646,2647,2650,2652,2655],{"class":136,"line":180},[134,2648,2649],{"class":577},"       o",[134,2651,556],{"class":160},[134,2653,2654],{"class":577},"owner",[134,2656,599],{"class":160},[134,2658,2659,2661,2663,2666],{"class":136,"line":196},[134,2660,2649],{"class":577},[134,2662,556],{"class":160},[134,2664,2665],{"class":577},"backup_owner",[134,2667,599],{"class":160},[134,2669,2670,2672,2674,2677],{"class":136,"line":203},[134,2671,2649],{"class":577},[134,2673,556],{"class":160},[134,2675,2676],{"class":577},"runbook_url",[134,2678,599],{"class":160},[134,2680,2681,2684,2686,2689,2691,2694,2696,2698,2700,2702,2705,2707,2710,2712,2715,2718,2720],{"class":136,"line":226},[134,2682,2683],{"class":577},"       max",[134,2685,1071],{"class":160},[134,2687,2688],{"class":577},"r",[134,2690,556],{"class":160},[134,2692,2693],{"class":577},"finished_at",[134,2695,2287],{"class":160},[134,2697,2290],{"class":140},[134,2699,2324],{"class":160},[134,2701,2410],{"class":140},[134,2703,2704],{"class":577}," r",[134,2706,556],{"class":160},[134,2708,2709],{"class":577},"status",[134,2711,945],{"class":140},[134,2713,2714],{"class":220}," 'success'",[134,2716,2717],{"class":160},")          ",[134,2719,2076],{"class":140},[134,2721,2722],{"class":160}," last_success,\n",[134,2724,2725,2728,2730,2732,2735,2737,2739,2741,2743,2745,2747,2749,2751,2753,2755,2757,2759,2761,2764,2766],{"class":136,"line":244},[134,2726,2727],{"class":140},"       now",[134,2729,2420],{"class":160},[134,2731,1670],{"class":140},[134,2733,2734],{"class":577}," max",[134,2736,1071],{"class":160},[134,2738,2688],{"class":577},[134,2740,556],{"class":160},[134,2742,2693],{"class":577},[134,2744,2287],{"class":160},[134,2746,2290],{"class":140},[134,2748,2324],{"class":160},[134,2750,2410],{"class":140},[134,2752,2704],{"class":577},[134,2754,556],{"class":160},[134,2756,2709],{"class":577},[134,2758,945],{"class":140},[134,2760,2714],{"class":220},[134,2762,2763],{"class":160},")  ",[134,2765,2076],{"class":140},[134,2767,2768],{"class":160}," age,\n",[134,2770,2771,2774,2776,2779,2781,2783,2785,2787,2789,2791,2793,2795,2797,2799,2801,2803,2805,2807,2809,2811,2813,2816,2818,2820,2823,2826,2828],{"class":136,"line":262},[134,2772,2773],{"class":577},"       coalesce",[134,2775,1071],{"class":160},[134,2777,2778],{"class":140},"now",[134,2780,2420],{"class":160},[134,2782,1670],{"class":140},[134,2784,2734],{"class":577},[134,2786,1071],{"class":160},[134,2788,2688],{"class":577},[134,2790,556],{"class":160},[134,2792,2693],{"class":577},[134,2794,2287],{"class":160},[134,2796,2290],{"class":140},[134,2798,2324],{"class":160},[134,2800,2410],{"class":140},[134,2802,2704],{"class":577},[134,2804,556],{"class":160},[134,2806,2709],{"class":577},[134,2808,945],{"class":140},[134,2810,2714],{"class":220},[134,2812,2287],{"class":160},[134,2814,2815],{"class":140},">",[134,2817,2637],{"class":577},[134,2819,556],{"class":160},[134,2821,2822],{"class":577},"max_staleness",[134,2824,2825],{"class":160},", true) ",[134,2827,2076],{"class":140},[134,2829,2830],{"class":160}," is_stale\n",[134,2832,2833,2835,2837,2839,2841],{"class":136,"line":267},[134,2834,2242],{"class":140},[134,2836,157],{"class":577},[134,2838,556],{"class":160},[134,2840,2491],{"class":577},[134,2842,2843],{"class":160}," o\n",[134,2845,2846,2849,2851,2853,2855,2858,2860,2862,2864,2866,2868,2870,2872],{"class":136,"line":274},[134,2847,2848],{"class":140},"LEFT JOIN",[134,2850,157],{"class":577},[134,2852,556],{"class":160},[134,2854,656],{"class":577},[134,2856,2857],{"class":160}," r ",[134,2859,317],{"class":140},[134,2861,2704],{"class":577},[134,2863,556],{"class":160},[134,2865,2642],{"class":577},[134,2867,945],{"class":140},[134,2869,2637],{"class":577},[134,2871,556],{"class":160},[134,2873,2874],{"class":577},"source\n",[134,2876,2877,2879,2881,2883,2885,2887,2890,2892,2894,2896,2898,2900,2902,2904,2906,2908,2910,2912,2914,2916,2918],{"class":136,"line":298},[134,2878,2433],{"class":140},[134,2880,2637],{"class":577},[134,2882,556],{"class":160},[134,2884,2642],{"class":577},[134,2886,1148],{"class":160},[134,2888,2889],{"class":577},"o",[134,2891,556],{"class":160},[134,2893,2654],{"class":577},[134,2895,1148],{"class":160},[134,2897,2889],{"class":577},[134,2899,556],{"class":160},[134,2901,2665],{"class":577},[134,2903,1148],{"class":160},[134,2905,2889],{"class":577},[134,2907,556],{"class":160},[134,2909,2676],{"class":577},[134,2911,1148],{"class":160},[134,2913,2889],{"class":577},[134,2915,556],{"class":160},[134,2917,2822],{"class":577},[134,2919,161],{"class":160},[11,2921,2922,2923,2926],{},"And a check that fails loudly. Cron mails a job's output to the ",[69,2924,2925],{},"MAILTO"," address, so a script that prints only when something is wrong turns a stale source into an email naming the owner and the runbook:",[125,2928,2932],{"className":2929,"code":2930,"language":2931,"meta":130,"style":130},"language-bash shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv bash\n# check_freshness.sh: print stale pipelines and exit 1 if any exist.\n# Needs psql and READER_DSN (a role that can read marts) in \u002Fopt\u002Freporting\u002Fenv.\nset -euo pipefail\nsource \u002Fopt\u002Freporting\u002Fenv\n\nstale=$(psql \"$READER_DSN\" --no-align --tuples-only --field-separator=' | ' --command \\\n  \"SELECT source, coalesce(age::text, 'never'), owner, backup_owner, runbook_url\n     FROM marts.pipeline_freshness WHERE is_stale\")\n\nif [[ -n \"$stale\" ]]; then\n  echo \"Stale reporting pipelines (source | age | owner | backup | runbook):\"\n  echo \"$stale\"\n  exit 1\nfi\n","bash",[69,2933,2934,2939,2944,2949,2960,2967,2971,3010,3015,3022,3026,3049,3057,3067,3074],{"__ignoreMap":130},[134,2935,2936],{"class":136,"line":137},[134,2937,2938],{"class":270},"#!\u002Fusr\u002Fbin\u002Fenv bash\n",[134,2940,2941],{"class":136,"line":164},[134,2942,2943],{"class":270},"# check_freshness.sh: print stale pipelines and exit 1 if any exist.\n",[134,2945,2946],{"class":136,"line":180},[134,2947,2948],{"class":270},"# Needs psql and READER_DSN (a role that can read marts) in \u002Fopt\u002Freporting\u002Fenv.\n",[134,2950,2951,2954,2957],{"class":136,"line":196},[134,2952,2953],{"class":577},"set",[134,2955,2956],{"class":577}," -euo",[134,2958,2959],{"class":220}," pipefail\n",[134,2961,2962,2964],{"class":136,"line":203},[134,2963,2642],{"class":577},[134,2965,2966],{"class":220}," \u002Fopt\u002Freporting\u002Fenv\n",[134,2968,2969],{"class":136,"line":226},[134,2970,200],{"emptyLinePlaceholder":199},[134,2972,2973,2976,2978,2981,2984,2987,2990,2992,2995,2998,3001,3004,3007],{"class":136,"line":244},[134,2974,2975],{"class":160},"stale",[134,2977,471],{"class":140},[134,2979,2980],{"class":160},"$(",[134,2982,2983],{"class":147},"psql",[134,2985,2986],{"class":220}," \"",[134,2988,2989],{"class":160},"$READER_DSN",[134,2991,1823],{"class":220},[134,2993,2994],{"class":577}," --no-align",[134,2996,2997],{"class":577}," --tuples-only",[134,2999,3000],{"class":577}," --field-separator=",[134,3002,3003],{"class":220},"' | '",[134,3005,3006],{"class":577}," --command",[134,3008,3009],{"class":577}," \\\n",[134,3011,3012],{"class":136,"line":262},[134,3013,3014],{"class":220},"  \"SELECT source, coalesce(age::text, 'never'), owner, backup_owner, runbook_url\n",[134,3016,3017,3020],{"class":136,"line":267},[134,3018,3019],{"class":220},"     FROM marts.pipeline_freshness WHERE is_stale\"",[134,3021,1107],{"class":160},[134,3023,3024],{"class":136,"line":274},[134,3025,200],{"emptyLinePlaceholder":199},[134,3027,3028,3030,3033,3036,3038,3041,3043,3046],{"class":136,"line":298},[134,3029,1168],{"class":140},[134,3031,3032],{"class":160}," [[ ",[134,3034,3035],{"class":140},"-n",[134,3037,2986],{"class":220},[134,3039,3040],{"class":160},"$stale",[134,3042,1823],{"class":220},[134,3044,3045],{"class":160}," ]]; ",[134,3047,3048],{"class":140},"then\n",[134,3050,3051,3054],{"class":136,"line":303},[134,3052,3053],{"class":577},"  echo",[134,3055,3056],{"class":220}," \"Stale reporting pipelines (source | age | owner | backup | runbook):\"\n",[134,3058,3059,3061,3063,3065],{"class":136,"line":309},[134,3060,3053],{"class":577},[134,3062,2986],{"class":220},[134,3064,3040],{"class":160},[134,3066,1502],{"class":220},[134,3068,3069,3072],{"class":136,"line":329},[134,3070,3071],{"class":577},"  exit",[134,3073,1992],{"class":577},[134,3075,3076],{"class":136,"line":372},[134,3077,3078],{"class":140},"fi\n",[11,3080,3081,3082,1148,3085,552,3088,3091,3092,3095,3096,3099,3100,3102,3103,3106],{},"The env file holds ",[69,3083,3084],{},"LOADER_DSN",[69,3086,3087],{},"READER_DSN",[69,3089,3090],{},"PSA_TOKEN",", and is readable only by the service account. Keep passwords out of the DSNs and put them in the service account's ",[69,3093,3094],{},"~\u002F.pgpass"," (mode ",[69,3097,3098],{},"0600","): the DSNs end up on the ",[69,3101,2983],{}," and Python command lines, where ",[69,3104,3105],{},"ps"," shows them to any local user. A small wrapper runs the loader:",[125,3108,3110],{"className":2929,"code":3109,"language":2931,"meta":130,"style":130},"#!\u002Fusr\u002Fbin\u002Fenv bash\n# run_psa_tickets.sh: load PSA tickets; stays quiet on success, prints errors on failure.\nset -euo pipefail\nsource \u002Fopt\u002Freporting\u002Fenv\nexport PSA_TOKEN\n\u002Fopt\u002Freporting\u002Fload_api_to_raw.py --dsn \"$LOADER_DSN\" --source psa_tickets \\\n    --url https:\u002F\u002Fpsa.example.com\u002Fapi\u002Fv1\u002Ftickets --since-param updatedSince >\u002Fdev\u002Fnull\n",[69,3111,3112,3116,3121,3129,3135,3143,3166],{"__ignoreMap":130},[134,3113,3114],{"class":136,"line":137},[134,3115,2938],{"class":270},[134,3117,3118],{"class":136,"line":164},[134,3119,3120],{"class":270},"# run_psa_tickets.sh: load PSA tickets; stays quiet on success, prints errors on failure.\n",[134,3122,3123,3125,3127],{"class":136,"line":180},[134,3124,2953],{"class":577},[134,3126,2956],{"class":577},[134,3128,2959],{"class":220},[134,3130,3131,3133],{"class":136,"line":196},[134,3132,2642],{"class":577},[134,3134,2966],{"class":220},[134,3136,3137,3140],{"class":136,"line":203},[134,3138,3139],{"class":140},"export",[134,3141,3142],{"class":160}," PSA_TOKEN\n",[134,3144,3145,3148,3151,3153,3156,3158,3161,3164],{"class":136,"line":226},[134,3146,3147],{"class":147},"\u002Fopt\u002Freporting\u002Fload_api_to_raw.py",[134,3149,3150],{"class":577}," --dsn",[134,3152,2986],{"class":220},[134,3154,3155],{"class":160},"$LOADER_DSN",[134,3157,1823],{"class":220},[134,3159,3160],{"class":577}," --source",[134,3162,3163],{"class":220}," psa_tickets",[134,3165,3009],{"class":577},[134,3167,3168,3171,3174,3177,3180,3183],{"class":136,"line":244},[134,3169,3170],{"class":577},"    --url",[134,3172,3173],{"class":220}," https:\u002F\u002Fpsa.example.com\u002Fapi\u002Fv1\u002Ftickets",[134,3175,3176],{"class":577}," --since-param",[134,3178,3179],{"class":220}," updatedSince",[134,3181,3182],{"class":140}," >",[134,3184,3185],{"class":220},"\u002Fdev\u002Fnull\n",[125,3187,3191],{"className":3188,"code":3190,"language":593,"meta":130},[3189],"language-text","# crontab for the reporting service account\nMAILTO=reporting-alerts@example.com\n*\u002F30 * * * *  \u002Fopt\u002Freporting\u002Frun_psa_tickets.sh\n15 * * * *    \u002Fopt\u002Freporting\u002Fcheck_freshness.sh\n",[69,3192,3190],{"__ignoreMap":130},[11,3194,3195,3196,3199,3200,3202,3203,3206,3207,3209,3210,3212],{},"The loader's stdout goes to ",[69,3197,3198],{},"\u002Fdev\u002Fnull"," but stderr doesn't, so failures still reach the mailbox. Create ",[69,3201,547],{}," and ",[69,3204,3205],{},"raw.pipeline_owners"," as ",[69,3208,79],{},", so the default privileges above let ",[69,3211,84],{}," (which owns the freshness view) read them. It's crude, and it's the kind of alerting that still works three years after its author left.",[18,3214,3216],{"id":3215},"it-doesnt-need-to-be-elegant-to-be-worth-it","It doesn't need to be elegant to be worth it",[11,3218,3219,3220,3224],{},"The reporting layers that succeed in this kind of environment aren't the technically impressive ones. They're the ones that answer the three or four questions leadership actually asks on a regular cadence, reliably enough that people stop pulling manual exports to double-check them. That's a lower bar than a proper BI team would set for itself, and it's still a real improvement over the spreadsheet it replaced. If the load on that single database eventually becomes the problem, that's a good problem, and ",[2258,3221,3223],{"href":3222},"\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse\u002F","when to stop querying production"," covers how to tell.",[18,3226,3228],{"id":3227},"references","References",[3230,3231,3232,3240,3247,3254,3261,3268,3275,3282],"ul",{},[500,3233,3234],{},[2258,3235,3239],{"href":3236,"rel":3237},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fsql-alterdefaultprivileges.html",[3238],"nofollow","PostgreSQL: ALTER DEFAULT PRIVILEGES",[500,3241,3242],{},[2258,3243,3246],{"href":3244,"rel":3245},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fsql-alterrole.html",[3238],"PostgreSQL: ALTER ROLE (per-role settings such as statement_timeout)",[500,3248,3249],{},[2258,3250,3253],{"href":3251,"rel":3252},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Ffunctions-json.html",[3238],"PostgreSQL: JSON functions and operators",[500,3255,3256],{},[2258,3257,3260],{"href":3258,"rel":3259},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fsql-expressions.html",[3238],"PostgreSQL: aggregate expressions and FILTER",[500,3262,3263],{},[2258,3264,3267],{"href":3265,"rel":3266},"https:\u002F\u002Fwww.psycopg.org\u002Fdocs\u002Fextras.html",[3238],"psycopg2.extras: execute_values and Json",[500,3269,3270],{},[2258,3271,3274],{"href":3272,"rel":3273},"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc6585#section-4",[3238],"RFC 6585, section 4: 429 Too Many Requests",[500,3276,3277],{},[2258,3278,3281],{"href":3279,"rel":3280},"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc8288",[3238],"RFC 8288: Web Linking (the Link header)",[500,3283,3284],{},[2258,3285,3288],{"href":3286,"rel":3287},"https:\u002F\u002Fman7.org\u002Flinux\u002Fman-pages\u002Fman5\u002Fcrontab.5.html",[3238],"crontab(5): MAILTO",[3290,3291,3292],"style",{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}",{"title":130,"searchDepth":164,"depth":164,"links":3294},[3295,3296,3297,3298,3299,3300],{"id":20,"depth":164,"text":21},{"id":30,"depth":164,"text":31},{"id":488,"depth":164,"text":489},{"id":2460,"depth":164,"text":2461},{"id":3215,"depth":164,"text":3216},{"id":3227,"depth":164,"text":3228},"techcolumnist",[3303,3304],"engineering","strategy","2026-05-20T14:00:00Z","A practical build for an IT operations reporting layer with no BI team: one Postgres database, raw\u002Fstaging\u002Fmarts schemas, an API loader script, and freshness checks.","md",false,null,{},"\u002Fblog\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team",{"title":6,"description":3306},[3301],"blog\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team",[3316,3317,129,752],"data-warehouse","reporting","\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team\u002F","DLQfIyS0Q1xDgNWWP71peIkG6pQqm7yaE09GE77AlbE",{"title":3321,"description":3322,"date":3323,"url":3222,"categories":3324,"tags":3325,"image":3309,"readingTime":262,"canonical":3301,"sites":3326,"series":3309,"seriesOrder":3309},"Data Warehouse: When to Stop Querying Production and Build a Warehouse","How to measure reporting load on a production Postgres database, the cheap fixes to try first, and the signals that mean a separate warehouse is finally justified.","2026-05-27T14:00:00Z",[3303,3304],[3316,129,752],[3301],{"title":3328,"description":3329,"date":3330,"url":3331,"categories":3332,"tags":3333,"image":3309,"readingTime":244,"canonical":3301,"sites":3335,"series":3309,"seriesOrder":3309},"Google Cloud: Organization Policy Constraints That Actually Matter","Which Google Cloud organization policy constraints prevent real incidents, with policy YAML, the dry-run workflow, tag-based exemptions, and an audit script.","2026-05-13T14:00:00Z","\u002F2026\u002F05\u002F13\u002Fgoogle-cloud-organization-policy-constraints-that-actually-matter\u002F",[3303],[3334],"gcloud",[3301],[3320,3337,3344],{"title":3338,"description":3339,"date":3340,"url":2260,"categories":3341,"tags":3342,"image":3309,"readingTime":244,"canonical":3301,"sites":3343,"series":3309,"seriesOrder":3309},"Data Warehouse: Reusable SQL Models for IT Operations Metrics","Layered, version-controlled SQL models for MTTR, ticket aging, SLA compliance, and patch compliance, with plain-SQL tests and the dbt equivalent.","2026-08-19T14:00:00Z",[3303],[3316,3317,129],[3301],{"title":3345,"description":3346,"date":3347,"url":3348,"categories":3349,"tags":3351,"image":3309,"readingTime":180,"canonical":3301,"sites":3353,"series":3309,"seriesOrder":3309},"Python: Data Warehouse – Catching Schema Drift Before an ETL Run","A Python pre-flight check that snapshots Postgres column definitions from information_schema and fails the ETL run when a column is dropped, retyped, or resized.","2026-01-21T14:00:00Z","\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run\u002F",[3350,3303],"scripts",[752,3316,3352,129],"etl",[3301],{"doc":3309,"posts":3355},[],1790052513938]