[{"data":1,"prerenderedAt":2363},["ShallowReactive",2],{"post:\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse\u002F":3},{"post":4,"newer":2324,"older":2334,"related":2342,"series":2361},{"id":5,"title":6,"body":7,"canonical":2306,"categories":2307,"date":2310,"description":2311,"extension":2312,"featured":2313,"hero":2314,"image":2314,"meta":2315,"navigation":735,"path":2316,"readingTime":470,"seo":2317,"series":2314,"seriesOrder":2314,"sites":2318,"source":2314,"stem":2319,"tags":2320,"updated":2314,"url":2322,"__hash__":2323},"blog\u002Fblog\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse.md","Data Warehouse: When to Stop Querying Production and Build a Warehouse",{"type":8,"value":9,"toc":2296},"minimark",[10,14,19,27,31,34,44,77,80,107,121,157,172,255,262,564,572,579,690,694,697,709,1816,1819,1827,1830,1834,1837,1951,1955,1958,1962,1965,1968,2181,2184,2188,2191,2206,2209,2222,2226,2292],[11,12,13],"p",{},"Every organization I've worked with starts the same way: someone needs a report, someone else knows enough SQL to write a query against the production database, and it works. It keeps working for a surprisingly long time, right up until the day it doesn't, and the move from \"querying production is fine\" to \"querying production is actively dangerous\" is gradual enough that most teams cross it without noticing. Building a warehouse is a real project with real cost, and I've seen it justified both far too early and far too late. Here's how I actually decide, with the queries I use to replace gut feel with evidence. The examples use PostgreSQL because that's where I see this most often, but the reasoning carries over to any OLTP database.",[15,16,18],"h2",{"id":17},"the-tell-isnt-data-volume-its-query-shape","The tell isn't data volume, it's query shape",[11,20,21,22,26],{},"The instinct is to think about warehouses in terms of row counts: \"we have ten million rows now, time for a warehouse.\" That's rarely the actual trigger. A well-indexed production table can serve point lookups against ten million rows all day without anyone noticing. What breaks things is query ",[23,24,25],"em",{},"shape",": a report that scans a large fraction of a table, joins across several large tables, or aggregates over a long time window. That kind of query competes with the transactional workload for the same buffer cache and the same I\u002FO. A full scan against a hot table during business hours is what shows up as latency spikes for everyone else using the application at that moment.",[15,28,30],{"id":29},"measure-it-before-you-argue-about-it","Measure it before you argue about it",[11,32,33],{},"The conversation gets much shorter once you can show which queries are doing the work and who is running them. Two built-in tools cover most of it.",[11,35,36,43],{},[37,38,39],"strong",{},[40,41,42],"code",{},"pg_stat_statements"," keeps cumulative execution statistics for every normalized statement. It has to be preloaded, which means a restart:",[45,46,51],"pre",{"className":47,"code":48,"language":49,"meta":50,"style":50},"language-ini shiki shiki-themes github-dark","# postgresql.conf\nshared_preload_libraries = 'pg_stat_statements'\n","ini","",[40,52,53,62],{"__ignoreMap":50},[54,55,58],"span",{"class":56,"line":57},"line",1,[54,59,61],{"class":60},"sAwPA","# postgresql.conf\n",[54,63,65,69,73],{"class":56,"line":64},2,[54,66,68],{"class":67},"snl16","shared_preload_libraries",[54,70,72],{"class":71},"s95oV"," = ",[54,74,76],{"class":75},"sU2Wk","'pg_stat_statements'\n",[11,78,79],{},"Then create it in the database you want to watch:",[45,81,85],{"className":82,"code":83,"language":84,"meta":50,"style":50},"language-sql shiki shiki-themes github-dark","CREATE EXTENSION IF NOT EXISTS pg_stat_statements;\n","sql",[40,86,87],{"__ignoreMap":50},[54,88,89,92,95,98,101,104],{"class":56,"line":57},[54,90,91],{"class":67},"CREATE",[54,93,94],{"class":71}," EXTENSION ",[54,96,97],{"class":67},"IF",[54,99,100],{"class":67}," NOT",[54,102,103],{"class":67}," EXISTS",[54,105,106],{"class":71}," pg_stat_statements;\n",[11,108,109,112,113,116,117,120],{},[37,110,111],{},"Slow query logging"," catches individual long runs, including the one-off ad hoc query that never repeats often enough to top the cumulative stats. ",[40,114,115],{},"log_min_duration_statement"," logs every completed statement that ran at least that long. It's off (",[40,118,119],{},"-1",") by default, and it can be changed with a reload:",[45,122,124],{"className":82,"code":123,"language":84,"meta":50,"style":50},"ALTER SYSTEM SET log_min_duration_statement = '2s';\nSELECT pg_reload_conf();\n",[40,125,126,149],{"__ignoreMap":50},[54,127,128,131,134,137,140,143,146],{"class":56,"line":57},[54,129,130],{"class":67},"ALTER",[54,132,133],{"class":67}," SYSTEM",[54,135,136],{"class":67}," SET",[54,138,139],{"class":71}," log_min_duration_statement ",[54,141,142],{"class":67},"=",[54,144,145],{"class":75}," '2s'",[54,147,148],{"class":71},";\n",[54,150,151,154],{"class":56,"line":64},[54,152,153],{"class":67},"SELECT",[54,155,156],{"class":71}," pg_reload_conf();\n",[11,158,159,160,163,164,167,168,171],{},"The most useful single step is giving reporting its ",[37,161,162],{},"own login role",", even if it's the same people. Every statistic below can then be split by role, and you get a place to hang guardrails. ",[40,165,166],{},"pg_read_all_data"," (PostgreSQL 14 and later) grants read access to every table, view, and sequence without handing out ownership. ",[40,169,170],{},"ALTER ROLE ... SET"," puts a hard ceiling on how long any one reporting query can run:",[45,173,175],{"className":82,"code":174,"language":84,"meta":50,"style":50},"CREATE ROLE reporting LOGIN PASSWORD '\u003Cpassword>';\nGRANT pg_read_all_data TO reporting;\nALTER ROLE reporting SET statement_timeout = '60s';\nALTER ROLE reporting SET default_transaction_read_only = on;\n",[40,176,177,198,212,234],{"__ignoreMap":50},[54,178,179,181,184,187,190,193,196],{"class":56,"line":57},[54,180,91],{"class":67},[54,182,183],{"class":67}," ROLE",[54,185,186],{"class":71}," reporting ",[54,188,189],{"class":67},"LOGIN",[54,191,192],{"class":67}," PASSWORD",[54,194,195],{"class":75}," '\u003Cpassword>'",[54,197,148],{"class":71},[54,199,200,203,206,209],{"class":56,"line":64},[54,201,202],{"class":67},"GRANT",[54,204,205],{"class":71}," pg_read_all_data ",[54,207,208],{"class":67},"TO",[54,210,211],{"class":71}," reporting;\n",[54,213,215,217,219,221,224,227,229,232],{"class":56,"line":214},3,[54,216,130],{"class":67},[54,218,183],{"class":67},[54,220,186],{"class":71},[54,222,223],{"class":67},"SET",[54,225,226],{"class":71}," statement_timeout ",[54,228,142],{"class":67},[54,230,231],{"class":75}," '60s'",[54,233,148],{"class":71},[54,235,237,239,241,243,245,248,250,253],{"class":56,"line":236},4,[54,238,130],{"class":67},[54,240,183],{"class":67},[54,242,186],{"class":71},[54,244,223],{"class":67},[54,246,247],{"class":71}," default_transaction_read_only ",[54,249,142],{"class":67},[54,251,252],{"class":67}," on",[54,254,148],{"class":71},[11,256,257,258,261],{},"With that in place, this query shows how much of the server's execution time and disk reads each role accounts for since the stats were last reset. Viewing other roles' query text needs superuser or membership in ",[40,259,260],{},"pg_read_all_stats",".",[45,263,265],{"className":82,"code":264,"language":84,"meta":50,"style":50},"SELECT r.rolname,\n       sum(s.calls)                                        AS calls,\n       round(sum(s.total_exec_time)::numeric \u002F 1000)       AS exec_seconds,\n       round((100 * sum(s.total_exec_time) \u002F sum(sum(s.total_exec_time)) OVER ())::numeric, 1) AS pct_exec_time,\n       sum(s.shared_blks_read)                             AS blocks_read_from_disk,\n       sum(s.temp_blks_written)                            AS temp_blocks_written\nFROM pg_stat_statements s\nJOIN pg_roles r ON r.oid = s.userid\nWHERE s.dbid = (SELECT oid FROM pg_database WHERE datname = current_database())\nGROUP BY r.rolname\nORDER BY exec_seconds DESC;\n",[40,266,267,283,308,347,415,437,459,468,498,537,550],{"__ignoreMap":50},[54,268,269,271,275,277,280],{"class":56,"line":57},[54,270,153],{"class":67},[54,272,274],{"class":273},"sDLfK"," r",[54,276,261],{"class":71},[54,278,279],{"class":273},"rolname",[54,281,282],{"class":71},",\n",[54,284,285,288,291,294,296,299,302,305],{"class":56,"line":64},[54,286,287],{"class":273},"       sum",[54,289,290],{"class":71},"(",[54,292,293],{"class":273},"s",[54,295,261],{"class":71},[54,297,298],{"class":273},"calls",[54,300,301],{"class":71},")                                        ",[54,303,304],{"class":67},"AS",[54,306,307],{"class":71}," calls,\n",[54,309,310,313,315,318,320,322,324,327,330,333,336,339,342,344],{"class":56,"line":214},[54,311,312],{"class":273},"       round",[54,314,290],{"class":71},[54,316,317],{"class":273},"sum",[54,319,290],{"class":71},[54,321,293],{"class":273},[54,323,261],{"class":71},[54,325,326],{"class":273},"total_exec_time",[54,328,329],{"class":71},")::",[54,331,332],{"class":67},"numeric",[54,334,335],{"class":67}," \u002F",[54,337,338],{"class":273}," 1000",[54,340,341],{"class":71},")       ",[54,343,304],{"class":67},[54,345,346],{"class":71}," exec_seconds,\n",[54,348,349,351,354,357,360,363,365,367,369,371,374,377,379,381,383,385,387,389,391,394,397,400,402,405,408,410,412],{"class":56,"line":236},[54,350,312],{"class":273},[54,352,353],{"class":71},"((",[54,355,356],{"class":273},"100",[54,358,359],{"class":67}," *",[54,361,362],{"class":273}," sum",[54,364,290],{"class":71},[54,366,293],{"class":273},[54,368,261],{"class":71},[54,370,326],{"class":273},[54,372,373],{"class":71},") ",[54,375,376],{"class":67},"\u002F",[54,378,362],{"class":273},[54,380,290],{"class":71},[54,382,317],{"class":273},[54,384,290],{"class":71},[54,386,293],{"class":273},[54,388,261],{"class":71},[54,390,326],{"class":273},[54,392,393],{"class":71},")) ",[54,395,396],{"class":67},"OVER",[54,398,399],{"class":71}," ())::",[54,401,332],{"class":67},[54,403,404],{"class":71},", ",[54,406,407],{"class":273},"1",[54,409,373],{"class":71},[54,411,304],{"class":67},[54,413,414],{"class":71}," pct_exec_time,\n",[54,416,418,420,422,424,426,429,432,434],{"class":56,"line":417},5,[54,419,287],{"class":273},[54,421,290],{"class":71},[54,423,293],{"class":273},[54,425,261],{"class":71},[54,427,428],{"class":273},"shared_blks_read",[54,430,431],{"class":71},")                             ",[54,433,304],{"class":67},[54,435,436],{"class":71}," blocks_read_from_disk,\n",[54,438,440,442,444,446,448,451,454,456],{"class":56,"line":439},6,[54,441,287],{"class":273},[54,443,290],{"class":71},[54,445,293],{"class":273},[54,447,261],{"class":71},[54,449,450],{"class":273},"temp_blks_written",[54,452,453],{"class":71},")                            ",[54,455,304],{"class":67},[54,457,458],{"class":71}," temp_blocks_written\n",[54,460,462,465],{"class":56,"line":461},7,[54,463,464],{"class":67},"FROM",[54,466,467],{"class":71}," pg_stat_statements s\n",[54,469,471,474,477,480,482,484,487,490,493,495],{"class":56,"line":470},8,[54,472,473],{"class":67},"JOIN",[54,475,476],{"class":71}," pg_roles r ",[54,478,479],{"class":67},"ON",[54,481,274],{"class":273},[54,483,261],{"class":71},[54,485,486],{"class":273},"oid",[54,488,489],{"class":67}," =",[54,491,492],{"class":273}," s",[54,494,261],{"class":71},[54,496,497],{"class":273},"userid\n",[54,499,501,504,506,508,511,513,516,518,521,524,527,529,532,534],{"class":56,"line":500},9,[54,502,503],{"class":67},"WHERE",[54,505,492],{"class":273},[54,507,261],{"class":71},[54,509,510],{"class":273},"dbid",[54,512,489],{"class":67},[54,514,515],{"class":71}," (",[54,517,153],{"class":67},[54,519,520],{"class":67}," oid",[54,522,523],{"class":67}," FROM",[54,525,526],{"class":71}," pg_database ",[54,528,503],{"class":67},[54,530,531],{"class":71}," datname ",[54,533,142],{"class":67},[54,535,536],{"class":71}," current_database())\n",[54,538,540,543,545,547],{"class":56,"line":539},10,[54,541,542],{"class":67},"GROUP BY",[54,544,274],{"class":273},[54,546,261],{"class":71},[54,548,549],{"class":273},"rolname\n",[54,551,553,556,559,562],{"class":56,"line":552},11,[54,554,555],{"class":67},"ORDER BY",[54,557,558],{"class":71}," exec_seconds ",[54,560,561],{"class":67},"DESC",[54,563,148],{"class":71},[11,565,566,568,569,571],{},[40,567,428],{}," counts blocks that weren't already in shared buffers, which is a decent proxy for \"this query is pushing the application's working set out of cache.\" ",[40,570,450],{}," means sorts or hashes spilled to disk, the classic sign of a big aggregate or join.",[11,573,574,575,578],{},"To catch it in the act, check ",[40,576,577],{},"pg_stat_activity"," for long-running active queries:",[45,580,582],{"className":82,"code":581,"language":84,"meta":50,"style":50},"SELECT pid, usename, now() - query_start AS runtime, wait_event_type, wait_event,\n       left(query, 100) AS query\nFROM pg_stat_activity\nWHERE backend_type = 'client backend'\n  AND state = 'active'\n  AND now() - query_start > interval '30 seconds'\nORDER BY runtime DESC;\n",[40,583,584,608,625,632,644,657,679],{"__ignoreMap":50},[54,585,586,588,591,594,597,600,603,605],{"class":56,"line":57},[54,587,153],{"class":67},[54,589,590],{"class":71}," pid, usename, ",[54,592,593],{"class":67},"now",[54,595,596],{"class":71},"() ",[54,598,599],{"class":67},"-",[54,601,602],{"class":71}," query_start ",[54,604,304],{"class":67},[54,606,607],{"class":71}," runtime, wait_event_type, wait_event,\n",[54,609,610,613,616,618,620,622],{"class":56,"line":64},[54,611,612],{"class":273},"       left",[54,614,615],{"class":71},"(query, ",[54,617,356],{"class":273},[54,619,373],{"class":71},[54,621,304],{"class":67},[54,623,624],{"class":71}," query\n",[54,626,627,629],{"class":56,"line":214},[54,628,464],{"class":67},[54,630,631],{"class":71}," pg_stat_activity\n",[54,633,634,636,639,641],{"class":56,"line":236},[54,635,503],{"class":67},[54,637,638],{"class":71}," backend_type ",[54,640,142],{"class":67},[54,642,643],{"class":75}," 'client backend'\n",[54,645,646,649,652,654],{"class":56,"line":417},[54,647,648],{"class":67},"  AND",[54,650,651],{"class":67}," state",[54,653,489],{"class":67},[54,655,656],{"class":75}," 'active'\n",[54,658,659,661,664,666,668,670,673,676],{"class":56,"line":439},[54,660,648],{"class":67},[54,662,663],{"class":67}," now",[54,665,596],{"class":71},[54,667,599],{"class":67},[54,669,602],{"class":71},[54,671,672],{"class":67},">",[54,674,675],{"class":71}," interval ",[54,677,678],{"class":75},"'30 seconds'\n",[54,680,681,683,686,688],{"class":56,"line":461},[54,682,555],{"class":67},[54,684,685],{"class":71}," runtime ",[54,687,561],{"class":67},[54,689,148],{"class":71},[15,691,693],{"id":692},"a-script-to-run-weekly","A script to run weekly",[11,695,696],{},"I run this every week (cron or a CI schedule) and paste the output into the thread whenever someone asks \"is reporting hurting production?\" It reports each role's share of execution time and disk reads, the top statements for the reporting roles, and anything running long right now.",[11,698,699,700,703,704,706,707,261],{},"It needs Python 3.10+, ",[40,701,702],{},"psycopg2-binary",", the ",[40,705,42],{}," extension, and a role with ",[40,708,260],{},[45,710,714],{"className":711,"code":712,"language":713,"meta":50,"style":50},"language-python shiki shiki-themes github-dark","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"\nreporting_pressure.py\n\nSummarizes how much of a PostgreSQL database's execution time and disk reads\ncome from reporting roles, using pg_stat_statements and pg_stat_activity.\n\nUsage:\n    python reporting_pressure.py --dsn \u003Cdsn> --reporting-roles reporting,metabase [--top 10]\nNeeds: pg_stat_statements installed in the database; a role with pg_read_all_stats.\n\"\"\"\n\nimport argparse\nimport sys\n\nimport psycopg2\n\nDB_FILTER = \"s.dbid = (SELECT oid FROM pg_database WHERE datname = current_database())\"\n\nSHARE_SQL = f\"\"\"\n    SELECT r.rolname = ANY(%s)          AS is_reporting,\n           sum(s.total_exec_time)       AS exec_ms,\n           sum(s.shared_blks_read)      AS blks_read,\n           sum(s.temp_blks_written)     AS temp_written\n    FROM pg_stat_statements s\n    JOIN pg_roles r ON r.oid = s.userid\n    WHERE {DB_FILTER}\n    GROUP BY 1\n\"\"\"\n\nTOP_SQL = f\"\"\"\n    SELECT r.rolname, s.calls,\n           round(s.total_exec_time::numeric \u002F 1000, 1) AS total_s,\n           round(s.mean_exec_time::numeric, 1)         AS mean_ms,\n           s.shared_blks_read,\n           left(regexp_replace(s.query, '\\\\s+', ' ', 'g'), 70) AS query\n    FROM pg_stat_statements s\n    JOIN pg_roles r ON r.oid = s.userid\n    WHERE {DB_FILTER} AND r.rolname = ANY(%s)\n    ORDER BY s.total_exec_time DESC\n    LIMIT %s\n\"\"\"\n\nLONG_SQL = \"\"\"\n    SELECT pid, usename, date_trunc('second', now() - query_start) AS runtime,\n           left(regexp_replace(query, '\\\\s+', ' ', 'g'), 70) AS query\n    FROM pg_stat_activity\n    WHERE backend_type = 'client backend' AND state = 'active'\n      AND now() - query_start > interval '30 seconds'\n    ORDER BY runtime DESC\n\"\"\"\n\n\ndef pct(part, whole):\n    return 0.0 if not whole else 100.0 * float(part) \u002F float(whole)\n\n\ndef main() -> int:\n    parser = argparse.ArgumentParser(description=\"Measure reporting load on a PostgreSQL database.\")\n    parser.add_argument(\"--dsn\", required=True)\n    parser.add_argument(\"--reporting-roles\", required=True, help=\"Comma-separated role names\")\n    parser.add_argument(\"--top\", type=int, default=10)\n    args = parser.parse_args()\n    roles = [r.strip() for r in args.reporting_roles.split(\",\") if r.strip()]\n\n    conn = psycopg2.connect(args.dsn)\n    try:\n        with conn, conn.cursor() as cur:\n            cur.execute(SHARE_SQL, (roles,))\n            totals = {row[0]: row[1:] for row in cur.fetchall()}\n            rep = totals.get(True, (0, 0, 0))\n            other = totals.get(False, (0, 0, 0))\n            print(\"Share of work from reporting roles since last stats reset:\")\n            for i, label in enumerate((\"execution time\", \"blocks read from disk\", \"temp blocks written\")):\n                print(f\"  {label:\u003C22} {pct(rep[i], rep[i] + other[i]):5.1f}%\")\n\n            cur.execute(TOP_SQL, (roles, args.top))\n            print(f\"\\nTop {args.top} reporting statements by total time:\")\n            for rolname, calls, total_s, mean_ms, blks, query in cur.fetchall():\n                print(f\"  {rolname:\u003C12} {calls:>8} calls {total_s:>9}s total {mean_ms:>9}ms avg {blks:>10} blks  {query}\")\n\n            cur.execute(LONG_SQL)\n            running = cur.fetchall()\n            print(f\"\\nActive queries over 30s right now: {len(running)}\")\n            for pid, user, runtime, query in running:\n                print(f\"  pid {pid} {user} {runtime}  {query}\")\n    finally:\n        conn.close()\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n","python",[40,715,716,721,726,731,737,742,747,751,756,761,766,770,775,784,792,797,805,810,821,826,839,845,851,857,863,869,875,884,890,895,900,912,918,924,930,936,948,953,958,969,975,981,986,991,1002,1008,1018,1024,1030,1036,1042,1047,1052,1057,1070,1109,1114,1119,1136,1159,1180,1208,1237,1248,1282,1287,1298,1306,1321,1332,1364,1393,1420,1433,1465,1513,1518,1528,1558,1571,1652,1657,1666,1677,1705,1718,1762,1770,1776,1784,1789,1794,1810],{"__ignoreMap":50},[54,717,718],{"class":56,"line":57},[54,719,720],{"class":60},"#!\u002Fusr\u002Fbin\u002Fenv python3\n",[54,722,723],{"class":56,"line":64},[54,724,725],{"class":75},"\"\"\"\n",[54,727,728],{"class":56,"line":214},[54,729,730],{"class":75},"reporting_pressure.py\n",[54,732,733],{"class":56,"line":236},[54,734,736],{"emptyLinePlaceholder":735},true,"\n",[54,738,739],{"class":56,"line":417},[54,740,741],{"class":75},"Summarizes how much of a PostgreSQL database's execution time and disk reads\n",[54,743,744],{"class":56,"line":439},[54,745,746],{"class":75},"come from reporting roles, using pg_stat_statements and pg_stat_activity.\n",[54,748,749],{"class":56,"line":461},[54,750,736],{"emptyLinePlaceholder":735},[54,752,753],{"class":56,"line":470},[54,754,755],{"class":75},"Usage:\n",[54,757,758],{"class":56,"line":500},[54,759,760],{"class":75},"    python reporting_pressure.py --dsn \u003Cdsn> --reporting-roles reporting,metabase [--top 10]\n",[54,762,763],{"class":56,"line":539},[54,764,765],{"class":75},"Needs: pg_stat_statements installed in the database; a role with pg_read_all_stats.\n",[54,767,768],{"class":56,"line":552},[54,769,725],{"class":75},[54,771,773],{"class":56,"line":772},12,[54,774,736],{"emptyLinePlaceholder":735},[54,776,778,781],{"class":56,"line":777},13,[54,779,780],{"class":67},"import",[54,782,783],{"class":71}," argparse\n",[54,785,787,789],{"class":56,"line":786},14,[54,788,780],{"class":67},[54,790,791],{"class":71}," sys\n",[54,793,795],{"class":56,"line":794},15,[54,796,736],{"emptyLinePlaceholder":735},[54,798,800,802],{"class":56,"line":799},16,[54,801,780],{"class":67},[54,803,804],{"class":71}," psycopg2\n",[54,806,808],{"class":56,"line":807},17,[54,809,736],{"emptyLinePlaceholder":735},[54,811,813,816,818],{"class":56,"line":812},18,[54,814,815],{"class":273},"DB_FILTER",[54,817,489],{"class":67},[54,819,820],{"class":75}," \"s.dbid = (SELECT oid FROM pg_database WHERE datname = current_database())\"\n",[54,822,824],{"class":56,"line":823},19,[54,825,736],{"emptyLinePlaceholder":735},[54,827,829,832,834,837],{"class":56,"line":828},20,[54,830,831],{"class":273},"SHARE_SQL",[54,833,489],{"class":67},[54,835,836],{"class":67}," f",[54,838,725],{"class":75},[54,840,842],{"class":56,"line":841},21,[54,843,844],{"class":75},"    SELECT r.rolname = ANY(%s)          AS is_reporting,\n",[54,846,848],{"class":56,"line":847},22,[54,849,850],{"class":75},"           sum(s.total_exec_time)       AS exec_ms,\n",[54,852,854],{"class":56,"line":853},23,[54,855,856],{"class":75},"           sum(s.shared_blks_read)      AS blks_read,\n",[54,858,860],{"class":56,"line":859},24,[54,861,862],{"class":75},"           sum(s.temp_blks_written)     AS temp_written\n",[54,864,866],{"class":56,"line":865},25,[54,867,868],{"class":75},"    FROM pg_stat_statements s\n",[54,870,872],{"class":56,"line":871},26,[54,873,874],{"class":75},"    JOIN pg_roles r ON r.oid = s.userid\n",[54,876,878,881],{"class":56,"line":877},27,[54,879,880],{"class":75},"    WHERE ",[54,882,883],{"class":273},"{DB_FILTER}\n",[54,885,887],{"class":56,"line":886},28,[54,888,889],{"class":75},"    GROUP BY 1\n",[54,891,893],{"class":56,"line":892},29,[54,894,725],{"class":75},[54,896,898],{"class":56,"line":897},30,[54,899,736],{"emptyLinePlaceholder":735},[54,901,903,906,908,910],{"class":56,"line":902},31,[54,904,905],{"class":273},"TOP_SQL",[54,907,489],{"class":67},[54,909,836],{"class":67},[54,911,725],{"class":75},[54,913,915],{"class":56,"line":914},32,[54,916,917],{"class":75},"    SELECT r.rolname, s.calls,\n",[54,919,921],{"class":56,"line":920},33,[54,922,923],{"class":75},"           round(s.total_exec_time::numeric \u002F 1000, 1) AS total_s,\n",[54,925,927],{"class":56,"line":926},34,[54,928,929],{"class":75},"           round(s.mean_exec_time::numeric, 1)         AS mean_ms,\n",[54,931,933],{"class":56,"line":932},35,[54,934,935],{"class":75},"           s.shared_blks_read,\n",[54,937,939,942,945],{"class":56,"line":938},36,[54,940,941],{"class":75},"           left(regexp_replace(s.query, '",[54,943,944],{"class":273},"\\\\",[54,946,947],{"class":75},"s+', ' ', 'g'), 70) AS query\n",[54,949,951],{"class":56,"line":950},37,[54,952,868],{"class":75},[54,954,956],{"class":56,"line":955},38,[54,957,874],{"class":75},[54,959,961,963,966],{"class":56,"line":960},39,[54,962,880],{"class":75},[54,964,965],{"class":273},"{DB_FILTER}",[54,967,968],{"class":75}," AND r.rolname = ANY(%s)\n",[54,970,972],{"class":56,"line":971},40,[54,973,974],{"class":75},"    ORDER BY s.total_exec_time DESC\n",[54,976,978],{"class":56,"line":977},41,[54,979,980],{"class":75},"    LIMIT %s\n",[54,982,984],{"class":56,"line":983},42,[54,985,725],{"class":75},[54,987,989],{"class":56,"line":988},43,[54,990,736],{"emptyLinePlaceholder":735},[54,992,994,997,999],{"class":56,"line":993},44,[54,995,996],{"class":273},"LONG_SQL",[54,998,489],{"class":67},[54,1000,1001],{"class":75}," \"\"\"\n",[54,1003,1005],{"class":56,"line":1004},45,[54,1006,1007],{"class":75},"    SELECT pid, usename, date_trunc('second', now() - query_start) AS runtime,\n",[54,1009,1011,1014,1016],{"class":56,"line":1010},46,[54,1012,1013],{"class":75},"           left(regexp_replace(query, '",[54,1015,944],{"class":273},[54,1017,947],{"class":75},[54,1019,1021],{"class":56,"line":1020},47,[54,1022,1023],{"class":75},"    FROM pg_stat_activity\n",[54,1025,1027],{"class":56,"line":1026},48,[54,1028,1029],{"class":75},"    WHERE backend_type = 'client backend' AND state = 'active'\n",[54,1031,1033],{"class":56,"line":1032},49,[54,1034,1035],{"class":75},"      AND now() - query_start > interval '30 seconds'\n",[54,1037,1039],{"class":56,"line":1038},50,[54,1040,1041],{"class":75},"    ORDER BY runtime DESC\n",[54,1043,1045],{"class":56,"line":1044},51,[54,1046,725],{"class":75},[54,1048,1050],{"class":56,"line":1049},52,[54,1051,736],{"emptyLinePlaceholder":735},[54,1053,1055],{"class":56,"line":1054},53,[54,1056,736],{"emptyLinePlaceholder":735},[54,1058,1060,1063,1067],{"class":56,"line":1059},54,[54,1061,1062],{"class":67},"def",[54,1064,1066],{"class":1065},"svObZ"," pct",[54,1068,1069],{"class":71},"(part, whole):\n",[54,1071,1073,1076,1079,1082,1085,1088,1091,1094,1096,1099,1102,1104,1106],{"class":56,"line":1072},55,[54,1074,1075],{"class":67},"    return",[54,1077,1078],{"class":273}," 0.0",[54,1080,1081],{"class":67}," if",[54,1083,1084],{"class":67}," not",[54,1086,1087],{"class":71}," whole ",[54,1089,1090],{"class":67},"else",[54,1092,1093],{"class":273}," 100.0",[54,1095,359],{"class":67},[54,1097,1098],{"class":273}," float",[54,1100,1101],{"class":71},"(part) ",[54,1103,376],{"class":67},[54,1105,1098],{"class":273},[54,1107,1108],{"class":71},"(whole)\n",[54,1110,1112],{"class":56,"line":1111},56,[54,1113,736],{"emptyLinePlaceholder":735},[54,1115,1117],{"class":56,"line":1116},57,[54,1118,736],{"emptyLinePlaceholder":735},[54,1120,1122,1124,1127,1130,1133],{"class":56,"line":1121},58,[54,1123,1062],{"class":67},[54,1125,1126],{"class":1065}," main",[54,1128,1129],{"class":71},"() -> ",[54,1131,1132],{"class":273},"int",[54,1134,1135],{"class":71},":\n",[54,1137,1139,1142,1144,1147,1151,1153,1156],{"class":56,"line":1138},59,[54,1140,1141],{"class":71},"    parser ",[54,1143,142],{"class":67},[54,1145,1146],{"class":71}," argparse.ArgumentParser(",[54,1148,1150],{"class":1149},"s9osk","description",[54,1152,142],{"class":67},[54,1154,1155],{"class":75},"\"Measure reporting load on a PostgreSQL database.\"",[54,1157,1158],{"class":71},")\n",[54,1160,1162,1165,1168,1170,1173,1175,1178],{"class":56,"line":1161},60,[54,1163,1164],{"class":71},"    parser.add_argument(",[54,1166,1167],{"class":75},"\"--dsn\"",[54,1169,404],{"class":71},[54,1171,1172],{"class":1149},"required",[54,1174,142],{"class":67},[54,1176,1177],{"class":273},"True",[54,1179,1158],{"class":71},[54,1181,1183,1185,1188,1190,1192,1194,1196,1198,1201,1203,1206],{"class":56,"line":1182},61,[54,1184,1164],{"class":71},[54,1186,1187],{"class":75},"\"--reporting-roles\"",[54,1189,404],{"class":71},[54,1191,1172],{"class":1149},[54,1193,142],{"class":67},[54,1195,1177],{"class":273},[54,1197,404],{"class":71},[54,1199,1200],{"class":1149},"help",[54,1202,142],{"class":67},[54,1204,1205],{"class":75},"\"Comma-separated role names\"",[54,1207,1158],{"class":71},[54,1209,1211,1213,1216,1218,1221,1223,1225,1227,1230,1232,1235],{"class":56,"line":1210},62,[54,1212,1164],{"class":71},[54,1214,1215],{"class":75},"\"--top\"",[54,1217,404],{"class":71},[54,1219,1220],{"class":1149},"type",[54,1222,142],{"class":67},[54,1224,1132],{"class":273},[54,1226,404],{"class":71},[54,1228,1229],{"class":1149},"default",[54,1231,142],{"class":67},[54,1233,1234],{"class":273},"10",[54,1236,1158],{"class":71},[54,1238,1240,1243,1245],{"class":56,"line":1239},63,[54,1241,1242],{"class":71},"    args ",[54,1244,142],{"class":67},[54,1246,1247],{"class":71}," parser.parse_args()\n",[54,1249,1251,1254,1256,1259,1262,1265,1268,1271,1274,1276,1279],{"class":56,"line":1250},64,[54,1252,1253],{"class":71},"    roles ",[54,1255,142],{"class":67},[54,1257,1258],{"class":71}," [r.strip() ",[54,1260,1261],{"class":67},"for",[54,1263,1264],{"class":71}," r ",[54,1266,1267],{"class":67},"in",[54,1269,1270],{"class":71}," args.reporting_roles.split(",[54,1272,1273],{"class":75},"\",\"",[54,1275,373],{"class":71},[54,1277,1278],{"class":67},"if",[54,1280,1281],{"class":71}," r.strip()]\n",[54,1283,1285],{"class":56,"line":1284},65,[54,1286,736],{"emptyLinePlaceholder":735},[54,1288,1290,1293,1295],{"class":56,"line":1289},66,[54,1291,1292],{"class":71},"    conn ",[54,1294,142],{"class":67},[54,1296,1297],{"class":71}," psycopg2.connect(args.dsn)\n",[54,1299,1301,1304],{"class":56,"line":1300},67,[54,1302,1303],{"class":67},"    try",[54,1305,1135],{"class":71},[54,1307,1309,1312,1315,1318],{"class":56,"line":1308},68,[54,1310,1311],{"class":67},"        with",[54,1313,1314],{"class":71}," conn, conn.cursor() ",[54,1316,1317],{"class":67},"as",[54,1319,1320],{"class":71}," cur:\n",[54,1322,1324,1327,1329],{"class":56,"line":1323},69,[54,1325,1326],{"class":71},"            cur.execute(",[54,1328,831],{"class":273},[54,1330,1331],{"class":71},", (roles,))\n",[54,1333,1335,1338,1340,1343,1346,1349,1351,1354,1356,1359,1361],{"class":56,"line":1334},70,[54,1336,1337],{"class":71},"            totals ",[54,1339,142],{"class":67},[54,1341,1342],{"class":71}," {row[",[54,1344,1345],{"class":273},"0",[54,1347,1348],{"class":71},"]: row[",[54,1350,407],{"class":273},[54,1352,1353],{"class":71},":] ",[54,1355,1261],{"class":67},[54,1357,1358],{"class":71}," row ",[54,1360,1267],{"class":67},[54,1362,1363],{"class":71}," cur.fetchall()}\n",[54,1365,1367,1370,1372,1375,1377,1380,1382,1384,1386,1388,1390],{"class":56,"line":1366},71,[54,1368,1369],{"class":71},"            rep ",[54,1371,142],{"class":67},[54,1373,1374],{"class":71}," totals.get(",[54,1376,1177],{"class":273},[54,1378,1379],{"class":71},", (",[54,1381,1345],{"class":273},[54,1383,404],{"class":71},[54,1385,1345],{"class":273},[54,1387,404],{"class":71},[54,1389,1345],{"class":273},[54,1391,1392],{"class":71},"))\n",[54,1394,1396,1399,1401,1403,1406,1408,1410,1412,1414,1416,1418],{"class":56,"line":1395},72,[54,1397,1398],{"class":71},"            other ",[54,1400,142],{"class":67},[54,1402,1374],{"class":71},[54,1404,1405],{"class":273},"False",[54,1407,1379],{"class":71},[54,1409,1345],{"class":273},[54,1411,404],{"class":71},[54,1413,1345],{"class":273},[54,1415,404],{"class":71},[54,1417,1345],{"class":273},[54,1419,1392],{"class":71},[54,1421,1423,1426,1428,1431],{"class":56,"line":1422},73,[54,1424,1425],{"class":273},"            print",[54,1427,290],{"class":71},[54,1429,1430],{"class":75},"\"Share of work from reporting roles since last stats reset:\"",[54,1432,1158],{"class":71},[54,1434,1436,1439,1442,1444,1447,1449,1452,1454,1457,1459,1462],{"class":56,"line":1435},74,[54,1437,1438],{"class":67},"            for",[54,1440,1441],{"class":71}," i, label ",[54,1443,1267],{"class":67},[54,1445,1446],{"class":273}," enumerate",[54,1448,353],{"class":71},[54,1450,1451],{"class":75},"\"execution time\"",[54,1453,404],{"class":71},[54,1455,1456],{"class":75},"\"blocks read from disk\"",[54,1458,404],{"class":71},[54,1460,1461],{"class":75},"\"temp blocks written\"",[54,1463,1464],{"class":71},")):\n",[54,1466,1468,1471,1473,1476,1479,1482,1485,1488,1491,1494,1497,1500,1503,1506,1508,1511],{"class":56,"line":1467},75,[54,1469,1470],{"class":273},"                print",[54,1472,290],{"class":71},[54,1474,1475],{"class":67},"f",[54,1477,1478],{"class":75},"\"  ",[54,1480,1481],{"class":273},"{",[54,1483,1484],{"class":71},"label",[54,1486,1487],{"class":67},":\u003C22",[54,1489,1490],{"class":273},"}",[54,1492,1493],{"class":273}," {",[54,1495,1496],{"class":71},"pct(rep[i], rep[i] ",[54,1498,1499],{"class":67},"+",[54,1501,1502],{"class":71}," other[i])",[54,1504,1505],{"class":67},":5.1f",[54,1507,1490],{"class":273},[54,1509,1510],{"class":75},"%\"",[54,1512,1158],{"class":71},[54,1514,1516],{"class":56,"line":1515},76,[54,1517,736],{"emptyLinePlaceholder":735},[54,1519,1521,1523,1525],{"class":56,"line":1520},77,[54,1522,1326],{"class":71},[54,1524,905],{"class":273},[54,1526,1527],{"class":71},", (roles, args.top))\n",[54,1529,1531,1533,1535,1537,1540,1543,1546,1548,1551,1553,1556],{"class":56,"line":1530},78,[54,1532,1425],{"class":273},[54,1534,290],{"class":71},[54,1536,1475],{"class":67},[54,1538,1539],{"class":75},"\"",[54,1541,1542],{"class":273},"\\n",[54,1544,1545],{"class":75},"Top ",[54,1547,1481],{"class":273},[54,1549,1550],{"class":71},"args.top",[54,1552,1490],{"class":273},[54,1554,1555],{"class":75}," reporting statements by total time:\"",[54,1557,1158],{"class":71},[54,1559,1561,1563,1566,1568],{"class":56,"line":1560},79,[54,1562,1438],{"class":67},[54,1564,1565],{"class":71}," rolname, calls, total_s, mean_ms, blks, query ",[54,1567,1267],{"class":67},[54,1569,1570],{"class":71}," cur.fetchall():\n",[54,1572,1574,1576,1578,1580,1582,1584,1586,1589,1591,1593,1595,1598,1600,1603,1605,1608,1611,1613,1616,1618,1621,1623,1625,1628,1630,1633,1636,1638,1641,1643,1646,1648,1650],{"class":56,"line":1573},80,[54,1575,1470],{"class":273},[54,1577,290],{"class":71},[54,1579,1475],{"class":67},[54,1581,1478],{"class":75},[54,1583,1481],{"class":273},[54,1585,279],{"class":71},[54,1587,1588],{"class":67},":\u003C12",[54,1590,1490],{"class":273},[54,1592,1493],{"class":273},[54,1594,298],{"class":71},[54,1596,1597],{"class":67},":>8",[54,1599,1490],{"class":273},[54,1601,1602],{"class":75}," calls ",[54,1604,1481],{"class":273},[54,1606,1607],{"class":71},"total_s",[54,1609,1610],{"class":67},":>9",[54,1612,1490],{"class":273},[54,1614,1615],{"class":75},"s total ",[54,1617,1481],{"class":273},[54,1619,1620],{"class":71},"mean_ms",[54,1622,1610],{"class":67},[54,1624,1490],{"class":273},[54,1626,1627],{"class":75},"ms avg ",[54,1629,1481],{"class":273},[54,1631,1632],{"class":71},"blks",[54,1634,1635],{"class":67},":>10",[54,1637,1490],{"class":273},[54,1639,1640],{"class":75}," blks  ",[54,1642,1481],{"class":273},[54,1644,1645],{"class":71},"query",[54,1647,1490],{"class":273},[54,1649,1539],{"class":75},[54,1651,1158],{"class":71},[54,1653,1655],{"class":56,"line":1654},81,[54,1656,736],{"emptyLinePlaceholder":735},[54,1658,1660,1662,1664],{"class":56,"line":1659},82,[54,1661,1326],{"class":71},[54,1663,996],{"class":273},[54,1665,1158],{"class":71},[54,1667,1669,1672,1674],{"class":56,"line":1668},83,[54,1670,1671],{"class":71},"            running ",[54,1673,142],{"class":67},[54,1675,1676],{"class":71}," cur.fetchall()\n",[54,1678,1680,1682,1684,1686,1688,1690,1693,1696,1699,1701,1703],{"class":56,"line":1679},84,[54,1681,1425],{"class":273},[54,1683,290],{"class":71},[54,1685,1475],{"class":67},[54,1687,1539],{"class":75},[54,1689,1542],{"class":273},[54,1691,1692],{"class":75},"Active queries over 30s right now: ",[54,1694,1695],{"class":273},"{len",[54,1697,1698],{"class":71},"(running)",[54,1700,1490],{"class":273},[54,1702,1539],{"class":75},[54,1704,1158],{"class":71},[54,1706,1708,1710,1713,1715],{"class":56,"line":1707},85,[54,1709,1438],{"class":67},[54,1711,1712],{"class":71}," pid, user, runtime, query ",[54,1714,1267],{"class":67},[54,1716,1717],{"class":71}," running:\n",[54,1719,1721,1723,1725,1727,1730,1732,1735,1737,1739,1742,1744,1746,1749,1751,1754,1756,1758,1760],{"class":56,"line":1720},86,[54,1722,1470],{"class":273},[54,1724,290],{"class":71},[54,1726,1475],{"class":67},[54,1728,1729],{"class":75},"\"  pid ",[54,1731,1481],{"class":273},[54,1733,1734],{"class":71},"pid",[54,1736,1490],{"class":273},[54,1738,1493],{"class":273},[54,1740,1741],{"class":71},"user",[54,1743,1490],{"class":273},[54,1745,1493],{"class":273},[54,1747,1748],{"class":71},"runtime",[54,1750,1490],{"class":273},[54,1752,1753],{"class":273},"  {",[54,1755,1645],{"class":71},[54,1757,1490],{"class":273},[54,1759,1539],{"class":75},[54,1761,1158],{"class":71},[54,1763,1765,1768],{"class":56,"line":1764},87,[54,1766,1767],{"class":67},"    finally",[54,1769,1135],{"class":71},[54,1771,1773],{"class":56,"line":1772},88,[54,1774,1775],{"class":71},"        conn.close()\n",[54,1777,1779,1781],{"class":56,"line":1778},89,[54,1780,1075],{"class":67},[54,1782,1783],{"class":273}," 0\n",[54,1785,1787],{"class":56,"line":1786},90,[54,1788,736],{"emptyLinePlaceholder":735},[54,1790,1792],{"class":56,"line":1791},91,[54,1793,736],{"emptyLinePlaceholder":735},[54,1795,1797,1799,1802,1805,1808],{"class":56,"line":1796},92,[54,1798,1278],{"class":67},[54,1800,1801],{"class":273}," __name__",[54,1803,1804],{"class":67}," ==",[54,1806,1807],{"class":75}," \"__main__\"",[54,1809,1135],{"class":71},[54,1811,1813],{"class":56,"line":1812},93,[54,1814,1815],{"class":71},"    sys.exit(main())\n",[11,1817,1818],{},"Sample output:",[45,1820,1825],{"className":1821,"code":1823,"language":1824,"meta":50},[1822],"language-text","Share of work from reporting roles since last stats reset:\n  execution time          41.3%\n  blocks read from disk   67.8%\n  temp blocks written     88.2%\n\nTop 10 reporting statements by total time:\n  metabase         1440 calls    9312.4s total    6466.9ms avg   48211904 blks  SELECT t.client_id, date_trunc($1, t.created_at) AS month, count(*) ...\n  reporting          96 calls    2210.7s total   23028.1ms avg   20113310 blks  SELECT * FROM tickets t JOIN ticket_events e ON e.ticket_id = t.id ...\n\nActive queries over 30s right now: 1\n  pid 48122 metabase 0:01:12  SELECT t.client_id, date_trunc($1, t.created_at) AS month, count(*) ...\n","text",[40,1826,1823],{"__ignoreMap":50},[11,1828,1829],{},"Those numbers are illustrative. The pattern to look for is a small number of reporting calls accounting for an outsized share of disk reads and temp spill. That shape means a handful of reports is doing most of the damage.",[15,1831,1833],{"id":1832},"try-the-cheap-fixes-first","Try the cheap fixes first",[11,1835,1836],{},"Each of these patches one symptom, and each one is much cheaper than a warehouse:",[1838,1839,1840,1855,1941],"ul",{},[1841,1842,1843,1846,1847,1850,1851,1854],"li",{},[37,1844,1845],{},"A read replica."," Streaming replication gives you a hot standby that accepts read-only queries. The catch is documented plainly: when replaying WAL on the standby conflicts with a running query (vacuum on the primary removing rows the query can still see, for example), the standby waits up to ",[40,1848,1849],{},"max_standby_streaming_delay"," and then cancels the query. Setting ",[40,1852,1853],{},"hot_standby_feedback = on"," prevents those cleanup conflicts, but it delays dead-row cleanup on the primary and can cause table bloat there. Long reports on a replica are a trade-off, not a free lunch.",[1841,1856,1857,1860,1861,404,1864,1867,1868,1871,1872,1937,1940],{},[37,1858,1859],{},"Logical replication of a few tables."," If reports only need ",[40,1862,1863],{},"tickets",[40,1865,1866],{},"clients",", and ",[40,1869,1870],{},"time_entries",", publish just those to a separate reporting database, where you're free to add indexes and materialized views production would never tolerate:",[45,1873,1875],{"className":82,"code":1874,"language":84,"meta":50,"style":50},"-- on the production (publisher) database; requires wal_level = logical.\n-- Each table needs a primary key (or REPLICA IDENTITY), or UPDATE and DELETE on it fail.\n-- repuser needs LOGIN, REPLICATION, SELECT on the tables, and a pg_hba.conf entry.\nCREATE PUBLICATION reporting_pub FOR TABLE tickets, clients, time_entries;\n\n-- on the reporting (subscriber) database, after creating the same tables\nCREATE SUBSCRIPTION reporting_sub\n    CONNECTION 'host=\u003Cprod-host> dbname=app user=repuser password=\u003Cpassword>'\n    PUBLICATION reporting_pub;\n",[40,1876,1877,1882,1887,1892,1908,1912,1917,1924,1932],{"__ignoreMap":50},[54,1878,1879],{"class":56,"line":57},[54,1880,1881],{"class":60},"-- on the production (publisher) database; requires wal_level = logical.\n",[54,1883,1884],{"class":56,"line":64},[54,1885,1886],{"class":60},"-- Each table needs a primary key (or REPLICA IDENTITY), or UPDATE and DELETE on it fail.\n",[54,1888,1889],{"class":56,"line":214},[54,1890,1891],{"class":60},"-- repuser needs LOGIN, REPLICATION, SELECT on the tables, and a pg_hba.conf entry.\n",[54,1893,1894,1896,1899,1902,1905],{"class":56,"line":236},[54,1895,91],{"class":67},[54,1897,1898],{"class":71}," PUBLICATION reporting_pub ",[54,1900,1901],{"class":67},"FOR",[54,1903,1904],{"class":67}," TABLE",[54,1906,1907],{"class":71}," tickets, clients, time_entries;\n",[54,1909,1910],{"class":56,"line":417},[54,1911,736],{"emptyLinePlaceholder":735},[54,1913,1914],{"class":56,"line":439},[54,1915,1916],{"class":60},"-- on the reporting (subscriber) database, after creating the same tables\n",[54,1918,1919,1921],{"class":56,"line":461},[54,1920,91],{"class":67},[54,1922,1923],{"class":71}," SUBSCRIPTION reporting_sub\n",[54,1925,1926,1929],{"class":56,"line":470},[54,1927,1928],{"class":67},"    CONNECTION",[54,1930,1931],{"class":75}," 'host=\u003Cprod-host> dbname=app user=repuser password=\u003Cpassword>'\n",[54,1933,1934],{"class":56,"line":500},[54,1935,1936],{"class":71},"    PUBLICATION reporting_pub;\n",[1938,1939],"br",{},"The PostgreSQL docs point out that schema and DDL changes are not replicated. Every production migration that touches those tables has to be applied to the subscriber by hand.",[1841,1942,1943,1946,1947,1950],{},[37,1944,1945],{},"Materialized views on the replica or reporting database"," for the few heavy aggregates everyone reuses, refreshed on a schedule. ",[40,1948,1949],{},"REFRESH MATERIALIZED VIEW CONCURRENTLY"," keeps readers unblocked, but it requires a unique index on the view that uses plain column names and covers all rows.",[15,1952,1954],{"id":1953},"schema-for-writes-and-schema-for-reading-are-different-jobs","Schema-for-writes and schema-for-reading are different jobs",[11,1956,1957],{},"Production schemas are normalized to make writes safe and consistent, which is exactly the wrong shape for the ad hoc, wide, historical queries reporting wants. I've watched analysts write eight-way joins to answer a question that a denormalized fact table would answer in one. That's not an analyst skill problem. It's a sign the two workloads want different physical layouts of the same data, and serving both from one schema means one of them is always going to be awkward. A replica doesn't fix this, since it has the same schema. Logical replication into a separate database starts to, because you can reshape the data there.",[15,1959,1961],{"id":1960},"historical-questions-are-a-different-signal-than-current-state-questions","Historical questions are a different signal than current-state questions",[11,1963,1964],{},"Production databases are optimized to answer \"what is true right now,\" and most are actively hostile to \"what was true a year ago.\" Old rows get updated in place, soft-deleted, or archived out as normal operational hygiene. Once a question needs trend data, period-over-period comparison, or \"what did this record look like before it changed,\" you're asking production for data that no longer exists. No index fixes that.",[11,1966,1967],{},"The honest workaround is to start capturing history yourself, and the moment you do, you've started a warehouse. For example, a nightly snapshot of open tickets is enough to answer \"how big was the backlog on the first of each month\" going forward:",[45,1969,1971],{"className":82,"code":1970,"language":84,"meta":50,"style":50},"CREATE TABLE IF NOT EXISTS reporting.ticket_daily_snapshot (\n    snapshot_date date NOT NULL,\n    ticket_id     bigint NOT NULL,\n    client_id     bigint,\n    status        text,\n    priority      text,\n    assigned_team text,\n    PRIMARY KEY (snapshot_date, ticket_id)\n);\n\nINSERT INTO reporting.ticket_daily_snapshot\nSELECT current_date, t.id, t.client_id, t.status, t.priority, t.assigned_team\nFROM tickets t\nWHERE t.status NOT IN ('closed', 'cancelled')\nON CONFLICT (snapshot_date, ticket_id) DO NOTHING;\n",[40,1972,1973,1997,2010,2022,2031,2041,2050,2059,2067,2072,2076,2088,2139,2146,2174],{"__ignoreMap":50},[54,1974,1975,1977,1979,1982,1984,1986,1989,1991,1994],{"class":56,"line":57},[54,1976,91],{"class":67},[54,1978,1904],{"class":67},[54,1980,1981],{"class":1065}," IF",[54,1983,100],{"class":67},[54,1985,103],{"class":67},[54,1987,1988],{"class":273}," reporting",[54,1990,261],{"class":71},[54,1992,1993],{"class":273},"ticket_daily_snapshot",[54,1995,1996],{"class":71}," (\n",[54,1998,1999,2002,2005,2008],{"class":56,"line":64},[54,2000,2001],{"class":71},"    snapshot_date ",[54,2003,2004],{"class":67},"date",[54,2006,2007],{"class":67}," NOT NULL",[54,2009,282],{"class":71},[54,2011,2012,2015,2018,2020],{"class":56,"line":214},[54,2013,2014],{"class":71},"    ticket_id     ",[54,2016,2017],{"class":67},"bigint",[54,2019,2007],{"class":67},[54,2021,282],{"class":71},[54,2023,2024,2027,2029],{"class":56,"line":236},[54,2025,2026],{"class":71},"    client_id     ",[54,2028,2017],{"class":67},[54,2030,282],{"class":71},[54,2032,2033,2036,2039],{"class":56,"line":417},[54,2034,2035],{"class":67},"    status",[54,2037,2038],{"class":67},"        text",[54,2040,282],{"class":71},[54,2042,2043,2046,2048],{"class":56,"line":439},[54,2044,2045],{"class":71},"    priority      ",[54,2047,1824],{"class":67},[54,2049,282],{"class":71},[54,2051,2052,2055,2057],{"class":56,"line":461},[54,2053,2054],{"class":71},"    assigned_team ",[54,2056,1824],{"class":67},[54,2058,282],{"class":71},[54,2060,2061,2064],{"class":56,"line":470},[54,2062,2063],{"class":67},"    PRIMARY KEY",[54,2065,2066],{"class":71}," (snapshot_date, ticket_id)\n",[54,2068,2069],{"class":56,"line":500},[54,2070,2071],{"class":71},");\n",[54,2073,2074],{"class":56,"line":539},[54,2075,736],{"emptyLinePlaceholder":735},[54,2077,2078,2081,2083,2085],{"class":56,"line":552},[54,2079,2080],{"class":67},"INSERT INTO",[54,2082,1988],{"class":273},[54,2084,261],{"class":71},[54,2086,2087],{"class":273},"ticket_daily_snapshot\n",[54,2089,2090,2092,2095,2098,2100,2103,2105,2107,2109,2112,2114,2116,2118,2121,2123,2125,2127,2130,2132,2134,2136],{"class":56,"line":772},[54,2091,153],{"class":67},[54,2093,2094],{"class":71}," current_date, ",[54,2096,2097],{"class":273},"t",[54,2099,261],{"class":71},[54,2101,2102],{"class":273},"id",[54,2104,404],{"class":71},[54,2106,2097],{"class":273},[54,2108,261],{"class":71},[54,2110,2111],{"class":273},"client_id",[54,2113,404],{"class":71},[54,2115,2097],{"class":273},[54,2117,261],{"class":71},[54,2119,2120],{"class":273},"status",[54,2122,404],{"class":71},[54,2124,2097],{"class":273},[54,2126,261],{"class":71},[54,2128,2129],{"class":273},"priority",[54,2131,404],{"class":71},[54,2133,2097],{"class":273},[54,2135,261],{"class":71},[54,2137,2138],{"class":273},"assigned_team\n",[54,2140,2141,2143],{"class":56,"line":777},[54,2142,464],{"class":67},[54,2144,2145],{"class":71}," tickets t\n",[54,2147,2148,2150,2153,2155,2157,2159,2162,2164,2167,2169,2172],{"class":56,"line":786},[54,2149,503],{"class":67},[54,2151,2152],{"class":273}," t",[54,2154,261],{"class":71},[54,2156,2120],{"class":273},[54,2158,100],{"class":67},[54,2160,2161],{"class":67}," IN",[54,2163,515],{"class":71},[54,2165,2166],{"class":75},"'closed'",[54,2168,404],{"class":71},[54,2170,2171],{"class":75},"'cancelled'",[54,2173,1158],{"class":71},[54,2175,2176,2178],{"class":56,"line":794},[54,2177,479],{"class":67},[54,2179,2180],{"class":71}," CONFLICT (snapshot_date, ticket_id) DO NOTHING;\n",[11,2182,2183],{},"It can't reconstruct the past, which is exactly why it's worth starting before anyone asks.",[15,2185,2187],{"id":2186},"the-actual-trigger-in-practice","The actual trigger, in practice",[11,2189,2190],{},"The point where I stop resisting a warehouse project is when at least two of these are true at once:",[2192,2193,2194,2200,2203],"ol",{},[1841,2195,2196,2197,2199],{},"Reporting queries show up in the slow query log, or at the top of the ",[40,2198,42],{}," breakdown, against tables the application also writes to, and the replica or timeout patches have already been tried.",[1841,2201,2202],{},"Someone is asking a question that needs data production has already purged or overwritten.",[1841,2204,2205],{},"More than one team wants a version of \"the same\" report with slightly different definitions of the same metric. That's really a request for a shared, agreed-upon source of truth, not another one-off query.",[11,2207,2208],{},"Any one of those alone can usually be patched: a read replica, an archive table, a shared view. All three together mean the patches have stopped being cheaper than the project, and it's time to build the thing properly instead of adding one more workaround to production.",[11,2210,2211,2212,2217,2218,261],{},"When it is time, the warehouse doesn't have to be big. A second Postgres fed by logical replication or scheduled loads works. So does a managed warehouse: BigQuery, for example, doesn't charge for batch load jobs on its default shared slot pool, gives 1 TiB of on-demand query processing free per month, and lists $6.25 per TiB after that in US regions as of this writing (check the pricing page for your region). The loading side is covered in ",[2213,2214,2216],"a",{"href":2215},"\u002F2025\u002F12\u002F17\u002Fpython-data-warehouse-incremental-loads-from-postgres-to-bigquery\u002F","incremental loads from Postgres to BigQuery",". What goes on top of it, when there's no BI team to own it, is in ",[2213,2219,2221],{"href":2220},"\u002F2026\u002F05\u002F20\u002Fdata-warehouse-building-a-reporting-layer-without-a-bi-team\u002F","building a reporting layer without a BI team",[15,2223,2225],{"id":2224},"references","References",[1838,2227,2228,2236,2243,2250,2257,2264,2271,2278,2285],{},[1841,2229,2230],{},[2213,2231,2235],{"href":2232,"rel":2233},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fpgstatstatements.html",[2234],"nofollow","PostgreSQL: pg_stat_statements",[1841,2237,2238],{},[2213,2239,2242],{"href":2240,"rel":2241},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fruntime-config-logging.html",[2234],"PostgreSQL: error reporting and logging (log_min_duration_statement)",[1841,2244,2245],{},[2213,2246,2249],{"href":2247,"rel":2248},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fmonitoring-stats.html",[2234],"PostgreSQL: the cumulative statistics system (pg_stat_activity)",[1841,2251,2252],{},[2213,2253,2256],{"href":2254,"rel":2255},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fpredefined-roles.html",[2234],"PostgreSQL: predefined roles (pg_read_all_data, pg_read_all_stats)",[1841,2258,2259],{},[2213,2260,2263],{"href":2261,"rel":2262},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fhot-standby.html",[2234],"PostgreSQL: hot standby and handling query conflicts",[1841,2265,2266],{},[2213,2267,2270],{"href":2268,"rel":2269},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Flogical-replication-quick-setup.html",[2234],"PostgreSQL: logical replication quick setup",[1841,2272,2273],{},[2213,2274,2277],{"href":2275,"rel":2276},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Flogical-replication-restrictions.html",[2234],"PostgreSQL: logical replication restrictions",[1841,2279,2280],{},[2213,2281,2284],{"href":2282,"rel":2283},"https:\u002F\u002Fwww.postgresql.org\u002Fdocs\u002Fcurrent\u002Fsql-refreshmaterializedview.html",[2234],"PostgreSQL: REFRESH MATERIALIZED VIEW",[1841,2286,2287],{},[2213,2288,2291],{"href":2289,"rel":2290},"https:\u002F\u002Fcloud.google.com\u002Fbigquery\u002Fpricing",[2234],"BigQuery pricing",[2293,2294,2295],"style",{},"html pre.shiki code .snl16, html code.shiki .snl16{--shiki-default:#F97583}html pre.shiki code .s95oV, html code.shiki .s95oV{--shiki-default:#E1E4E8}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html pre.shiki code .sU2Wk, html code.shiki .sU2Wk{--shiki-default:#9ECBFF}html pre.shiki code .sDLfK, html code.shiki .sDLfK{--shiki-default:#79B8FF}html pre.shiki code .sAwPA, html code.shiki .sAwPA{--shiki-default:#6A737D}html pre.shiki code .svObZ, html code.shiki .svObZ{--shiki-default:#B392F0}html pre.shiki code .s9osk, html code.shiki .s9osk{--shiki-default:#FFAB70}",{"title":50,"searchDepth":64,"depth":64,"links":2297},[2298,2299,2300,2301,2302,2303,2304,2305],{"id":17,"depth":64,"text":18},{"id":29,"depth":64,"text":30},{"id":692,"depth":64,"text":693},{"id":1832,"depth":64,"text":1833},{"id":1953,"depth":64,"text":1954},{"id":1960,"depth":64,"text":1961},{"id":2186,"depth":64,"text":2187},{"id":2224,"depth":64,"text":2225},"techcolumnist",[2308,2309],"engineering","strategy","2026-05-27T14:00:00Z","How to measure reporting load on a production Postgres database, the cheap fixes to try first, and the signals that mean a separate warehouse is finally justified.","md",false,null,{},"\u002Fblog\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse",{"title":6,"description":2311},[2306],"blog\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse",[2321,84,713],"data-warehouse","\u002F2026\u002F05\u002F27\u002Fdata-warehouse-when-to-stop-querying-production-and-build-a-warehouse\u002F","qfenP2rXsgUClHzK9sElMfqvEUIAjvBhHZ19FLVOSWI",{"title":2325,"description":2326,"date":2327,"url":2328,"categories":2329,"tags":2330,"image":2314,"readingTime":439,"canonical":2306,"sites":2333,"series":2314,"seriesOrder":2314},"Hyper-V: Deploying a Two-Node Failover Cluster on a Budget","Building a two-node Hyper-V cluster without a SAN: Storage Spaces Direct requirements, nested resiliency, witness options, hardware rules and scripts to build and test it.","2026-06-03T14:00:00Z","\u002F2026\u002F06\u002F03\u002Fhyper-v-deploying-a-two-node-failover-cluster-on-a-budget\u002F",[2308],[2331,2332],"hyper-v","cluster",[2306],{"title":2335,"description":2336,"date":2337,"url":2220,"categories":2338,"tags":2339,"image":2314,"readingTime":439,"canonical":2306,"sites":2341,"series":2314,"seriesOrder":2314},"Data Warehouse: Building a Reporting Layer Without a BI Team","A practical build for an IT operations reporting layer with no BI team: one Postgres database, raw\u002Fstaging\u002Fmarts schemas, an API loader script, and freshness checks.","2026-05-20T14:00:00Z",[2308,2309],[2321,2340,84,713],"reporting",[2306],[2334,2343,2353],{"title":2344,"description":2345,"date":2346,"url":2347,"categories":2348,"tags":2350,"image":2314,"readingTime":214,"canonical":2306,"sites":2352,"series":2314,"seriesOrder":2314},"Python: Data Warehouse – Catching Schema Drift Before an ETL Run","A Python pre-flight check that snapshots Postgres column definitions from information_schema and fails the ETL run when a column is dropped, retyped, or resized.","2026-01-21T14:00:00Z","\u002F2026\u002F01\u002F21\u002Fpython-data-warehouse-catching-schema-drift-before-an-etl-run\u002F",[2349,2308],"scripts",[713,2321,2351,84],"etl",[2306],{"title":2354,"description":2355,"date":2356,"url":2215,"categories":2357,"tags":2358,"image":2314,"readingTime":417,"canonical":2306,"sites":2360,"series":2314,"seriesOrder":2314},"Python: Data Warehouse – Incremental Loads from Postgres to BigQuery","A Python script that copies new and changed Postgres rows into BigQuery with one load job and a MERGE, advancing the watermark in the same transaction.","2025-12-17T14:00:00Z",[2349,2308],[713,2321,2351,2359,84],"gcloud",[2306],{"doc":2314,"posts":2362},[],1790052513958]