How the sum was computed</th>	Result</th>	Off by</th></tr></thead>
Exact (`numeric</code>)</td>`	`2500519211.787503050…</code></td>`	reference</td></tr>
Serial `float8</code>, physical order</td>`	`2500519211.787516</code></td>`	~1.3e-5</td></tr>
Parallel `float8</code>, run A</td>`	`2500519211.7874823</code></td>`	~2.1e-5</td></tr>
Parallel `float8</code>, run B</td>`	`2500519211.787477</code></td>`	~2.6e-5</td></tr>
Ordered by value ascending</td>	`2500519211.7876067</code></td>`	~1.0e-4</td></tr> </tbody></table> Every floating-point total is wrong, including the "stable" serial one. They differ from the exact value in the fifth and sixth decimal places. Parallelism did not introduce the error. It removed the one comforting property the serial sum had, which was getting the same wrong answer every time</em>. That sameness is what people mistake for correctness.</p> Summing smallest-to-largest is the classic textbook advice for reducing floating-point error, and here ascending order happens to land further from the exact value than physical order does. Order changes the error; it does not reliably shrink it. If you need a smaller error there are compensated-summation algorithms (Kahan and friends), but inside a SQL aggregate you do not get to choose them.</div> Where it actually bites</a> </h2> In the sixth decimal of a five-million-row random sum, nobody cares. The problem is the places where someone does.</p> The obvious one is reconciliation. A finance team sums a revenue column stored as `double precision</code>, exports it, and the number disagrees with the same query run an hour later, or with the same logic in a different tool running a different parallel degree. Nothing in the data changed, so the hunt goes to caching, to replicas, to "a dirty read somewhere" long before anyone suspects the aggregate itself.</p>` `The same thing wrecks regression tests, just less visibly. Assert that a float total equals an exact literal, or compare two pipelines for byte-identical sums, and the test will flap. It passes on a small fixture that runs serially, then fails in CI on a larger one that crosses the parallel threshold, with no code change in between. The test is asserting a property the type does not have, and it will keep flapping until someone notices that.</p>` `And anywhere a float sum feeds an equality check, a join, or a GROUP BY</code> key, you are building on sand. Two totals that "should" be equal can differ in their last bits, and now your join silently drops rows.</p>` `Postgres already knows this is dangerous`</a> </h2> The most convincing evidence that this is a real, known hazard and not a corner case is that Postgres deliberately gives up an optimization because of it.</p> Window functions can compute a moving sum efficiently using an inverse transition function</em>: as the frame slides forward, subtract the value leaving the window instead of re-summing the whole frame. It works for numeric</code> and integers. For float4</code> and float8</code>, Postgres refuses to do it. From the documentation on user-defined aggregates</a>:</p> An example of an aggregate for which adding an inverse transition function seems easy at first, yet where this requirement cannot be met is sum</code> over float4</code> or float8</code> inputs.</p> </blockquote> The docs then show why, with the same absorption we saw earlier. If a window holds 1e20</code> and 1</code>, subtracting 1e20</code> as it leaves the frame yields 0</code>, not the 1</code> that should remain. So the built-in float sum simply does not use the inverse optimization, and a moving sum(float8)</code> recomputes the frame to stay correct:</p> SELECT</span> n, </span>sum</span>(x) </span>OVER</span> (</span>ORDER BY</span> n </span>ROWS BETWEEN</span> CURRENT </span>ROW AND</span> 1</span> FOLLOWING</span>)</span></span> FROM</span> (</span>VALUES</span> (</span>1</span>, </span>1</span>.0e20::float8), (</span>2</span>, </span>1</span>.</span>0</span>::float8)) </span>AS</span> v(n, x);</span></span></code></pre> n \| sum</span></span> ---+-------</span></span> 1 \| 1e+20</span></span> 2 \| 1</span></span></code></pre> The second row is 1</code>, correct, because Postgres declined the shortcut that would have made it 0</code>. The engine is quietly working around the imprecision of float sums on your behalf. The non-determinism in the parallel case is the same hazard showing up somewhere the engine cannot paper over it.</p> What to do about it</a> </h2> The fix is almost always the type, not the query.</p> Use numeric</code> for anything anyone reconciles, compares, or reads as a figure.</strong> Money, totals on reports, anything that has to match another system or itself across runs. numeric</code> sums in exact decimal and is order-independent: same rows, same answer, parallel or not, in any order.</p> SET</span> max_parallel_workers_per_gather </span>=</span> 4</span>;</span></span> SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements; </span>-- 2500519211.787503050078417764</span></span> SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements; </span>-- 2500519211.787503050078417764</span></span></code></pre> The cast above stabilizes a single query, but the durable fix is storing the column as numeric</code> in the first place, so the exact values are what live on disk.</p> numeric</code> is not free. It is arbitrary-precision arithmetic done in software, so it is slower to compute and takes more space on disk than a hardware float the CPU adds in a single instruction. For anything that has to reconcile, that is the price of correctness, and it is almost always worth paying. For a column nobody ever sums into a figure a person reads, it usually is not.</p> If the data is genuinely floating point</strong> (sensor readings, scientific measurements, anything where the input is already an approximation), then accept that aggregates over it are approximate and do not assert exact equality on them. Round to a sensible number of decimals before comparing or displaying, and never use a raw float sum as a join or grouping key.</p> Pinning max_parallel_workers_per_gather = 0</code> makes a float sum repeatable, but does not make it correct,</strong> and it costs you parallelism on every other query in the session. It is a way to get a stable wrong answer, not a fix. Reach for the type instead.</p> None of this is Postgres-specific. Any database that parallelizes aggregation, and they all do, has the same behavior over the same IEEE 754 floats. The trigger that makes it visible is simply crossing whatever threshold turns parallelism on, which is why it so often shows up when a table grows past a certain size and "nothing else changed."</p> The NULL in your NOT IN 2026-06-14T23:30:00+00:00 A NOT IN</code> query can return the wrong answer without telling you. It is valid SQL, it runs without an error, and it hands back a perfectly well-formed result set that happens to be empty when it should not be. No warning, no hint, nothing in the logs: just zero rows where you expected hundreds, and a database that considers it correct.</p> Almost always the cause is a single NULL</code> sitting somewhere you forgot to look, combined with two keywords you have typed a thousand times: NOT IN</code>. None of it is a Postgres bug. This is exactly what the SQL standard mandates, implemented faithfully. That is precisely what makes it so easy to walk into, and why the planner could not safely optimize around it for the better part of Postgres's history. It comes down to one if</code> statement in the parser.</p> Sample schema</a> </h2> Nothing elaborate. A table of products, one of which has no category assigned yet, and a table of archived categories that happens to contain a NULL</code>:</p> CREATE TABLE</span> products</span> (id </span>int</span>, category_id </span>int</span>);</span></span> INSERT INTO</span> products </span>VALUES</span> (</span>1</span>, </span>10</span>), (</span>2</span>, </span>20</span>), (</span>3</span>, </span>NULL</span>), (</span>4</span>, </span>10</span>);</span></span> </span> CREATE TABLE</span> archived</span> (category_id </span>int</span>);</span></span> INSERT INTO</span> archived </span>VALUES</span> (</span>20</span>), (</span>NULL</span>);</span></span></code></pre> The NULL</code> in archived</code> is not contrived. The moment a column is nullable (and most are, by default), a NULL</code> can find its way into any subquery you point a NOT IN</code> at. That is the whole point: this is not an exotic data condition, it is the ordinary one.</p> The query that returns nothing</a> </h2> Here is the request you have written a hundred times: give me the products whose category is not</em> archived.</p> SELECT</span> id, category_id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived);</span></span></code></pre> You expect products 1 and 4 (category 10, which is not in the archived set). What comes back is:</p> id \| category_id</span></span> ----+-------------</span></span> (0 rows)</span></span></code></pre> Every row gone. Not a subset, not an off-by-one: all of them. Drop the NULL</code> from archived</code> and the same query behaves:</p> SELECT</span> id, category_id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived</span></span> WHERE</span> category_id </span>IS NOT NULL</span>);</span></span></code></pre> id \| category_id</span></span> ----+-------------</span></span> 1 \| 10</span></span> 4 \| 10</span></span> (2 rows)</span></span></code></pre> To understand why a single NULL</code> empties the entire result, we have to stop thinking of NOT IN</code> as a single thing and watch the parser take it apart.</p> IN is an OR, NOT IN is an AND</a> </h2> IN</code> is not a primitive operator. It is shorthand that the parser rewrites into a chain of equality comparisons joined by OR</code>:</p> x </span>IN</span> (a, b, c)</span></span> -- becomes</span></span> x </span>=</span> a </span>OR</span> x </span>=</span> b </span>OR</span> x </span>=</span> c</span></span></code></pre> NOT IN</code> is the logical negation of that, and by De Morgan's law negating an OR</code> of equalities gives you an AND</code> of inequalities:</p> x </span>NOT IN</span> (a, b, c)</span></span> -- becomes</span></span> x </span><></span> a </span>AND</span> x </span><></span> b </span>AND</span> x </span><></span> c</span></span></code></pre> This is not an analogy. It is literally the expression Postgres builds, and you can read it straight off an EXPLAIN</code>. The literal-list forms collapse into array operators whose names give the whole game away:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> products </span>WHERE</span> category_id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span> -- Filter: (category_id = ANY ('{1,2,3}'::integer[]))</span></span> </span> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> products </span>WHERE</span> category_id </span>NOT IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span> -- Filter: (category_id <> ALL ('{1,2,3}'::integer[]))</span></span></code></pre> IN</code> is = ANY</code>: equal to any</em> element, an OR</code>. NOT IN</code> is <> ALL</code>: different from all</em> elements, an AND</code>.</p> The actual node types matter here, because they are what you end up staring at when you dump a parse tree or read a normalized pg_stat_statements</code></a> entry. A literal list compiles to a single ScalarArrayOpExpr</code>: the scalar on the left, the array on the right, and a useOr</code> flag that is the entire difference between = ANY</code> and <> ALL</code>. The subquery forms are a different node altogether, a SubLink</code>. Recognising those two names on sight tells you immediately which path the planner is on.</p> If "IN</code> and = ANY</code> are the same operator" is news: they compile to the same parse node and the same plan, with the spellings diverging only in plan-cache churn and selectivity estimates. The NOT IN</code> case in front of you here is the one corner where the choice is not cosmetic but a matter of correctness.</div> Three-valued logic does the rest</a> </h2> SQL does not have two truth values, it has three: true</strong>, false</strong>, and unknown</strong>. Any comparison against NULL</code> yields unknown</code>, because NULL</code> means "no value here" and you cannot ask whether an absent value is different from 20:</p> -- not false. unknown (displayed as a blank)</span></span> SELECT</span> 10</span> <> NULL</span>;</span></span></code></pre> Now walk the NOT IN</code> expansion for product 1 (category 10) against the archived set of 20 and NULL</code>:</p> 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> </div> true AND unknown</code> is unknown</code>, not true</code>. A WHERE</code> clause keeps a row only when its predicate evaluates to true</strong>. Both false</code> and unknown</code> cause the row to be discarded. So product 1 is dropped. Run the same arithmetic for product 4 and you land on unknown</code> again.</p> The mechanism in one sentence:</strong> the instant a single NULL</code> enters the right-hand side, the trailing AND unknown</code> term can never be true</code>, so the whole NOT IN</code> can never be true</code>, so every</strong> row is discarded, regardless of how many million rows you have or what they contain.</p> </div> NULLs on the left side too</a> </h2> Keeping NULL</code>s out of the subquery is not enough. The same unknown</code> arises from NULL</code>s on the left</em>: product 3 (whose category_id</code> is NULL</code>) evaluates to unknown AND unknown</code>, so it is dropped even against a spotless right-hand set. IN</code> and NOT IN</code> are not complements: a row can fail both tests simultaneously. There is a NULL</code>-shaped gap between them that belongs to neither.</p> The seam, in the source</a> </h2> All of this reduces to one branch in one function. Open src/backend/parser/parse_expr.c</code> and find transformAExprIn</code>, the routine that turns both IN</code> and NOT IN</code> list expressions into something the planner can chew on. The very first thing it decides is whether it is building an OR</code> or an AND</code>:</p> /</span></span> If the operator is <>, combine with AND not OR.</span></span> /</span></span> if</span> (</span>strcmp</span>(</span>strVal</span>(</span>linitial</span>(a</span>-></span>name</span>)),</span> "<>"</span>)</span> ==</span> 0</span>)</span></span> useOr </span>=</span> false</span>;</span></span> else</span></span> useOr </span>=</span> true</span>;</span></span></code></pre> That is the entire fork. IN</code> arrives carrying the operator =</code> and gets useOr = true</code>; NOT IN</code> arrives carrying <></code> and gets useOr = false</code>. The flag rides all the way down to where the boolean tree is finally assembled, several hundred lines later:</p> result </span>=</span> (Node </span></span>)</span> makeBoolExpr</span>(useOr </span>?</span> OR_EXPR </span>:</span> AND_EXPR,</span></span> list_make2</span>(result, cmp),</span></span> a</span>-></span>location</span>);</span></span></code></pre> OR_EXPR</code> for IN</code>, AND_EXPR</code> for NOT IN</code>. There is no special-casing of NULL</code> anywhere in this function, and there does not need to be: the three-valued behavior is an emergent property of having chosen AND</code>. The parser does the obvious, correct thing, and the NULL</code> semantics fall straight out of standard boolean logic. The "bug", if you insist on the word, belongs to the SQL standard, which Postgres implements faithfully.</p> A grammar asymmetry: list vs. subquery</a> </h3> A list and a subquery are built differently. The list form is the <> ALL</code> chain of inequalities you just saw. The subquery form is not a <> ALL</code> at all: it becomes NOT (foo = ANY (subquery))</code>. Different shapes, same truth table, a NULL</code> in the comparison makes the result unknown</code>, and unknown</code> loses. That NOT (... = ANY ...)</code> shape is the one the planner sees.</p> Why the planner won't save you</a> </h2> The correctness problem has a plan-shape twin. Watch what the planner does with three sibling queries on a larger schema of 200,000 orders</code> and 1,000 vip</code> rows.</p> First, the positive case, IN</code> against a subquery:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span> WHERE</span> customer_id </span>IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre> Hash Semi Join</span></span> Hash Cond: (orders.customer_id = vip.customer_id)</span></span> -> Seq Scan on orders</span></span> -> Hash</span></span> -> Seq Scan on vip</span></span></code></pre> A clean semi-join</strong>. The planner promotes the subquery to a first-class relation, picks a hash join, and is free to reorder it against the rest of the query. Now the natural mirror, orders whose customer is not</em> a VIP, written first with NOT EXISTS</code>:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders o</span></span> WHERE NOT EXISTS</span> (</span>SELECT</span> 1</span> FROM</span> vip v </span>WHERE</span> v</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span>);</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------</span></span> Hash Anti Join</span></span> Hash Cond: (o.customer_id = v.customer_id)</span></span> -> Seq Scan on orders o</span></span> -> Hash</span></span> -> Seq Scan on vip v</span></span></code></pre> A hash anti-join</strong>, the efficient and symmetric counterpart to the semi-join. Now the same intent expressed with NOT IN</code>:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span> WHERE</span> customer_id </span>NOT IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------</span></span> Seq Scan on orders</span></span> Filter: (NOT (ANY (customer_id = (hashed SubPlan 1).col1)))</span></span> SubPlan 1</span></span> -> Seq Scan on vip</span></span> (4 rows)</span></span></code></pre> No join at all. The subquery collapses into an opaque SubPlan</code> filter bolted onto a sequential scan, the NOT (... = ANY ...)</code> shape from the grammar is right there in the filter.</p> The wording of that filter line is version-specific. PostgreSQL 16 prints the terser Filter: (NOT (hashed SubPlan 1))</code>; a later version changed EXPLAIN</code> to expose the inner comparison, which is the NOT (ANY (... = (hashed SubPlan 1).col1))</code> form shown here on 18.4. The plan underneath is the same opaque subplan in every released version.</div> The difference between those last two plans is structural, not a tuning detail. A Hash Anti Join</code> builds one hash table from the inner relation and streams the outer relation through it exactly once, and when the inner side outgrows work_mem</code> it partitions into batches and spills to disk without changing the result. A SubPlan</code> gives the planner none of that. Even the hashed</code> variant you see here is evaluated in place as a filter on the scan: it cannot be reordered during the global join search, it cannot have the outer query's join clauses pushed into it to drive an index scan, and it has no multi-batch spill logic to fall back on if the hashed set turns out larger than the estimate. The subquery stops being a relation the optimizer can plan and becomes an opaque function the optimizer has to call, once per outer row. This is the same optimization-fence effect as an uninlined CTE</a>: the planner simply cannot see inside. On small inputs nobody notices; on large ones it is the difference between milliseconds and minutes.</p> The reason the planner is stuck with that shape is precisely the NULL</code> semantics from earlier. An anti-join keeps a row when it finds no match</em>. But NOT IN</code> must discard</em> a row when the comparison goes unknown</code>, and "unknown" is not the same as "no match". Those two behaviors diverge exactly when a NULL</code> is in play, so the planner historically could not prove the rewrite safe and never attempted it.</p> Declaring the columns NOT NULL</code> does not rescue you on released Postgres.</strong> I constrained both the outer and inner columns NOT NULL</code> on 18.4 and re-ran the query, and the plan was still the opaque SubPlan</code> filter. The released planner does not even look. So NOT IN (subquery)</code> has been a correctness trap for as long as Postgres has existed, and a planner pessimization ever since anti-joins arrived in 8.4 (2009) and every other negated subquery learned to use them — neither of which the released optimizer rescues you from.</p> </div> The fix is landing in PostgreSQL 19</a> </h2> That last paragraph is finally going out of date. This is a performance fix, not a semantic one: it rescues the case that was already correct</em> but badly planned</em> (NOT NULL</code> columns stuck behind the opaque SubPlan</code> from the previous section) and leaves nullable columns exactly where the SQL standard puts them. Tracing this through a checkout of the development branch, I found a function that exists in no released Postgres: sublink_testexpr_is_not_nullable</code>, in src/backend/optimizer/plan/subselect.c</code>. It guards a brand-new branch inside convert_ANY_sublink_to_join</code>:</p> /</span></span> Per SQL spec, NOT IN is not ordinarily equivalent to an anti-join, so</span></span> * that by default we have to fail when under_not. However, if we can</span></span> * prove that neither the outer query's expressions nor the sub-select's</span></span> * output columns can be NULL, and further that the operator itself cannot</span></span> * return NULL for non-null inputs, then the logic is identical and it's</span></span> * safe to convert NOT IN to an anti-join.</span></span> /</span></span> if</span> (under_not </span>&&</span></span> (</span>!</span>sublink_testexpr_is_not_nullable</span>(root, sublink)</span> \|\|</span></span> !</span>query_outputs_are_not_nullable</span>(subselect)))</span></span> return</span> NULL</span>;</span></span></code></pre> git blame</code> dates it to commit 383eb21ebff</code>, "Convert NOT IN sublinks to anti-joins when safe"</em>, merged in March 2026 by Richard Guo. It lives on master</code>, bound for PostgreSQL 19</strong>. It appears in no REL_18</code> tag, which is exactly why my 18.4 box still produced the SubPlan</code> even with NOT NULL</code> columns. The commit message states the bargain plainly:</p> if we can prove that neither side of the comparison can yield NULL values, and further that the operator itself cannot return NULL for non-null inputs, the behavior of NOT IN and anti-join becomes identical.</p> </blockquote> The proof has to establish three things:</p> Both operands provably non-NULL</code>.</strong> Established from schema NOT NULL</code> constraints (via a NOT-NULL-attnums hash table), from outer-join nullability tracking (so a Var</code> from the nullable side of an outer join does not qualify), and from qual clauses that force a Var</code> non-null.</li> The subquery's output columns provably non-NULL</code></strong>, the query_outputs_are_not_nullable</code> half of the guard.</li> A NULL</code>-safe operator.</strong> The operator must belong to a B-tree or Hash operator family. That is a proxy for "behaves like a normal boolean comparison and won't return NULL</code> on non-null inputs", because an operator that did</em> return NULL</code> there would break the very index it claims to support.</li> </ul> When all three hold, the NULL</code>-handling mismatch evaporates and NOT IN</code> is finally allowed to become a JOIN_ANTI</code>:</p> result</span>-></span>jointype </span>=</span> under_not </span>?</span> JOIN_ANTI </span>:</span> JOIN_SEMI;</span></span></code></pre> It lifts a planner limitation that has stood since anti-joins themselves arrived in 8.4, but only when the planner can prove</em> no NULL</code> can reach the comparison. If your column is nullable, you are exactly where you have always been, in PostgreSQL 19 as in 9.x. The semantics never changed; the optimizer merely learned to recognize the cases where NULL</code> is provably absent.</p> Decision matrix</a> </h2> You wrote</th> Internal shape</th> NULL on right →</th> NULL on left →</th> Planner (≤ PG 18)</th></tr></thead> IN (1,2,3)</code></td> = ANY</code>, an OR</code></td> absorbed, no harm</td> row dropped (no match)</td> ScalarArrayOpExpr</code></td></tr> NOT IN (1,2,3)</code></td> <> ALL</code>, an AND</code></td> all rows dropped</strong></td> row dropped</strong></td> ScalarArrayOpExpr</code></td></tr> IN (subquery)</code></td> = ANY</code> sublink</td> absorbed</td> row dropped</td> Semi Join</strong></td></tr> NOT IN (subquery)</code></td> NOT (= ANY)</code> sublink</td> all rows dropped</strong></td> row dropped</strong></td> opaque SubPlan</code> filter ¹</td></tr> NOT EXISTS (...)</code></td> anti-join sublink</td> row kept</td> row kept</td> Anti Join</strong></td></tr> </tbody></table> ¹ PostgreSQL 19 promotes this to an Anti Join when both the outer expression and the subquery output column are provably NOT NULL</code>.</p> What to do instead</a> </h2> Don't wait for PostgreSQL 19, and don't lean on it once it ships: it only fires on provably NOT NULL</code> columns. Every portable fix comes down to one rule: keep NULL</code> away from NOT IN</code>. In practice that means changing the query, and there are three moves, in order of preference.</p> Default to NOT EXISTS</code>.</strong> Make this the habit and you never hit the trap again. Same anti-join semantics and the same plan as the working NOT IN</code> would have wanted, on every version, nullable columns or not:</p> SELECT</span> id, category_id </span>FROM</span> products p</span></span> WHERE NOT EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> archived a </span>WHERE</span> a</span>.</span>category_id</span> =</span> p</span>.</span>category_id</span></span> );</span></span></code></pre> It keeps product 3, the NULL</code> category, because NOT EXISTS</code> asks whether a matching archived row exists, not whether category_id = NULL</code> is true. That is usually the answer you wanted. If you do want the NULL</code>-category rows gone, add AND p.category_id IS NOT NULL</code> and you have decided it on purpose.</p> Filter the NULL</code>s out of the subquery</strong> when you have to keep the NOT IN</code> (a generated query you can only edit inside the parentheses, say):</p> SELECT</span> id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span></span> SELECT</span> category_id </span>FROM</span> archived </span>WHERE</span> category_id </span>IS NOT NULL</span></span> );</span></span></code></pre> This covers the right-hand NULL</code> and nothing else: product 3 still disappears, because the left-hand NULL</code> is untouched. Use it only when you cannot reach for NOT EXISTS</code>.</p> Use EXCEPT</code> for whole-set comparisons.</strong> It matches rows with IS NOT DISTINCT FROM</code>, so two NULL</code>s count as equal and the three-valued trap never fires. But that same rule means a NULL</code> in archived</code> removes the NULL</code> rows from products</code>:</p> SELECT</span> category_id </span>FROM</span> products</span></span> EXCEPT</span></span> SELECT</span> category_id </span>FROM</span> archived;</span></span> -- returns: {10}</span></span> -- product 3 (NULL category) is dropped because archived also has a NULL</span></span></code></pre> So NOT EXISTS</code> keeps product 3 and EXCEPT</code> drops it. Pick the one whose answer matches what you're after.</p> EDIT (2026-06-15): I originally said this stood for about 25 years, which mixed up two things. The correctness trap is as old as NOT IN</code>; the planner pessimization only goes back to 8.4 (2009), when anti-joins arrived. Reworded to keep the two separate.</em></p> pg_stat_statements: everything it can't 2026-06-03T08:35:00+00:00 Part one</a> made the core case: pg_stat_statements</code> counts, it doesn't record. It walked through how the queryid jumble fragments one logical query into many rows, how the first-seen text freezes your per-request tags, and how the averages bury the p99 that actually pages you. All of that was about data the extension has</em> and distorts.</p> This part is about the rest: the entries it silently throws away, the query text that can vanish all at once, the plans and replicas it never records, and the knobs that bite. It ends where part one started, with the question the whole investigation was really about: is this the query store Postgres is missing, or just the floor you'd build one on?</p> The table fills up and evicts your tail</a> </h2> pg_stat_statements.max</code> defaults to 5000. It's a hard cap on entries, set when the server starts (changing it needs a restart, because the hash table is sized in shared memory up front). When the 5001st distinct shape arrives, Postgres doesn't grow the table. It evicts, throwing out the least-executed entries to make room:</p> If more distinct statements than that are observed, information about the least-executed statements is discarded.</p> </blockquote> On a healthy app with a few hundred steady shapes, 5000 is plenty and you never think about it. But remember the row explosion</a> from part one. An ORM that splinters one query into hundreds of shapes, or a pre-18 app building dynamic IN</code> lists, can chew through thousands of entries an hour. Once that starts, the view becomes a sliding window over recent noise. Your steady, important queries get evicted to make room for thousands of one-offs, then rebuilt with fresh counters and a fresh first-seen text when they run again. The stats you were trusting reset themselves, and the view never says a word about it.</p> One place does say something. The companion view pg_stat_statements_info</code> has exactly two columns, and both matter:</p> SELECT</span> dealloc, stats_reset </span>FROM</span> pg_stat_statements_info;</span></span></code></pre> dealloc \| stats_reset</span></span> ---------+-------------------------------</span></span> 18342 \| 2026-05-01 03:00:11.482+00</span></span></code></pre> dealloc</code>: Total number of times pg_stat_statements</code> entries about the least-executed statements were deallocated because more distinct statements than pg_stat_statements.max</code> were observed.</p> </blockquote> When dealloc</code> climbs steadily between resets, your table is too small for how many shapes your workload produces. You're thrashing, and your stats are lossy. The fix is either to raise pg_stat_statements.max</code> and pay the shared memory, or to cut the cardinality at the source by normalizing shapes, moving to PG 18, or switching to array binding. Few people watch dealloc</code>, so a thrashing table quietly drops part of the workload it's meant to report, and nobody notices until the numbers stop adding up.</p> The second column, stats_reset</code>, is the time of the last global reset of the whole view, and it isn't decoration. It's the validity stamp on every diff you compute. If you snapshot the view hourly to get usable averages, stats_reset</code> tells you whether a given window is even legitimate: if it moved forward between your two snapshots, a reset happened inside the window and every delta across it is meaningless.</p> Watch dealloc</code> and stats_reset</code>.</strong> A steadily climbing dealloc</code> means the table is too small and your tail is being evicted, so the stats are lossy. A stats_reset</code> that moved between two snapshots means a reset landed inside your window, so any diff across it is meaningless. Almost no monitoring setup checks either. </div> When the query text vanishes</a> </h3> There's a worse failure mode hiding in the external query-text file. Because representative texts pile up on disk and the file can grow large (long queries, a high max</code>, lots of churn), the extension has a last resort:</p> pg_stat_statements</code> may choose to discard the query texts, whereupon all existing entries in the pg_stat_statements</code> view will show null query</code> fields, though the statistics associated with each queryid</code> are preserved. If this happens, consider reducing pg_stat_statements.max</code>.</p> </blockquote> So under pressure it keeps the counters and throws away the text. Your whole view goes query = NULL</code> while the numbers keep marching on, and you're left with a list of anonymous queryids and no idea what they used to be. The docs leave this vague, but the source is specific enough to alert on. Two thresholds are at work.</p> The first is the garbage-collection trigger. As entries get evicted and rebuilt, their old texts leave dead space in the file. The extension rewrites the file to drop that dead space once it's more than about 50% bloat, specifically when the file on disk grows past mean_query_len × pg_stat_statements.max × 2</code>. With the default max</code> of 5000 and, say, a 1 KB average query, that's a rewrite around the 10 MB mark. That part is normal housekeeping. It's not the failure.</p> The failure is when that rewrite itself fails, from an I/O error or because it can't allocate room to build the replacement file. At that point the file's contents can't be trusted, so the extension marks every entry's text as invalid (internally, query_len = -1</code>) and the whole query</code> column reads NULL</code>. The counters survive. The texts are gone all at once.</p> On PostgreSQL 13 and earlier, stats_temp_directory</code> could relocate that file; the GUC was removed in 14.</div> The practical version: the file lives at $PGDATA/pg_stat_tmp/pgss_query_texts.stat</code>. Alert on its size against mean_query_len × max × 2</code>. If you're routinely near that line you're rewriting the file constantly, and you're one bad write()</code> away from a blank view. Because the trigger scales with your average query length, the cluster most at risk is the one with both lots of entries and very long query texts, which is the giant generated IN</code>-list app all over again. The fix the manual suggests, lowering pg_stat_statements.max</code>, sounds backwards but is right: a smaller table means a smaller file, less bloat to collect, and less to go wrong.</p> What it never records at all</a> </h2> Everything so far has been about data the extension records but degrades, by normalizing, freezing, averaging, or evicting it. This section is about absence: things pg_stat_statements</code> never captures at all, because they sit outside what it models.</p> Plans.</strong> This is the big one. The extension records that a query ran and how long it took. It doesn't record how</em> it ran. No plan tree, no per-node costs, no join order, no index choice. A query whose plan flipped from an Index Scan to a Seq Scan overnight, the single most common cause of a sudden slowdown, shows up only as a mean_exec_time</code> that went up. The reason is gone. For plans you need EXPLAIN</code>, auto_explain</code>, or a plan-capturing hook. pg_stat_statements</code> sits at the wrong layer to ever see one.</p> Actual versus estimated rows.</strong> With no plan there's no per-node actual_rows</code> to compare against the planner's estimate. The estimation error, which is the leading sign of stale statistics</a> and the thing that predicts a future plan flip, is invisible here. The rows</code> column is total rows returned across all calls, not an estimate measured against reality.</p> Parameter values.</strong> The constants were jumbled out. You can't recover which id</code> was slow, only that the shape WHERE id = $1</code> was slow on average.</p> Failed queries.</strong> This one surprises people. Stats are updated at the end of a successful phase, and only then. A query that errors out mid-execution isn't counted. The manual lays out the asymmetry exactly:</p> plans</code> and calls</code> aren't always expected to match because planning and execution statistics are updated at their respective end phase, and only for successful operations. For example, if a statement is successfully planned but fails during the execution phase, only its planning statistics will be updated.</p> </blockquote> So the query that runs 30 seconds and then dies on a statement timeout, exactly the one you most want to find, contributes nothing to calls</code> or total_exec_time</code>. It planned, so it might bump plans</code> if track_planning</code> is on, and that's the only trace it leaves. Some of your worst queries are invisible precisely because they're your worst queries.</p> There's no error counter either. A statement that fails leaves no row of its own, so you can't ask pg_stat_statements</code> how often a given shape errors, only how often it succeeded.</div> Spills, parallel workers, per-node buffers.</strong> Sort and hash spills to disk, workers_planned</code> versus workers_launched</code>, the shared/local/temp buffer split per plan node, all of that lives in EXPLAIN (ANALYZE, BUFFERS)</code></a> output, not in the aggregated shared_blks_</code> totals. The view gives you a sum of blocks touched. It can't tell you where in the plan they were touched, or whether a sort spilled.</p> It only sees one node</a> </h2> pg_stat_statements</code> is per-instance. Its hash table lives in this server's shared memory and counts the queries this server ran. It has no idea other nodes exist.</p> In a single-primary world that was fine. In a modern deployment (a primary with read replicas, an Aurora reader fleet, a Patroni cluster, Citus) it's a serious blind spot. Read traffic is the majority of the work for most apps, and it runs on the replicas. Each replica has its own independent pg_stat_statements</code>, with its own counters, its own eviction, its own first-seen texts. Snapshot only the primary, which is what most monitoring setups do, and you see the write path plus whatever reads happen to land on the primary, while the bulk of the read workload goes unmeasured.</p> Merging the nodes isn't clean either. You'd want to add up calls</code> for "the same query" across nodes, but the queryid is platform- and version-sensitive, so on a mixed fleet (mid-upgrade, different architectures) the same query can carry different ids on different nodes. Merging a cluster's workload under one canonical fingerprint, with per-node drill-down, is real work that lives above the extension. The extension only ever hands you one node's local tally.</p> The knobs, and what they cost</a> </h2> A quick tour of the settings, because the defaults hide a few traps.</p> GUC</th> Default</th> What to know</th></tr></thead> pg_stat_statements.max</code></td> 5000</code></td> Hard cap on entries; restart to change. Watch pg_stat_statements_info.dealloc</code>.</td></tr> pg_stat_statements.track</code></td> top</code></td> top</code> tracks client-issued statements only. Queries inside</em> PL/pgSQL functions stay invisible until you set all</code>.</td></tr> pg_stat_statements.track_utility</code></td> on</code></td> Counts non-DML (CREATE</code>, VACUUM</code>, and so on). Can flood the table with one-off DDL on schema-churning systems.</td></tr> pg_stat_statements.track_planning</code></td> off</code></td> Adds planning-time stats, but the manual warns of "a noticeable performance penalty" from contention when many connections run identical shapes and fight over one entry's spinlock. Off for a reason.</td></tr> pg_stat_statements.save</code></td> on</code></td> Persists stats across clean restarts. A crash</em> still loses everything since the last save.</td></tr> </tbody></table> Three that aren't obvious from the table:</p> track = 'top'</code> hides function internals.</strong> If your hot logic lives in PL/pgSQL or stored procedures, the default shows you the SELECT do_the_thing()</code> wrapper and nothing about the queries it runs inside. You have to opt into all</code> and pay for the extra entries.</p> track_utility</code> counts your transaction verbs.</strong> BEGIN</code>, COMMIT</code>, and ROLLBACK</code> are utility statements, so with the default track_utility = on</code> each one gets its own entry. They normalize to a single shape apiece, so it's a handful of rows, not a flood, but an ORM that opens and closes a transaction around every query turns COMMIT</code> into one of your highest-calls</code> rows: a content-free verb sitting at the top of any ranking by call count, ahead of the queries you actually want to see. If your top-by-calls view is mostly transaction control, that's the signal to turn track_utility</code> off.</p> I/O timings need a second switch.</strong> The shared_blk_read_time</code> and shared_blk_write_time</code> columns are only populated when the server-wide track_io_timing</code> is on, and it's off by default. Leave it off and those columns read zero, making every query look like it did no I/O, which is a quietly misleading kind of empty.</p> Why track_planning</code> in particular hurts</a> </h3> The warning on track_planning</code> is worth unpacking, because the obvious objection is the one the manual seems to skip. If a hot query gets executed</em> by thousands of backends billions of times and that's fine, why would planning</em> the same shape be the thing that tips over?</p> The answer is in how the counters get written. Every entry has its own spinlock (entry->mutex</code>), and updating the entry means grabbing that lock. The update happens once per phase. The extension writes the entry at the end of planning and again at the end of execution, in two separate critical sections. With track_planning</code> off, a query touches its entry's spinlock once per call, at exec-end. With it on, it touches the same lock twice, once at plan-end and once at exec-end. For a shape that lots of connections run at once, that single entry's lock is already the hottest in the system, and doubling the grabs roughly doubles the contention on it.</p> So it isn't that planning is contended and execution is free. Execution-side contention on that per-entry lock is itself a well-known bottleneck for ultra-hot identical queries. It's that track_planning</code> adds a second grab of the already-hot lock to every call, and piles that extra traffic onto exactly the entries that were most contended to begin with (one shape, run everywhere). That's why a setting that sounds harmless, "also record planning time," ships off by default, and why you should leave it off on any workload dominated by a handful of extremely hot shapes.</p> Mind the column renames (PG 17)</a> </h3> PostgreSQL 17 renamed the I/O timing columns.</strong> blk_read_time</code> became shared_blk_read_time</code> and blk_write_time</code> became shared_blk_write_time</code>, with new local_blk_</code> columns added alongside. A query selecting the old names errors after the upgrade; a SELECT </code> dashboard silently shifts. See the PG 17 release notes</a>. </div> If you have dashboards or scrapers older than PostgreSQL 17, they may have broken on the upgrade without saying so. The same release also added the stats_since</code> and minmax_stats_since</code> timestamps and a minmax_only</code> argument to pg_stat_statements_reset()</code>. The view's shape isn't a stable contract across majors, so pin your column lists and re-check them on every upgrade.</p> Who can see what</a> </h2> One security corner to know before you wire this into a shared dashboard. The statistics are visible to everyone, but the SQL text and queryid of other users'</em> queries are not:</p> For security reasons, only superusers and roles with privileges of the pg_read_all_stats</code> role are allowed to see the SQL text and queryid</code> of queries executed by other users. Other users can see the statistics, however, if the view has been installed in their database.</p> </blockquote> So a low-privilege monitoring role sees rows full of <insufficient privilege></code> where the query text should be. The fix is to grant pg_read_all_stats</code>, a built-in role, to the monitoring user. Not superuser. If your "the dashboard shows no query text" mystery has a boring cause, this is usually it.</p> So what is it good for, and where does it leave you?</a> </h2> None of this is an argument against pg_stat_statements</code>. It's still the first thing to turn on, and for its real job, ranking your query shapes by total time, calls, or I/O on one node since the last reset, nothing else is this cheap or this immediate. Reach for it first, every time. Just be clear about what you're holding:</p> It counts, it doesn't record. Cumulative totals, no time dimension. Snapshot and diff it yourself if you want history.</li> It gives you an average, not a spread. Great for "which shape dominates total time," useless for "what's my p99."</li> It's shape-fragmented. ORMs explode one query into many entries, and pulling them back together is a job for a normalizer on top.</li> It keeps first-seen text. Static comment tags survive, dynamic per-request tags don't, so it can never be an APM join target.</li> It's lossy under pressure. A full table evicts your tail. Watch dealloc</code>.</li> It's plan-blind. It tells you a query is slow, never why. No plan, no estimate error, no spills.</li> It's single-node. The replica fleet that serves most of your reads is invisible from the primary.</li> </ul> So pg_stat_statements</code> answers one question, what is slow?</em>, and stops. The next questions are outside its model: why</em> it got slow (a plan flip, stats drift), who</em> is calling it, where</em> across the cluster, and what would happen</em> if you changed something. That isn't a missing feature; it's what a pile of aggregated counters fundamentally can't do.</p> The per-execution detail isn't truly gone, it's in the logs. log_min_duration_statement = 0</code> or auto_explain</code> records every call with its parameters, tags, and timing, which is where the full distribution and the dynamic tags live. The catch is cost: you can't log every query on a busy system, which is the whole reason the cheap aggregate exists.</div> That answers the question we opened on. Postgres doesn't ship the query store you came looking for, and pg_stat_statements</code> isn't it. It's the floor you'd build one on, not the thing itself.</p> So stop reading pg_stat_statements</code> as a picture of your workload. Read it for what it is: a cheap, lossy box of counters that tells you where to point the real instruments next.</p> pg_stat_statements: everything it tells you 2026-06-02T22:00:00+00:00 If not first, pg_stat_statements</code> is one of the most used extensions in the PostgreSQL ecosystem. It ships in contrib</strong> and costs almost nothing to use. Most of us turn to it to answer the question: what is the database actually doing?</em> It's genuinely useful. You can use it to get a snapshot of what happened in a given timeframe, and make a faster decision about what to fix.</p> Coming from other database engines, you might reach for it expecting something a bit more, a query store. The built-in feature that keeps normalized queries and their plan history. Except pg_stat_statements</code> is not this. This article is going to deep dive into what the extension really provides.</p> Because once you lean on it, you might start noticing the rough edges. The gap comes from what "a query store" might come with and what it actually tells you.</p> The very same query might show up in many separate rows. Discrepancies between what your monitoring says and what mean_exec_time</code> shows. Missing queries. Numbers that changed overnight.</p> None of that is a bug. It all follows from what pg_stat_statements</code> really is: a fixed-size hash table of running counters, kept in shared memory, keyed by a hash of your parse tree. It counts; it does not record. Hold that one idea in your head and every surprising thing on the list above stops being surprising.</p> Everything below is reproducible. Paste this into a scratch database:</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> created_at </span>timestamptz NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i </span>FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, created_at)</span></span> SELECT</span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> '2024-01-01'</span>::</span>date +</span> (random()</span> </span> 700</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> ANALYZE;</span></span></code></pre> And turn the extension on:</p> -- in postgresql.conf config</span></span> -- shared_preload_libraries = 'pg_stat_statements' (requires restart)</span></span> CREATE</span> EXTENSION pg_stat_statements;</span></span></code></pre>What it actually stores</a> </h2> We've just turned it on, so start with what it puts in front of you.</p> SELECT * FROM</span> pg_stat_statements;</span></span></code></pre> You get one row per distinct query shape</em> and a set of counters behind it: calls</code>, total_exec_time</code>, rows</code>, shared_blks_hit</code>, shared_blks_read</code>, and a lot more.</p> While there is a query</code> column, you might think it keeps your SQL. But it does not. If you look carefully query text is normalized, with the literals stripped out (WHERE id = 42</code> is stored as WHERE id = $1</code>), and there is exactly one row per shape no matter how many times it ran. The counters only add up. A statement runs, the counters for its shape tick up, and that individual execution, its values, its timing, its plan, is gone. Each row is an entry</strong>: a running total per shape, not a record of executions. Most of the limitations in this article follow from that.</p> First, where the data lives. The counters sit in shared memory allocated at startup, and as we said, it's a fixed hash map. The number of tracked statements is capped (pg_stat_statements.max</code>, default 5000</strong>). The query text lives in a file on disk, $PGDATA/pg_stat_tmp/pgss_query_texts.stat</code>, which the view joins against the in-memory counters when you read it.</p> Each entry in that map is keyed by queryid</code>, and that's where the trouble starts.</p> The queryid</a> </h2> The jumble moved into core in PostgreSQL 14; before that it lived inside pg_stat_statements</code> itself. It's gated by the compute_query_id</code> setting, whose default auto</code> turns it on whenever a module that needs it, such as this one, is loaded.</div> The `queryid` is computed by Postgres core, not by the extension. When Postgres parses and analyzes a statement, it walks the resulting tree and hashes the parts that define the query's structure into single 64-bit number, leaving the constant values out. The Postgres source calls this step the query jumble; `pg_stat_statements` just reads the result and uses it as the key for each row. Leaving the constants out is the whole trick. WHERE id = 42</code> and WHERE id = 99</code> jumble to the same number, with the literal shown as a $n</code> placeholder, so one entry can stand in for millions of executions that differ only in their values.</p> SELECT * FROM</span> orders </span>WHERE</span> id </span>=</span> 42</span>;</span></span> SELECT * FROM</span> orders </span>WHERE</span> id </span>=</span> 99</span>;</span></span> SELECT * FROM</span> orders </span>WHERE</span> id </span>=</span> 12345</span>;</span></span> </span> SELECT</span> queryid, query, calls</span></span> FROM</span> pg_stat_statements</span></span> WHERE</span> query </span>LIKE</span> 'SELECT * FROM orders WHERE id%'</span>;</span></span></code></pre> queryid \| query \| calls</span></span> ----------------------+------------------------------------+-------</span></span> -8198406948996196422 \| SELECT * FROM orders WHERE id = $1 \| 3</span></span></code></pre> Three executions, one row. Now look at the queryid: it's negative. It's a signed 64-bit integer, which can easily break a dashboard that stored it in an unsigned column or printed it as a friendly identifier. It's a hash, not a sequence number. Treat it as an opaque token that might be negative and move on.</p> This is also where the surprises start, because the jumble is more sensitive than "it ignores constants" makes it sound.</p> The queryid isn't stable across versions</a> </h3> The manual is blunt about it:</p> it is not safe to assume that queryid</code> will be stable across major versions of PostgreSQL.</p> </blockquote> The hashing process is also sensitive to differences in machine architecture and other facets of the platform.</p> </blockquote> If you followed the previous section carefully, you'll have spotted that the hash comes from the shape of the parsed tree, not the query text. Upgrade from Postgres 16 to 17, or move to a different CPU architecture, and identical SQL can hash to a different queryid.</p> Anything that adds up query costs across a fleet by joining on queryid quietly falls apart the moment the fleet isn't uniform. The docs call out replicas specifically:</p> logical replication schemes do not promise to keep replicas identical in all relevant details, so queryid</code> will not be a useful identifier for accumulating costs across a set of logical replicas.</p> </blockquote> Don't treat queryid</code> as a durable key.</strong> Within one server it's stable across minor versions, and that is the only guarantee. Across a major upgrade, a different CPU architecture, or the replicas in a logical-replication set, the same SQL can hash to a different value. Any dashboard or rollup that joins on queryid</code> across machines is one upgrade away from quietly double-counting or dropping rows. </div> The queryid tracks the OID, not the name</a> </h3> The jumble uses the OIDs</strong> of objects a query touches, not their names. Two behaviors fall out of that, both straight from the manual, and both catch people off guard:</p> pg_stat_statements</code> will consider two apparently-identical queries to be distinct, if they reference for example a function that was dropped and recreated between the executions of the two queries.</p> </blockquote> Conversely, if a table is dropped and recreated between the executions of queries, two apparently-identical queries may be considered the same.</p> </blockquote> Drop and recreate a table, which happens all the time during migrations, dump-and-restore, or a blue-green cutover, and every query against it gets a fresh queryid, because the table now has a fresh OID. Your old stats for that table are stranded under the old id, looking like queries that simply stopped running. The dead query on your dashboard isn't dead; its OID moved.</p> search_path</code> hides the same trap. SELECT * FROM orders</code> resolves to tenant_a.orders</code> on one connection and tenant_b.orders</code> on another, so a schema-per-tenant app scatters one logical query across every tenant's OID. Correct behaviour, but rolling up "all the reads against orders</code>" means resolving OIDs yourself.</div> Aliases and structure leak in too</a> </h3> if the alias for a table is different for otherwise-similar queries, these queries will be considered distinct.</p> </blockquote> That last one, alias sensitivity, is what hurts ORM users most.</p> The ORM row explosion</a> </h2> The jumble normalizes constants. It does not normalize structure</em>. Two queries you'd read as "the same query" but that differ in the parsed tree, say a different column list, a table alias, an extra LIMIT</code>, or a reordered AND</code>, are different shapes with different queryids in different rows.</p> ORMs generate these variations constantly. Here are four queries an ActiveRecord, Hibernate, or Ecto app might emit for what is really one operation, "load a customer by id":</p> SELECT</span> id, </span>name FROM</span> customers </span>WHERE</span> id </span>=</span> $</span>1</span>; </span>-- queryid A</span></span> SELECT</span> id, </span>name</span>, created_at </span>FROM</span> customers </span>WHERE</span> id </span>=</span> $</span>1</span>; </span>-- queryid B (.select added a column)</span></span> SELECT</span> c</span>.</span>id</span>, </span>c</span>.</span>name</span> FROM</span> customers c </span>WHERE</span> c</span>.</span>id</span> =</span> $</span>1</span>; </span>-- queryid C (aliased)</span></span> SELECT</span> id, </span>name FROM</span> customers </span>WHERE</span> id </span>=</span> $</span>1</span> LIMIT</span> $</span>2</span>; </span>-- queryid D (.first)</span></span></code></pre> Four rows. The calls split four ways, and each one looks minor even when the real operation behind them might be your single hottest path. The mean_exec_time</code> is per-row, so a slowdown that hits all four equally gets diluted across all four.</p> ORMs don't stop at four. Every conditional .where</code>, every optional .select</code>, every eager-load is another shape; one logical query spreads across dozens or hundreds of entries. You can see real production snapshots where a single logical query lands under hundreds</strong> of distinct queryids, each looking minor on its own.</p> This is the job of a tool like qshape</a>, part of the boringSQL stack.</div> The extension can't fix this, and it isn't supposed to. The jumble runs before anything understands that "these read different columns from the same table for the same reason." Clustering the variants back together means re-parsing each entry's text, canonicalizing the tree (strip aliases, sort AND</code> predicates, spot column-set supersets), and re-fingerprinting. pg_stat_statements</code> hands you the raw, fragmented shapes; reassembling them into logical queries is a layer it doesn't have.</p> A downstream re-parse can work because normalization only looks like it threw the information away. The jumble strips one thing, the constant values (= 42</code> becomes = $1</code>), and leaves all the structure in the query</code> text: aliases, column list, the AND</code>/OR</code> tree, LIMIT</code>, join order. That structure is everything a canonicalizer needs to see that WHERE a = $1 AND b = $2</code> and WHERE b = $1 AND a = $2</code> are the same predicate, or that c.id</code> and id</code> are the same column. The extension over-normalizes constants and under-normalizes structure.</p> One thing genuinely is gone, though: anything that was evicted</strong>. A canonicalizer only sees the rows currently in the view, so if your ORM has flooded the hash table and dealloc</code> has dropped the tail, there's nothing left to fingerprint. Keep cardinality under control at the source first, by raising pg_stat_statements.max</code>, getting on PG 18, or switching to array binding (more on that next), or you'll just be fingerprinting whichever rows happened to survive, not your real workload.</p> The IN-list special case, and its PG 18 fix</a> </h3> There's one structural variation Postgres historically handled badly: lists of constants.</p> SELECT * FROM</span> orders </span>WHERE</span> id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span> SELECT * FROM</span> orders </span>WHERE</span> id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>, </span>4</span>);</span></span> SELECT * FROM</span> orders </span>WHERE</span> id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>, </span>4</span>, </span>5</span>);</span></span></code></pre> Before PostgreSQL 18, each of those was its own entry, because the length of the list changed the parsed tree. An app that builds IN</code> clauses on the fly, which is most apps, could burn through thousands of entries with a single query shape and evict everything else on the way (more on eviction below).</p> PostgreSQL 18 finally collapses them. The three statements above, differing only in list length, now land in a single entry:</p> query \| SELECT * FROM orders WHERE id IN ($1 /, ... /)</span></span> calls \| 3</span></span></code></pre> PostgreSQL 18: IN</code>-lists squashed.</strong> A constant list that differs only in length now collapses to one entry, shown as IN ($1 /, ... /)</code>. It's unconditional, with no GUC to turn on. The limit: it only fires for inline literals, so a driver that binds IN ($1, $2, $3)</code> as parameters still gets a fresh entry for every list length. </div> That's a real improvement, and a good reason to be on 18. Two things keep it from being the whole answer. Most production clusters are still on 17 or earlier, where the explosion is alive and well. And the parameter-binding drivers it skips, JDBC and friends, are often the worst offenders: they send IN ($1, $2, $3)</code> as bind parameters, so the values were never constants in the parsed tree and there is nothing to squash. IN ($1,$2,$3)</code> and IN ($1,$2,$3,$4)</code> stay separate entries, one new shape per list length.</p> The squashing briefly sat behind a GUC, query_id_squash_values</code>, during PG 18 development. It was removed before release</a>, so the behaviour is now always on.</div> If you're on one of those drivers, you're stuck regardless of your Postgres version, but there's a real fix, and it beats the PG 18 squashing because it works everywhere. Stop sending IN</code>-lists and send an array. Rewrite this:</p> SELECT * FROM</span> orders </span>WHERE</span> id </span>IN</span> ($</span>1</span>, $</span>2</span>, $</span>3</span>, ... $n); </span>-- n params, n distinct shapes</span></span></code></pre> as this:</p> SELECT * FROM</span> orders </span>WHERE</span> id </span>=</span> ANY($</span>1</span>); </span>-- one param, one shape, forever</span></span></code></pre> = ANY(array)</code> takes a single parameter, one array value, so the parsed tree is identical whether the array holds 3 elements or 3,000. That's one queryid, on every Postgres version and every driver, prepared or not, because the array length is never part of the structure the jumble sees. Most drivers and ORMs support array binding directly (setArray</code>, ARRAY[...]</code>, pgx's slice binding, ActiveRecord's where(id: array)</code>), and the planner uses it for index selection much like an IN</code>-list. For an ORM-heavy team it's the highest-leverage way to keep pg_stat_statements</code> readable.</p> The first-seen text is frozen</a> </h2> Every entry shows one query text, and it's the first one that created the entry. Not the most recent, not the most common, the first. After that the counters keep climbing on every matching execution, but the text on disk never changes, at least until the entry gets evicted and rebuilt.</p> Most of the time that's fine, because the constants are normalized to $1</code> anyway. It stops being fine the moment your queries carry comments, and modern apps carry a lot of them. The sqlcommenter</a> and marginalia conventions tack structured key-value context onto every query:</p> SELECT</span> id, </span>name FROM</span> customers </span>WHERE</span> id </span>=</span> $</span>1</span></span> /application='checkout',controller='orders',action='show'/</span></span></code></pre> Comments aren't part of the parsed tree, so they don't change the queryid. Every execution still maps to one entry. But the text on that entry is whatever showed up first. That splits comment tags into two kinds, and the split is the whole story:</p> Tag class</th> Examples</th> What pg_stat_statements</code> does with it</th></tr></thead> Static</strong>(same value every call from one place)</td> application</code>, controller</code>, action</code>, job</code>, framework</code></td> Survives.</strong> The first-seen value is representative, because every call from that spot carries the same value. You can scrape it from the query</code> text and trust it.</td></tr> Dynamic</strong>(a new value every request)</td> traceparent</code>, request_id</code>, trace_id</code>, tenant_id</code>, user_id</code></td> Lost.</strong> The first request's value is frozen into the text forever, and every later per-request value is silently dropped.</td></tr> </tbody></table> That second row is why pg_stat_statements</code> can never be a join target for an APM or tracing tool. You can't ask it "show me the queries from the trace that timed out at 14:32," because it kept exactly one traceparent</code>, the first it ever saw, and threw the rest away. A counter is a sum, and a sum has no room to remember which request contributed what. If you need dynamic tags, you need a per-execution capture path running alongside the extension (auto_explain, or a hook-based harvester), not the extension itself.</p> A traceparent</code> in the pg_stat_statements</code> text is a warning sign, not a feature: a per-request value reached the entry first and is now frozen there as if it were static context. If you scrape tags from these texts into a catalog, drop the hex-id-shaped ones.</p> The averages lie to you</a> </h2> This is the limitation that catches the most people, because the numbers look so trustworthy.</p> Every counter is cumulative since the last reset. There's no time dimension inside the extension at all. The view is one snapshot of running totals, and that has two consequences that feed each other.</p> There's no history built in.</strong> Say you want to know what got slower this week. The obvious move is to read mean_exec_time</code>, and that's exactly where it goes wrong. That number is the average over all</em> time since the last reset, so on a cluster that's been up for months it blends a cold-cache Monday morning, the nightly batch job, and a quiet Sunday into one figure that describes no real moment. To get a "this week" number you have to do the bookkeeping yourself: snapshot the view on a schedule, keep each snapshot in a table, and subtract two of them. The average over the last hour is just (total_exec_time_now - total_exec_time_then) / (calls_now - calls_then)</code>. The view hands you a running total; turning it into a rate is on you.</p> That subtraction is less safe than it looks. It quietly assumes the "now" counters carry on from the "then" counters, that they only ever climb. Usually true, not always. Eviction and resets, both covered in part two</a>, break that assumption, and a snapshot diff you'd actually trust has to handle three cases:</p> The entry was evicted and rebuilt.</strong> If a query drops out of the table and later runs again, it comes back with counters at zero. Now total_exec_time_now</code> is smaller than total_exec_time_then</code>, and the subtraction gives you a negative delta. Don't read that as "the query got faster." Read it as "this window is invalid for this queryid," because the entry's history was cut.</li> A global reset happened between snapshots.</strong> Someone ran pg_stat_statements_reset()</code>, or a deploy script did. Every row's "now" is smaller than its "then." This one is easy to catch: compare pg_stat_statements_info.stats_reset</code> against the time of your earlier snapshot, and if the reset is newer, the whole window crossed a reset and the entire diff is garbage.</li> The queryid is on only one side.</strong> A query in "then" but missing from "now" was evicted, or its table was dropped and recreated (new OID, new id). A query in "now" but not "then" is new or rebuilt. Join the two snapshots with a full outer join, and treat the rows that don't match on both sides as events, not as zeros to subtract from.</li> </ul> PostgreSQL 17 adds a cleaner per-entry guard: every row carries stats_since</code>, the moment the entry was created or recreated. If a row's stats_since</code> is newer than your earlier snapshot, its counters reset under you, so discard that window no matter what the subtraction says.</div> Before PG 17 the global stats_reset</code> is your only built-in signal, and per-entry eviction is invisible except through the counters-went-backwards trick. The bare formula is right; the monitoring layer around it is mostly this bookkeeping, and skipping it is how homegrown dashboards end up reporting impossible negative latencies.</p> The average hides the spread.</strong> Even inside one window, the extension only keeps four numbers per shape: min</code>, max</code>, mean</code>, and stddev</code> of exec time. It builds them on the fly, folding each execution into running aggregates and then dropping the timing, so there's no histogram, no percentiles, no per-call record to go back to. A query that runs in 1ms 99% of the time and 2 seconds the other 1% has a mean_exec_time</code> around 21ms, a number that matches none of its real executions and hides the p99 that's paging your on-call. Two-humped latency, a common shape for real performance problems, just disappears into an average and a standard deviation. The extension can tell you a query is slow on average, not that it's sometimes catastrophically slow, which is usually the more dangerous case.</p> The mean is just total_exec_time / calls</code>, but the extension maintains it (and the variance behind stddev</code>) with Welford's online algorithm: each execution folds into a running mean and sum-of-squares, then its timing is dropped. An average and a standard deviation are all that survive, never a percentile, because no samples are kept to rank.</div> If you want percentiles or a latency timeline, this is the wrong tool, and querying it harder won't help. It dropped the spread when it wrote the row.</p> That's the first half of the story: everything pg_stat_statements</code> tells you, and the ways it quietly bends what it reports. One logical query fragmented into hundreds of shapes, the text frozen at first sight, the whole distribution flattened into a single average.</p> The other half is what it never records at all: the entries it silently evicts, the query text that can vanish in one stroke, the plans it can't see, and the replicas it doesn't know exist. That's part two</a>, which also returns to the question we opened on: is this the query store Postgres is missing, or just the floor you'd build one on?</p> Welcome to ORDER BY jungle 2026-05-15T11:43:00+02:00 SQL is fun and not at all boring. The latest article by Markus Winand on Order by Has Come a Long Way</a> sent me on quite a journey.</p> First, set up a table called nums</code> with one integer column and four rows:</p> CREATE TABLE</span> nums</span> (a </span>int</span>);</span></span> INSERT INTO</span> nums </span>VALUES</span> (</span>0</span>), (</span>1</span>), (</span>2</span>), (</span>3</span>);</span></span></code></pre> Try to guess what these two queries return.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> Most of us would guess the same rows in a different order. The actual answer is that they produce exactly the same rows in exactly the same order</strong>. By the same logic you might expect</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>c;</span></span></code></pre> to do exactly the same. Except it does not. It errors with column "c" does not exist</code> despite the alias being right there in the statement. Welcome to ORDER BY jungle.</p> Names and expressions are not the same</a> </h2> If you ask most developers how ORDER BY</code> works, they will say "you put a column name there and it sorts the rows". In 99% of queries that is exactly what happens. People sort by created_at</code> or id</code> and move on.</p> Strictly speaking, three, if you count ORDER BY 1</code>. Positional references are their own can of worms and out of scope for this post.</div>But `ORDER BY` accepts two different kinds of things: SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY</span> created_at;</span></span> SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY date</span>(created_at);</span></span></code></pre> Both feel natural. And the thing nobody tells you is that they go down completely different code paths in the parser. Different scope rules, different lookups, different error messages. The first looks at your SELECT</code> list. The second looks at your FROM</code> clause. They never look at the same place.</p> Same answer, two different sorts</a> </h2> Look at the first query again.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span></code></pre> You wrote ORDER BY a</code>. A bare identifier, no decoration. Postgres goes down the name</em> path. It scans the SELECT</code> list for something called a</code>, finds the aliased column -a AS a</code>, and sorts by its output values. The negated values are -3, -2, -1, 0</code>, ascending is -3, -2, -1, 0</code>. That is what comes out.</p> Now the twin.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> You wrote ORDER BY -a</code>. This is no longer an identifier. It's an expression: unary minus around a column reference. The parser does not even try the same logic.</p> Instead it switches to the expression path</strong>, where the only a</code> it knows is the column in nums</code>, and sorts the input values negated. And by arithmetic luck, the two queries land on the same row order. Same output, completely different logic. If you don't believe it is just luck, drop the negation from the SELECT</code> list and keep it in ORDER BY</code>:</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> c</span></span> ---</span></span> 3</span></span> 2</span></span> 1</span></span> 0</span></span> (4 rows)</span></span></code></pre> ORDER BY -a</code> is an expression, so it sorts by -input_a</code> ascending, which is input_a</code> descending. The alias c</code> was never consulted. The result has nothing to do with whatever c</code> happens to be.</p> And ORDER BY -c</code> is now obvious. -c</code> is an expression, so the parser looks for column c</code> in FROM</code>, doesn't find it, and errors. The alias exists, but in a scope this code path cannot see.</p> Above the identifier, or around it</a> </h2> Once the rule is clear (bare identifier hits the SELECT</code> list, anything else hits the table) the rest of the surprises fall out.</p> SELECT</span> 'hello'</span> AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x::</span>text</span>;</span></span> -- ERROR: column "x" does not exist</span></span></code></pre> It is probably not surprising that casts</strong> count as expressions and push the lookup to the table.</p> The surprise might come with</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY</span> c </span>DESC NULLS FIRST</span>;</span></span></code></pre> Which will work as expected. Both DESC</code> and NULLS FIRST</code> are part of the sort clause itself, not of the sort expression. They sit above the identifier in the parse tree, so they never touch it. The parser still sees a bare c</code>, takes the fast path, finds the alias, sorts by it, and then applies "descending, nulls first" on top of the resolved key.</p> The same cannot be said about collation</strong>.</p> SELECT</span> 'A'</span>::</span>text AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x </span>COLLATE</span> "C"</span>;</span></span> -- ERROR: column "x" does not exist</span></span></code></pre> This is a really bad one. COLLATE</code> might look the same as a sort modifier, but it is not. It wraps the expression in the parse tree.</p> Parentheses are a special case.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> (a);</span></span> -- works, sorts by alias</span></span></code></pre> Postgres collapses redundant parens before the bare-identifier check, so (a)</code> is still bare a</code>. The seam is asymmetric in the way that maximises confusion: COLLATE</code> is "still a name to a human, an expression to the parser", and (a)</code> is "an expression to a human, still a name to the parser". You get both flavours of wrong intuition mixed here.</p> Unary plus.</strong> +a</code> and a</code> evaluate to the same value, but they do not parse to the same node.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY +</span>a;</span></span></code></pre> A plus sign you would not even think about changes which rows come out in which order.</p> The parser stores a column reference as a list of name parts: one part when it is unqualified, two or more once you add a table or schema. The fast path only fires on lists of length one.</div> Finally, schema- and table-qualified references</strong>. ORDER BY nums.a</code> looks like an identifier, but it is not.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> nums</span>.</span>a</span>; </span></span></code></pre> a</span></span> ----</span></span> -3</span></span> -2</span></span> -1</span></span> 0</span></span> (4 rows)</span></span> </span> a</span></span> ----</span></span> 0</span></span> -1</span></span> -2</span></span> -3</span></span> (4 rows)</span></span></code></pre>Aliases that aren't the names you think</a> </h2> Here is one that cost me an afternoon once. Easy to come across once an ORM or a generated view declared the alias for you. SQLAlchemy, Hibernate, jOOQ, and most code generators quote anything that isn't pure lowercase. Two queries, identical except that the alias is quoted in one. Two different result sets.</p> SELECT -</span>a </span>AS</span> A </span>FROM</span> nums </span>ORDER BY</span> a; </span>-- sorts by alias (-3,-2,-1,0)</span></span> SELECT -</span>a </span>AS</span> "A"</span> FROM</span> nums </span>ORDER BY</span> a; </span>-- sorts by input (0,-1,-2,-3)</span></span></code></pre> The bare-identifier check compares names with strcmp</code>. Unquoted A</code> folds to lowercase a</code> and matches. Quoted "A"</code> preserves case, stays A</code>, and does not match the lowercase a</code> in the ORDER BY</code>. The lookup fails, the parser falls through to the expression path, the expression path finds the column a</code> in nums</code>, and the query runs successfully while doing something different from what you meant.</p> GROUP BY checks the opposite scope first</a> </h2> Both GROUP BY</code> and ORDER BY</code> accept a bare identifier, and both can resolve it either way: to a table column or to a SELECT</code>-list alias. The difference is the order they check:</p> ORDER BY a</code> looks at the SELECT</code> list first, then the table.</li> GROUP BY a</code> looks at the table first, then the SELECT</code> list.</li> </ul> For most queries this never matters. The two clauses end up picking the same thing because nothing is shadowed. The surprise happens when an alias has the same name as a base column but a different value:</p> SELECT</span> a</span>/</span>2</span> AS</span> a, </span>count</span>(</span></span>)</span></span> FROM</span> nums</span></span> GROUP BY</span> a</span></span> ORDER BY</span> a;</span></span></code></pre> Now the two clauses disagree about what a</code> means. GROUP BY a</code> picks the input column (four distinct values, four groups, one row each). ORDER BY a</code> picks the alias, which is a/2</code>. The result has four rows because the grouping was on a finer-grained key than the projection:</p> a \| count</span></span> ---+-------</span></span> 0 \| 1</span></span> 0 \| 1</span></span> 1 \| 1</span></span> 1 \| 1</span></span></code></pre> Two rows where a/2 = 0</code> (from input 0</code> and 1</code>), two where a/2 = 1</code> (from input 2</code> and 3</code>). The duplicates are real. The same identifier means two different columns in two adjacent clauses of one query.</p> Window ORDER BY does not even pretend</a> </h2> This one trips people up because it does not look like a different clause.</p> SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY</span> neg) </span>FROM</span> nums;</span></span> -- ERROR: column "neg" does not exist</span></span></code></pre> OVER (ORDER BY ...)</code> is a different parse path entirely. It does not check the targetlist at all, only the FROM scope. The bare-name fast path simply does not exist here.</p> SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY -</span>a) </span>FROM</span> nums;</span></span> -- this works</span></span></code></pre> Two ORDER BY</code> clauses in the same query, two different scoping rules.</p> UNION ORDER BY is name-only</a> </h2> When ORDER BY</code> follows a UNION</code>, neither path is fully open.</p> -- ok</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a;</span></span> -- ERROR</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY -</span>a;</span></span> -- ERROR</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre> The error message is unusually helpful:</p> Only result column names can be used, not expressions or functions. HINT: Add the expression/function to every SELECT, or move the UNION into a FROM clause.</p> </blockquote> Set operations do not have a single FROM scope to fall back to, so the expression path is closed entirely. Bare names or nothing.</p> The seam, in the source</a> </h2> Full disclosure: I got this section wrong three times before Claude Code helped me trace the actual parse tree. Lack of sleep from a whole night of geeking out over ORDER BY</code> is the other plausible explanation.</div>Open `src/backend/parser/parse_clause.c` and find `findTargetlistEntrySQL92`. It is forty lines of comment, two `if` blocks, and a final `return`. SQL92's two resolution rules are tried first; SQL99 is the fallback. Block one: the bare-name path.</strong> The gate is a ColumnRef</code> node with exactly one name part, and that part must be a string identifier (not </code>, which is also a ColumnRef</code> but with an A_Star</code> field). If the node passes, the function walks the target list looking for a non-resjunk</code> entry whose resname</code> equals the identifier. The loop keeps going past the first match to detect ambiguity: identical expressions are fine (this is why SELECT a, a FROM nums ORDER BY a</code> works), different expressions error out. On a unique match, return.</p> If the loop finds nothing, the block does not</strong> return. Control falls through. This is the case behind the quoted-alias surprise earlier in the post: AS "A"</code> stores resname = "A"</code>, ORDER BY a</code> looks up resname = "a"</code>, the strcmp</code> fails, and the function moves on as if no SQL92 fast path applied.</p> GROUP BY</code> is the small exception inside this block. The name is first tested against the FROM</code> scope, and a hit there causes the targetlist loop to be skipped. That is how GROUP BY</code> ends up preferring the input column.</p> Block two: the positional path.</strong> The gate is IsA(node, A_Const)</code>. A non-integer constant errors immediately ("non-integer constant in ..."), which catches ORDER BY NULL</code>, ORDER BY 'a'</code>, ORDER BY TRUE</code>. An integer is used as a 1-based position into the non-resjunk</code> target list; anything outside the range errors as "position %d is not in select list". Block two never falls through.</p> Both 1</code> and -1</code> arrive here as integer A_Const</code>s. doNegate</code> in the grammar folds '-' Iconst</code> into a single integer constant before the function ever runs, so ORDER BY 1</code> and ORDER BY -1</code> go through the same code, with only the integer value (and the result of the position lookup) differing.</p> The fallthrough.</strong> Anything not caught above reaches the last line:</p> /</span></span> Otherwise, we have an expression, so process it per SQL99 rules.</span></span> /</span></span> return</span> findTargetlistEntrySQL99</span>(pstate, node, tlist, exprKind);</span></span></code></pre> That is the seam. SQL92 succeeds in two narrow shapes: a bare identifier with a matching alias, or an in-range positive integer. Everything else, including a bare identifier whose alias lookup found nothing, becomes a SQL99 expression resolved against FROM</code>.</p> A useful workaround</a> </h2> If you want the alias inside an expression in ORDER BY</code>, the portable trick is to wrap the query in a subselect:</p> SELECT </span></span> FROM</span> (</span>SELECT -</span>a </span>AS</span> x </span>FROM</span> nums) s</span></span> ORDER BY</span> x </span>+</span> 0</span>;</span></span></code></pre> Now x</code> is a real column in the FROM scope of the outer query. The expression path finds it. The seam has been moved out of the way.</p> This is, conceptually, what you would want the engine to do for you when you write ORDER BY x + 0</code> directly. The SQL-99 standard does not actually require that, though, and Postgres (along with SQL Server) documents explicitly that an alias inside an ORDER BY</code> expression is not supported. So you do it by hand.</p> The boring takeaway</a> </h2> Most of the time none of this matters. You sort by a column you just selected, the alias and the input column have the same name and the same value, and either parser path gives the same answer. The seam is invisible.</p> The minute the alias and the input column disagree in expression, value, case, or anything wrapped around the identifier, the parser picks one or the other silently, by a rule older than most working programmers.</p> There are two parsers. The bare-name path is SQL-92, the expression path is SQL-99, and they were stitched together in the late 1990s. They still disagree about which scope your identifiers live in, and knowing which one you triggered tells you which scope.</p> If after reading this post you still have to stop and think for a minute before predicting what</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre> does, that is the right reaction. It means you have the mental model.</p> The opening puzzle queries are from Jamie Brandon's comment on the Lobsters thread</a> discussing Markus Winand's history of ORDER BY</code></a> on modern-sql.com. Everything that follows here is the explanation that comment did not give. Both pieces are worth reading on their own.</em></p> Strong views on PostgreSQL VIEWs 2026-05-10T00:00:00+00:00 VIEWs should be the cleanest abstraction SQL, and therefore Postgres, has on offer. I love the concept. The promise of decoupling logical intent from physical storage is perfect on paper. In practice, few things in the database world trigger such a heated debate or carry as much historical baggage. VIEWs mix big promises with false hopes, and the promises rarely survive contact with production.</p> The appeal is straightforward. Abstract</strong> "active customer" once and reuse it everywhere. Every query, report and dashboard uses the same definition. The "active customer" then becomes the foundation of a "customer orders" view, which in turn powers an operational "customer summary" view.</p> -- layer 1: who counts as an active customer?</span></span> CREATE VIEW</span> active_customers</span> AS</span></span> SELECT</span> c.</span></span></span> FROM</span> customers c</span></span> WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span> AND</span> c</span>.</span>status</span> =</span> 'active'</span></span> AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span> </span> -- layer 2: active customers with their recent orders</span></span> CREATE VIEW</span> customer_orders</span> AS</span></span> SELECT</span></span> ac.</span></span>,</span></span> o</span>.</span>id</span> AS</span> order_id,</span></span> o</span>.</span>total_cents</span>,</span></span> o</span>.</span>created_at</span> AS</span> ordered_at,</span></span> o</span>.</span>status</span> AS</span> order_status</span></span> FROM</span> active_customers ac</span></span> LEFT JOIN</span> orders o </span>ON</span> o</span>.</span>customer_id</span> =</span> ac</span>.</span>id</span></span> WHERE</span> o</span>.</span>created_at</span> > now</span>()</span> -</span> interval </span>'12 months'</span></span> OR</span> o</span>.</span>created_at</span> IS NULL</span>;</span></span> </span> -- layer 3: one row per customer, ready for the dashboard</span></span> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> co</span>.</span>id</span>,</span></span> co</span>.</span>email</span>,</span></span> co</span>.</span>name</span>,</span></span> COUNT</span>(</span>co</span>.</span>order_id</span>) </span>AS</span> orders_12mo,</span></span> COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>) </span>AS</span> revenue_12mo_cents,</span></span> MAX</span>(</span>co</span>.</span>ordered_at</span>) </span>AS</span> last_order_at,</span></span> COUNT</span>(</span></span>) </span>FILTER</span> (</span>WHERE</span> co</span>.</span>order_status</span> =</span> 'refunded'</span>) </span>AS</span> refunds_12mo</span></span> FROM</span> customer_orders co</span></span> GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre> Each layer has one job. "Active customer" is defined exactly once - if marketing changes the ninety-day rule tomorrow, it is one line in one place, and the dashboard query collapses to SELECT FROM customer_summary WHERE id = $1</code>.</p> VIEWs also have the potential to be a real security boundary</strong>. Three hand-written queries means three places where a predicate can be forgotten, and three subtly different result sets a quarter later. With one definition, you have one result set.</p> For simple views, there is no performance penalty either. PostgreSQL inlines them directly into the calling query, so the planner sees through the view and plans as if you had written the underlying SQL by hand. That mechanism, and the cases where it stops working, has a deep-dive of its own</a>. This article assumes it as background and focuses on the other side: what views are made of, and what happens when the schema underneath them moves.</p> VIEWs are just like tables</a> </h2> The trouble starts with the syntax.</p> SELECT * FROM customer_summary</code> reads exactly like SELECT * FROM customers</code>. The catalog reinforces it: pg_class.relkind = 'v'</code> instead of 'r'</code>, but everything else (column lists, grants, comments, even \d</code> output) looks the same. So developers reach for them the same way: joining them, nesting them, wrapping them in other views without thinking about it.</p> A table is a heap with indexes. A view is a stored parse tree that gets expanded into your query at planning time. Most of the time the planner is clever enough that the difference does not matter: simple views are inlined, predicates are pushed down, the resulting plan is what you would have written by hand.</p> The nested view spiral</a> </h3> In practice, what you get is the dependency tree from the introduction (customer_summary</code> on top of customer_orders</code> on top of active_customers</code>) and a new engineer who has no way to know that SELECT * FROM customer_summary WHERE id = $1</code> expands into a three-layer rewrite, a LEFT JOIN</code> against twelve months of orders, and a GROUP BY</code> that the planner cannot push the id</code> predicate through.</p> Nothing in the query, the schema browser, or the ORM signals this. You only learn the shape of the tree by reading every definition top to bottom, which is exactly the work the abstraction was meant to save you.</p> You cannot reason about the performance of a query against a view without reading the view, the views it depends on, and the tables underneath.</p> Views are rewrite rules</a> </h3> A view is a macro. When you reference one, Postgres pastes its body into your query before the planner runs. There is no stored result, no view "object" the executor consults. The pg_class</code> row you see when you query the catalog is an empty shell: it holds the name, the column list, and the grants, but no definition.</p> The definition itself, the SELECT</code> you wrote, is stored separately in pg_rewrite</code>:</p> SELECT</span> ev_class::regclass, ev_type, is_instead</span></span> FROM</span> pg_rewrite</span></span> WHERE</span> ev_class </span>=</span> 'active_customers'</span>::regclass;</span></span></code></pre> ev_class \| ev_type \| is_instead</span></span> ------------------+---------+------------</span></span> active_customers \| 1 \| t</span></span></code></pre> That row is the view. ev_type = 1</code> is SELECT</code>, is_instead = t</code> means "replace, do not augment" (the pg_class</code> shell has no rows to read, so substitution is the only option that makes sense). Between the parser and the planner Postgres runs a stage called the rewriter</strong>, and the rewriter's job is to walk the parse tree, find references to relations that have rules, and substitute the rule body in place. This way a query like SELECT * FROM active_customers WHERE id = $1</code> arrives at the planner already rewritten:</p> -- what the planner actually sees</span></span> SELECT </span></span> FROM</span> (</span></span> SELECT</span> c</span>.</span>id</span>, </span>c</span>.</span>email</span>, </span>c</span>.</span>name</span>, </span>c</span>.</span>status</span>,</span></span> c</span>.</span>last_login_at</span>, </span>c</span>.</span>deleted_at</span></span> FROM</span> customers c</span></span> WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span> AND</span> c</span>.</span>status</span> =</span> 'active'</span></span> AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span></span> ) active_customers</span></span> WHERE</span> id </span>=</span> $</span>1</span>;</span></span></code></pre> Notice that c.</code> is already expanded into an explicit column list. That happened at CREATE VIEW</code> time and was frozen into the stored parse tree; the rewriter does not re-expand it. From here the planner can usually flatten the subquery into the outer query through a pass called subquery pull-up</strong>, leaving id = $1</code> as a predicate next to the others so an index on customers.id</code> is reachable.</p> That flattening is what makes simple views free. When it bails out (because the view body has LIMIT</code>, DISTINCT</code>, an aggregate, a set operation, or a few other shapes) the subquery stays put and outer predicates cannot move past it. The full list of planner barriers and how to spot them in EXPLAIN</code></a> lives in the inlining post.</p> Historically Postgres exposed this same machinery as user-facing CREATE RULE</code>, on the theory that arbitrary query rewriting was a general-purpose feature. It mostly was not. Rules are effectively deprecated outside of views; triggers do the same jobs without the surprises. The rule infrastructure survives because views need it.</span></p> Once you hold that model, every awkward thing about views falls out of it. Two in particular drive the rest of this article.</p> Columns are referenced by attribute number, not name.</strong> The stored parse tree does not remember that it reads customers.email</code>; it remembers that it reads attribute 2 of relation 16385. Rename the column and the view keeps working, because the OID and the position are unchanged. Drop a column in the middle and Postgres refuses upfront, because the stored attribute numbers would no longer line up. Every dependency error later in this article traces back to this.</p> The body is expanded once per reference.</strong> Every mention of the view in a query produces an independent copy of its parse tree in the rewritten plan. For a stable expression that is invisible. For a volatile one like random()</code> or clock_timestamp()</code>, each copy evaluates separately:</p> CREATE VIEW</span> v_rand</span> AS</span></span> SELECT</span> id, random()</span> AS</span> r </span>FROM</span> customers;</span></span> </span> SELECT</span> a</span>.</span>id</span>, </span>a</span>.</span>r</span> AS</span> r_a, </span>b</span>.</span>r</span> AS</span> r_b</span></span> FROM</span> v_rand a</span></span> JOIN</span> v_rand b </span>ON</span> a</span>.</span>id</span> =</span> b</span>.</span>id</span></span> LIMIT</span> 3</span>;</span></span></code></pre> id \| r_a \| r_b</span></span> ----+----------------------+---------------------</span></span> 1 \| 0.84043639656488 \| 0.0833458769902089</span></span> 2 \| 0.009846241116064247 \| 0.6574000469586228</span></span> 3 \| 0.14667469313524628 \| 0.21862693208148087</span></span></code></pre> Same view, same row, two different r</code> values. The rewriter expanded v_rand</code> twice, so the rewritten query contains two independent random()</code> calls, and the planner has no reason to share them. You might hit this expecting a.r = b.r</code> and end up rewriting the view to push the volatile call outside.</p> This is also why CREATE OR REPLACE VIEW</code> can only append columns to the end of the column list. Existing attribute numbers must stay stable for every dependent, so anything that would shift them is forbidden.</span></p> A useful side effect of the same mechanism: views are immune to the search-path attacks that bite SECURITY DEFINER</code> functions. Identifiers are resolved to OIDs at CREATE VIEW</code> time, so a later SET search_path</code> cannot redirect a view to a different table. Functions resolve identifiers per call against the caller's search_path</code> unless you pin it; views never have that exposure.</p> Writable views, the half-kept promise</a> </h3> Funnily enough, I was not aware writable views existed until earlier this year. It came up during a hallway conversation with Ervin Weber</a> and Jesper St John</a> at PgDay Nordic 2026 in Helsinki. Twelve years writing PostgreSQL and the auto-updatable rules had simply never crossed my desk.</span></p> The original promise was symmetry: if a view looks like a table for reads, it should look like one for writes. Postgres delivers this for the easy case (a view over a single base table, no joins, no aggregates, no DISTINCT</code>) and calls them auto-updatable</strong>. You can INSERT</code>, UPDATE</code>, and DELETE</code> through them, and the rewrite rule translates the operation against the underlying table.</p> INSERT INTO</span> active_customers (email, </span>name</span>)</span></span> VALUES</span> (</span>'alice@example.com'</span>, </span>'Alice'</span>);</span></span> -- becomes: INSERT INTO customers (email, name, status, last_login_at, deleted_at)</span></span> -- VALUES ('alice@example.com', 'Alice', DEFAULT, DEFAULT, DEFAULT);</span></span></code></pre> Add a join, an aggregate, a GROUP BY</code>, or anything else outside the auto-updatable rules and the write path goes silent. SELECT</code> still works, but INSERT</code> errors and you have to wire up INSTEAD OF</code> triggers by hand. So customer_orders</code> and customer_summary</code> are read-only by accident, not by design, and the rules for what counts are not visible in the view definition itself. You find out at write time, in production.</p> WITH CHECK OPTION</code> rejects any write that would produce a row the view cannot see, with LOCAL</code> checking only this view's predicate and CASCADED</code> (the default) checking every underlying view up the chain. It is how you hand an application a writable, scoped slice of a table without trusting it to enforce the scope itself.</p> Views and row-level security</a> </h3> RLS policies live on tables, not views. For a regular view that matters less than it sounds: the policies still fire when the view's query hits the base table. What determines whether they fire correctly is whose context the view runs under.</p> By default, a view runs as its owner. Superusers bypass RLS unconditionally; table owners bypass it unless FORCE ROW LEVEL SECURITY</code> is set. A view owned by a privileged role is therefore a hole in any RLS scheme, silently, because nothing in the view definition or the catalog signals it.</p> Since PostgreSQL 15 the fix is a single option:</p> CREATE VIEW</span> my_view</span> WITH</span> (security_invoker </span>=</span> true) </span>AS</span></span> SELECT</span> ... </span>FROM</span> base_table;</span></span></code></pre> With security_invoker</code> set, the view runs as the calling user and RLS evaluates against the actual caller. Before 15, the workaround was a dedicated low-privilege view owner plus FORCE ROW LEVEL SECURITY</code> on the base table.</p> Materialized views are a different problem. REFRESH MATERIALIZED VIEW</code> runs as the owner, so the snapshot contains whatever the owner can see. Every user then reads that same materialized data. There is no per-row filtering because the rows are physically stored; security_invoker</code> has nothing to attach to.</p> The architect's stigma</a> </h3> When I first got into databases, views were the thing senior people warned you off. "Don't use views in production", "views are slow", "views hide the real query". You learned to write the SQL inline, even when the same predicate appeared in twenty places, because the alternative was disapproval at the next code review.</p> In Oracle shops the warning had specific roots. The cost-based optimiser of that era struggled to merge predicates through views with non-trivial projections, joins, or set operations, so a clean three-layer view could expand into a plan that scanned everything and filtered late. The shop-wide answer was almost always materialised views or PL/SQL packages that encapsulated every access path; a simple view sat awkwardly between the two, not cached like a matview, not encapsulated like a package, and with optimiser surprises on top.</span></p> Postgres carried the same baggage by association. Nested-view spirals were real, rule-system surprises were real, and the optimiser of fifteen years ago was less capable than today's. The rule of thumb hardened into doctrine and outlived its conditions.</p> The improvements are real; the reputation has not caught up. Reach for a view in a code review today and someone will still object on principle. Worth knowing before your PR hits review.</p> What happens when you change a table</a> </h2> Everything covered so far stays invisible while the schema underneath holds still. View bodies live in pg_rewrite</code>, columns are pinned to attribute numbers, types are frozen at CREATE VIEW</code> time, and the planner just expands and runs.</p> The moment someone tries to change a column the view tree references, every one of those mechanics surfaces at once. Suppose someone needs to drop the name</code> column from customers</code>:</p> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span></code></pre>ERROR: cannot drop column name of table customers because other objects depend on it</span></span> DETAIL: view active_customers depends on column name of table customers</span></span> HINT: Use DROP ... CASCADE to drop the dependent objects too.</span></span></code></pre> active_customers</code> references name</code> (the c.</code> was expanded at view-creation time), customer_orders</code> inherits the dependency through ac.</code>, and customer_summary</code> is another layer down. The whole tree is frozen on one column.</p> It is not just dropping. Try widening a type:</p> ALTER TABLE</span> customers </span>ALTER</span> COLUMN email </span>TYPE TEXT</span>;</span></span></code></pre>ERROR: cannot alter type of a column used by a view or rule</span></span> DETAIL: rule _RETURN on view active_customers depends on column "email"</span></span></code></pre> The same wall blocks any structural change to a column a view references: drop, rename, type change. Even widening VARCHAR(255)</code> to TEXT</code>, a change that loses no data and adds no constraint, is forbidden because the view's stored definition references the old type OID.</p> The hint suggests CASCADE</code>:</p> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span> CASCADE;</span></span></code></pre>NOTICE: drop cascades to 3 other objects</span></span> DETAIL: drop cascades to view active_customers</span></span> drop cascades to view customer_orders</span></span> drop cascades to view customer_summary</span></span></code></pre> All three views are dropped, not modified, along with their grants and any other dependent objects. On a schema with dozens of interconnected views, CASCADE</code> is nightmare.</p> Never use CASCADE on production views without a full recreation script ready.</strong> CASCADE doesn't modify views, it drops them. All GRANTs, row-level security policies, and downstream dependencies disappear with them. There is no undo. </div> So the manual path is: save every view definition, drop them in reverse dependency order, alter the table, recreate them in forward dependency order, reapply all grants. For three views this is tedious. For thirty views across multiple schemas, it is a full migration project.</p> -- 1. save definitions </span></span> SELECT</span> pg_get_viewdef(</span>'customer_summary'</span>, true);</span></span> SELECT</span> pg_get_viewdef(</span>'customer_orders'</span>, true);</span></span> SELECT</span> pg_get_viewdef(</span>'active_customers'</span>, true);</span></span> </span> -- 2. drop in leaf-first order</span></span> DROP VIEW</span> customer_summary;</span></span> DROP VIEW</span> customer_orders;</span></span> DROP VIEW</span> active_customers;</span></span> </span> -- 3. now you can alter the table</span></span> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span> </span> -- 4. recreate in correct order</span></span> CREATE VIEW</span> active_customers</span> AS</span></span> SELECT</span> id, email, </span>status</span>, last_login_at, deleted_at</span></span> FROM</span> customers</span></span> WHERE</span> deleted_at </span>IS NULL</span></span> AND status =</span> 'active'</span></span> AND</span> last_login_at </span>> now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span> </span> CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span> </span> -- 5. reapply grants and so on</span></span> -- ...</span></span></code></pre>The SELECT * trap</a> </h2> You might think SELECT </code> saves you from column-level dependencies. It does not. It makes things worse.</p> CREATE VIEW</span> all_customers</span> AS</span></span> SELECT FROM</span> customers;</span></span></code></pre> This looks flexible. PostgreSQL expands SELECT </code> at view creation time</strong> and freezes the result. Check what the database actually stored:</p> SELECT</span> pg_get_viewdef(</span>'all_customers'</span>::regclass, true);</span></span></code></pre> SELECT id,</span></span> email,</span></span> name,</span></span> status,</span></span> last_login_at,</span></span> deleted_at</span></span> FROM customers;</span></span></code></pre> The </code> was expanded to the columns that existed when the view was created. Now add a column to the base table:</p> ALTER TABLE</span> customers </span>ADD</span> COLUMN phone </span>TEXT</span>;</span></span> SELECT * FROM</span> all_customers;</span></span></code></pre> The phone</code> column is not there</strong>. The view still returns the original columns. To pick up new columns, you have to CREATE OR REPLACE VIEW all_customers AS SELECT * FROM customers</code>, which re-expands the </code> against the current table definition.</p> Drop a column that was in the original expansion and the migration is blocked the same way it would be with an explicit column list, except now the dependency is hidden in the catalog instead of visible in the view body. SELECT </code> gives the illusion of flexibility while creating the same rigid coupling.</p> This is actually documented behaviour and follows the SQL standard. But it surprises nearly everyone the first time they encounter it, and it is the source of many "my view is missing columns" bug reports.</span></p> Always use explicit column lists in views.</strong> At least then the dependency is visible and the breakage is predictable.</p> The dependency wall, the type-OID coupling, the frozen SELECT </code> expansion: all of it is the same trade-off, applied consistently. Rigidity is the feature, not a bug.</p> Where this isn't a PostgreSQL bug</a> </h2> The rigidity is a design choice, not a defect. Other databases made different choices, and none of them are free.</p> Oracle</strong> marks dependent views as INVALID</code> rather than blocking the DDL. The views are recompiled automatically on next access. If they still work, great; if not, you get an error at query time. This sounds better until you realize what it means in practice: you can deploy a migration, get a clean exit code, and not discover that a critical reporting view is broken until a user hits it on Monday morning. The breakage moves from migration time, where you are paying attention, to runtime, where you may not be. For teams that value deployment confidence, this is arguably worse.</p> SQL Server</strong> has sp_refreshview</code>, which recompiles a view's metadata against the current table definitions. You can alter a table and then refresh the dependent views to pick up the changes. But sp_refreshview</code> works on one view at a time. There is no built-in way to refresh an entire dependency chain in the right order. And if a view references a dropped column, sp_refreshview</code> fails; it does not remove the reference for you. You still have to manually edit and recreate the view. It is a convenience, not a solution.</p> PostgreSQL chose compile-time safety: no surprise INVALID</code> views in production, no silently wrong results from a lazy recompilation papering over a structural change. The cost is manual dependency management on every schema change, and it is high enough to push experienced teams away from views entirely.</p> The saving grace: transactional DDL</a> </h2> Before the workarounds, there is one PostgreSQL feature that materially changes the risk calculus: DDL is transactional</strong>. The entire drop-alter-recreate-regrant sequence can live inside a single BEGIN/COMMIT</code>:</p> BEGIN</span>;</span></span> DROP VIEW</span> customer_summary;</span></span> DROP VIEW</span> customer_orders;</span></span> DROP VIEW</span> active_customers;</span></span> </span> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span> </span> CREATE VIEW</span> active_customers</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span> </span> GRANT SELECT ON</span> customer_summary </span>TO</span> reporting;</span></span> COMMIT</span>;</span></span></code></pre> If any statement raises (a typo in a recreated view, a missing grant, an unexpected dependency) the whole migration rolls back and the database is exactly as it was. Oracle and SQL Server cannot do this for most DDL; their CREATE VIEW</code> auto-commits, so a partial failure leaves you with a half-migrated schema and no ROLLBACK</code> to lean on. PostgreSQL's "you have to drop and recreate everything" pain is real, but the recovery story is much better than Oracle's or SQL Server's.</p> This does not solve the locking problem. DROP VIEW</code> and CREATE VIEW</code> take AccessExclusiveLock</code> on the view; ALTER TABLE</code> takes one on the table. For the duration of the transaction, anything that touches those objects waits behind it. A long view-rebuild transaction on a busy system is a stop-the-world window. Keep the transaction tight: prepare the new SQL outside the transaction, hold no other locks, and do not run it during peak traffic. Transactional DDL gives you safety, not concurrency.</p> The workarounds</a> </h2> None of these solve the fundamental problem; they make it manageable.</p> Avoid views entirely.</strong> Replace views with application-level query builders or ORM scopes. You lose the shared abstraction: when three services need the same definition of "active customer," each one implements it independently. Definitions drift. For teams with full control over a small codebase, this is a reasonable trade. For shared data platforms, it is not.</p> Script the recreation.</strong> Use pg_get_viewdef()</code> to extract definitions, then write migration scripts that drop and recreate views around schema changes. The problems: no impact analysis (you have to figure out the dependency order yourself), no grant preservation (view definitions do not include permissions), and it does not scale past a handful of views. Your migration framework will not help here either. They run your SQL files in order, but figuring out what</em> SQL to write is entirely on you.</p> Version your views.</strong> Instead of modifying views in place, create active_customers_v2</code> alongside active_customers_v1</code>. Migrate consumers one at a time. Drop the old version when nothing uses it. This works well when views are consumed by multiple independent teams, since you cannot force everyone to migrate in the same deployment window. The trade-off is naming discipline and version proliferation: if customer_orders_v1</code> depends from active_customers_v1</code>, you now need customer_orders_v2</code> too.</p> Schema-based versioning (CREATE SCHEMA api_v2; CREATE VIEW api_v2.active_customers AS ...</code>) is cleaner than suffixes for public-facing APIs. It gives you a natural namespace and lets you SET search_path</code> to switch versions.</span></p> Query pg_depend yourself.</strong> PostgreSQL tracks all object dependencies in the pg_depend</code> system catalog. The catch: views do not depend on tables directly. The rewrite rule that implements the view does, so every traversal goes pg_depend -> pg_rewrite -> pg_class</code>. Direct view dependents of customers</code>:</p> SELECT</span></span> depns</span>.</span>nspname</span> AS</span> dependent_schema,</span></span> depc</span>.</span>relname</span> AS</span> dependent_view,</span></span> COALESCE</span>(</span></span> (</span>SELECT</span> a</span>.</span>attname</span> FROM</span> pg_attribute a</span></span> WHERE</span> a</span>.</span>attrelid</span> =</span> d</span>.</span>refobjid</span> AND</span> a</span>.</span>attnum</span> =</span> d</span>.</span>refobjsubid</span>),</span></span> ''</span></span> ) </span>AS</span> source_column</span></span> FROM</span> pg_depend d</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>refobjid</span> =</span> 'customers'</span>::regclass</span></span> AND</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> d</span>.</span>refclassid</span> =</span> 'pg_class'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> AND</span> depc</span>.</span>oid</span> <></span> 'customers'</span>::regclass</span></span> ORDER BY</span> depns</span>.</span>nspname</span>, </span>depc</span>.</span>relname</span>;</span></span></code></pre> Useful, but only one hop. customer_summary</code> depends on customers</code> transitively through two intermediate views, which this query misses. A recursive CTE walks the chain:</p> WITH RECURSIVE</span> view_deps </span>AS</span> (</span></span> -- direct dependents of the target table</span></span> SELECT DISTINCT</span></span> depc</span>.</span>oid</span> AS</span> view_oid,</span></span> depns</span>.</span>nspname</span> \|\|</span> '.'</span> \|\|</span> depc</span>.</span>relname</span> AS</span> view_name,</span></span> 1</span> AS</span> depth</span></span> FROM</span> pg_depend d</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>refobjid</span> =</span> 'customers'</span>::regclass</span></span> AND</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> </span> UNION</span></span> </span> -- views depending on views we already found</span></span> SELECT</span> depc</span>.</span>oid</span>,</span></span> depns</span>.</span>nspname</span> \|\|</span> '.'</span> \|\|</span> depc</span>.</span>relname</span>,</span></span> vd</span>.</span>depth</span> +</span> 1</span></span> FROM</span> view_deps vd</span></span> JOIN</span> pg_depend d </span>ON</span> d</span>.</span>refobjid</span> =</span> vd</span>.</span>view_oid</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> AND</span> depc</span>.</span>oid</span> <></span> vd</span>.</span>view_oid</span> -- skip the view's own _RETURN rule</span></span> )</span></span> SELECT</span> view_name, </span>MIN</span>(depth) </span>AS</span> depth</span></span> FROM</span> view_deps</span></span> GROUP BY</span> view_name</span></span> ORDER BY</span> depth, view_name;</span></span></code></pre> The depc.oid <> vd.view_oid</code> filter matters: each view's _RETURN</code> rule has a self-dependency entry that without the filter sends the walk back to the view it just visited.</p> This works, but at this point you are writing a dependency-analysis tool. Add topological sort, grant capture, matview handling, RLS-policy preservation, and you have small product.</p> Materialized views have it worse.</strong> Everything above applies, plus you lose the cached result set on DROP</code>. After recreating, REFRESH</code> rebuilds from scratch: minutes to hours on large datasets, with no CONCURRENTLY</code> option until a unique index exists. For matviews that back dashboards, this means downtime. RLS is also off the table: REFRESH</code> runs as the matview owner, so the snapshot reflects the owner's access, not the querying user's. </div> Prior art hiding in pg_dump</a> </h2> The mechanism the manual workaround keeps reinventing, rewriting a view's body while preserving its identity, grants, and dependents, already exists inside PostgreSQL. It happens to live in pg_dump</code>, used only when needed.</p> Most of the time pg_dump</code> does the obvious thing: builds a dependency graph, topologically sorts it, and emits each view in base-first order with a normal CREATE VIEW</code>. Dump our four-view chain and that is exactly what you get.</p> The interesting case is when the sort fails. Cycles are uncommon but real: a view body that calls a function whose own body references the view back, a trigger on a base table that reads through a view reading the same table, cross-referencing matviews with RLS policies pointing back at views. When pg_dump</code> hits one, it falls back to the placeholder view trick</strong>: emit one of the views early as a stub with the right column list and types but a dummy body, then come back with CREATE OR REPLACE VIEW</code> to install the real definition once the rest of the cycle exists.</p> -- emitted early, as a placeholder</span></span> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> NULL</span>::</span>integer AS</span> id,</span></span> NULL</span>::</span>text AS</span> email,</span></span> NULL</span>::</span>text AS name</span>,</span></span> NULL</span>::</span>bigint AS</span> orders_12mo,</span></span> NULL</span>::</span>bigint AS</span> revenue_12mo_cents;</span></span> </span> -- emitted later, once dependencies exist</span></span> CREATE OR REPLACE VIEW</span> customer_summary</span> AS</span></span> SELECT</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>,</span></span> COUNT</span>(</span>co</span>.</span>order_id</span>) </span>AS</span> orders_12mo,</span></span> COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>) </span>AS</span> revenue_12mo_cents</span></span> FROM</span> customer_orders co</span></span> GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre> Grants, comments, and policies attach to the stub and survive the rewrite because the OID never changes. CREATE OR REPLACE VIEW</code> mutates the same pg_class</code> row in place. PostgreSQL has been doing this on every pg_dump --schema-only</code> for years. The mechanism is there; it is simply not exposed as user-facing DDL.</p> Before you reach for a VIEW</a> </h2> Views are worth using. The abstraction is real: shared definitions, clean layering, column-level security, and no runtime cost when the planner can inline them. The trouble is that none of the rigidity is visible from the outside. A view looks like a table in the catalog, in \d</code>, in the ORM, and in every query that touches it. The teardown cost only appears the first time someone tries to drop a column, widen a type, or CASCADE</code> their way out of a migration on a Friday afternoon.</p> So before you reach for one, hold the trade-off in mind:</p> SELECT </code> in a view body is a trap.</strong> It freezes the column list at creation time, hides the dependency in the catalog, and still blocks the same DDL an explicit list would. Always write the columns out.</li> Every layer multiplies the teardown cost.</strong> A three-deep view chain means three drops, three recreates, three sets of grants and policies to reapply, in the right order, for any structural change to a column at the bottom. Keep dependency trees shallow, and be honest about whether the abstraction is paying for itself.</li> CASCADE</code> is not a fix.</strong> It drops dependent views along with their grants, RLS policies, and downstream dependencies, with no undo. Never run it in production without a recreation script ready.</li> security_invoker</code> is not the default.</strong> A view runs as its owner, which silently bypasses RLS on base tables when the owner is privileged. On PostgreSQL 15+, WITH (security_invoker = true)</code> is what makes the security boundary real. For materialized views there is no equivalent. Enforce access at the layer that reads the matview, never below it.</li> </ul> When the migration finally has to happen, two things make the cycle survivable. Wrap the whole drop-alter-recreate-regrant sequence in a transaction so a typo or missed grant rolls back cleanly, and keep that transaction tight — AccessExclusiveLock</code> blocks everything else for its duration. Then map the dependency graph before you cut: pg_depend</code> joined through pg_rewrite</code>, walked with a recursive CTE, tells you what actually breaks. Run that query before the migration, not after the incident.</p> The pain traces back to one missing primitive. ALTER VIEW</code> today handles renames, owner and schema changes, column defaults, and options like security_barrier</code>, but nothing structural. CREATE OR REPLACE VIEW</code> can append columns at the end and nothing else. A real ALTER VIEW DROP COLUMN</code>, ADD COLUMN</code>, ALTER COLUMN TYPE</code> is what would make views safe to evolve. The catalog and the in-place-rewrite mechanism are already there, as pg_dump</code>'s placeholder trick demonstrates; the user-facing DDL is what is missing.</p> Even without it, views are still worth using. Just don't pretend they're tables.</p> EDIT (2026-05-17)</em>: The original article understated how RLS interacts with views. Added a section on security_invoker</code> and why the ownership model matters, plus a note on why materialized views are structurally incompatible with RLS.</p> Don't let AI touch your production database 2026-04-06T22:24:00+02:00 Not so long ago, the biggest threat to production databases was the developer who claimed it worked on their machine. If you've attended my sessions, you know this is a topic I'm particularly sensitive to.</p> These days, AI agents are writing your SQL. The models are getting incredibly good at producing plausible</em> code. It looks right, it feels right, and often it passes a cursory glance. But "plausible" isn't a performance metric, and it doesn't care about your execution plan or locking strategy.</p> AI-generated SQL is syntactically correct, which is the easy part. The hard part is knowing what a statement does</em> to a running system: which locks it takes, how long it holds them, whether it rewrites the table on disk.</p> The spectacular failures get the headlines. In July 2025, an AI coding agent wiped a production database</a> during a code freeze -- ran destructive commands, panicked, then lied about what it had done.</p> But the biggest damage is quieter. It's the migration that passes every test, ships through CI and then locks a production table during peak traffic. The query written with random assumptions. Indexes added based on copy/paste from psql</code>. The cumulative effect that builds over time, when nobody is looking.</p> How to give AI agent "eyes"</a> </h2> When you're running Claude Code or any other agentic coding tool, the bottleneck isn't the model's intelligence. It's the fidelity of the environment. Standard AI coding involves the agent guessing column names based on your description, or in better cases parsing a schema.sql</code> file or using a local database with seed data.</p> None of these give the agent the one thing it actually needs: awareness of your production</em> schema. The table sizes, the indexes, the constraints, the statistics that determine whether a query index-scans in 2ms or sequential-scans for 40 seconds.</p> The obvious fix is to give it a database connection. Let it query pg_catalog</code>, read table sizes, check existing indexes. This is how Anthropic's reference PostgreSQL MCP server worked, and that's the one Datadog Security Labs found a SQL injection in</a>. The vulnerable npm package still gets 21,000 downloads per week. The server wrapped queries in read-only transactions, but accepted semicolons. COMMIT; DROP SCHEMA public CASCADE;</code> works because the COMMIT</code> ends the read-only transaction and everything after runs with full privileges. Read-only transactions are not a security boundary when the protocol accepts semicolons.</p> But here's the thing: the AI doesn't need a connection</em>. It needs the information</em>. Schema metadata is structural data. You can extract it once, save it as a JSON file, commit it to git, and hand that file to every tool in your stack.</p> MCP</a> is an open standard for connecting AI assistants to external tools and data sources. Instead of relying on training data, the agent calls structured tools exposed by an MCP server and gets deterministic answers back. </div> What's been missing is a standard way to wire that information into the agent's workflow. Not a one-time dump that sits in a docs folder, but a protocol that lets the agent call tools</em> and get structured answers back. That protocol is MCP</a> (Model Context Protocol).</p> What are PostgreSQL MCP servers actually for?</a> </h2> This has been bugging me. If you search for "PostgreSQL MCP server" today, you'll find two kinds.</p> The first is a connection wrapper. You give it a DATABASE_URL</code>, it exposes tools like query</code> and execute</code>. This is psql</code> with extra steps. The Anthropic reference server that got the SQL injection? Connection wrapper. The dozen community forks that replaced it? Connection wrappers with slightly better input validation.</p> The second is a managed-provider gateway. Cloud database platforms expose their platform APIs through MCP. Create a branch, provision a database, manage users. Useful if you're on that platform, but not database intelligence. Control-plane operations wearing a protocol hat.</p> Neither kind answers the question an AI agent actually needs answered: "is this SQL going to be a problem?"</em></p> A connection wrapper can run EXPLAIN</code>, if you trust it with credentials. But who tells the agent that SET NOT NULL</code> on a 4-million-row table takes an ACCESS EXCLUSIVE</code> lock? Who tells it that its query defeats partition pruning? That's not a connection problem. It's a knowledge problem.</p> dryrun</a> </h2> I didn't set out to build an AI tool. My primary project is RegreSQL</a>, a statistics-based SQL regression testing framework. RegreSQL needs production statistics to produce realistic query plans in test environments, and dryrun's schema capture was originally just the plumbing to get those statistics into the right place.</p> But driving adoption of a testing tool is hard. The best place to plug in is where the coding actually happens. And these days, that's increasingly AI agents. Instead of fighting it, I turned the plumbing into a proper tool.</p> dryrun</a> is a PostgreSQL schema intelligence tool. It ships as a CLI and as an MCP server. It doesn't connect to your database at runtime. Instead, it works from a JSON snapshot of your schema, captured once, committed to git, and available to everyone.</p> Offline first</a> </h3> One person with production access captures the schema:</p> dryrun</span> dump-schema</span> --source</span> "</span>$PROD_URL</span>"</span> --pretty --name</span> "production"</span> -o</span> schema.json</span></span></code></pre> The JSON contains everything structural: tables, columns, constraints, indexes, and the rest of what you'd find in pg_catalog</code>. It also captures per-table statistics from pg_statistic</code> (histograms, most common values, correlation, n_distinct</code>) and runtime counters from pg_class</code> like reltuples</code>, seq_scan</code>, and idx_scan</code>. One file, typically 200-500KB. Structural metadata, not row data.</p> If your schema uses COMMENT ON</code> (and it should), dryrun captures those too. A column named status</code> is ambiguous. COMMENT ON COLUMN orders.status IS 'pending \| completed \| refunded, set by payment processor webhook'</code> tells the agent what the values mean, where they come from, and that it shouldn't invent new ones. If you don't take my word for it, check what Bruce Momjian has to say</a> about it. MCP clients can only work with schemas, object names, and constraints. Comments are the simplest way to supply the semantic context that makes AI-generated queries meaningful.</p> </div> Import it and every subsequent command works offline:</p> dryrun</span> import schema.json</span></span></code></pre> No connection needed, no credentials on developer machines or in CI.</p> As an MCP server</a> </h3> claude</span> mcp add dryrun</span> --</span> dryrun mcp-serve</span></span></code></pre> Claude now has 16 tools for exploring your schema, validating queries, checking migrations, and suggesting improvements, all from the offline snapshot.</p> What the agent gets</a> </h3> dryrun exposes 16 tools through MCP. The four that matter most for the argument in this article:</p> check_migration</strong> is the one I find myself using most often, even outside of AI workflows. It analyzes DDL statements for lock implications, table rewrites, and version-specific behavior, and returns the safe multi-step alternative when one exists. Every migration example in this article was caught by this tool.</p> suggest_index</strong> reads column statistics from the snapshot and recommends indexes informed by your actual data distribution. Not "add an index on this column" but "add a partial index on the 3% of rows that match your query pattern." It also uses correlation data to recommend BRIN indexes where appropriate: 3MB instead of 400MB for a B-tree on a column with near-perfect insertion order.</p> validate_query</strong> parses SQL against the schema snapshot without executing it. It catches references to tables and columns that don't exist, detects anti-patterns, and flags issues before the query goes anywhere near the database.</p> lint_schema</strong> runs 18 convention rules and 14 structural audit checks: naming, types, constraints, timestamps, partition health, duplicate indexes, foreign key columns missing indexes, tables without primary keys. All disableable in dryrun.toml</code>, because not every convention fits every project.</p> The remaining tools cover schema exploration (list_tables</strong>, describe_table</strong>, search_schema</strong>, find_related</strong>), operations (compare_nodes</strong>, detect</strong>, vacuum_health</strong>), execution plans (explain_query</strong>), and schema maintenance (refresh_schema</strong>, check_drift</strong>). The full list is in the documentation</a>.</p> Try it yourself</a> </h2> Here's a schema and seed data you can run in any local PostgreSQL:</p> CREATE TABLE</span> customers</span> (</span></span> id </span>bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span>,</span></span> email </span>text</span>,</span></span> created_at </span>timestamptz NOT NULL DEFAULT now</span>()</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>bigint NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> total_amount </span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> created_at </span>timestamptz NOT NULL DEFAULT now</span>()</span></span> );</span></span></code></pre> Seed it with data that has the distributions that matter, like skewed status values and nullable emails:</p> -- 100K customers, ~5% with NULL email</span></span> INSERT INTO</span> customers (</span>name</span>, email, created_at)</span></span> SELECT</span></span> 'customer_'</span> \|\|</span> i,</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>05</span> THEN NULL</span></span> ELSE</span> 'user'</span> \|\|</span> i </span>\|\|</span> '@example.com'</span></span> END</span>,</span></span> '2023-01-01'</span>::</span>timestamptz +</span> (i </span>\|\|</span> ' seconds'</span>)::interval</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>) </span>AS</span> i;</span></span> </span> -- 1M orders, 97% completed / 3% pending</span></span> INSERT INTO</span> orders (customer_id, </span>status</span>, total_amount, created_at)</span></span> SELECT</span></span> 1</span> +</span> (random()</span> </span> 99999</span>)::</span>int</span>,</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>03</span> THEN</span> 'pending'</span></span> ELSE</span> 'completed'</span></span> END</span>,</span></span> (random()</span> </span> 500</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> '2024-01-01'</span>::</span>timestamptz +</span> (i </span>\|\|</span> ' seconds'</span>)::interval</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>1000000</span>) </span>AS</span> i;</span></span> </span> ANALYZE;</span></span></code></pre> Now capture the schema for dryrun and install it as an MCP server:</p> dryrun</span> dump-schema</span> --source</span> "</span>$DATABASE_URL</span>"</span> -o</span> schema.json</span></span> dryrun</span> init</span></span> dryrun</span> import schema.json</span></span> claude</span> mcp add dryrun</span> --</span> dryrun mcp-serve</span></span></code></pre> Open Claude Code and try these prompts. First without the dryrun MCP server, then with it.</p> "Add an index to speed up queries that filter orders by status"</a> </h3> Without dryrun, Claude sees a text</code> column and does the obvious thing:</p> CREATE INDEX</span> idx_orders_status</span> ON</span> orders (</span>status</span>);</span></span></code></pre> A full B-tree index on 1 million rows. It works, but 97% of those rows are 'completed'</code> and nobody queries for completed orders.</p> With dryrun, Claude calls suggest_index</strong>, sees most_common_freqs</code> showing 'completed'</code> at 97% and 'pending'</code> at 3%, and suggests:</p> CREATE INDEX</span> idx_orders_status_pending</span> ON</span> orders (</span>status</span>)</span></span> WHERE status =</span> 'pending'</span>;</span></span></code></pre> The full index maintains 1 million entries. The partial index maintains 30,000. Every INSERT that creates a 'completed'</code> order no longer pays for index maintenance it'll never use.</p> "Make the email column on customers required"</a> </h3> Without dryrun, Claude writes a single statement and moves on:</p> ALTER TABLE</span> customers </span>ALTER</span> COLUMN email </span>SET NOT NULL</span>;</span></span></code></pre> This fails immediately because 5% of rows have NULL emails. Claude didn't know the data existed. Even if the column had no NULLs, the statement would still take an ACCESS EXCLUSIVE</code> lock and perform a full table scan to verify.</p> With dryrun, Claude calls check_migration</strong> and describe_table</strong>. It sees 100,000 rows, knows the column is nullable, and knows from the statistics that NULLs exist. It generates a three-step migration:</p> -- Step 1: backfill NULLs</span></span> UPDATE</span> customers</span></span> SET</span> email </span>=</span> 'unknown-'</span> \|\|</span> id </span>\|\|</span> '@example.com'</span></span> WHERE</span> email </span>IS NULL</span>;</span></span> </span> -- Step 2: add constraint without full table scan</span></span> ALTER TABLE</span> customers</span></span> ADD CONSTRAINT</span> customers_email_not_null</span></span> CHECK</span> (email </span>IS NOT NULL</span>) </span>NOT</span> VALID;</span></span> </span> -- Step 3: validate in the background with a weaker lock</span></span> ALTER TABLE</span> customers</span></span> VALIDATE </span>CONSTRAINT</span> customers_email_not_null;</span></span></code></pre> The first step handles the data. The second adds the constraint without scanning existing rows. The third validates with a weaker lock that allows concurrent reads and writes. Three statements instead of one.</p> Keeping the snapshot fresh</a> </h2> dryrun is only as good as your last snapshot. If the schema changed yesterday and nobody re-dumped, the agent is working from stale data. The check_drift</strong> tool can detect this, but only if someone runs it.</p> The best approach is to treat schema snapshots like you treat production backups: automate them. If you're already capturing production schema dumps on a schedule, add dryrun to the same pipeline. A nightly dump-schema</code> from the primary gives you current structure and statistics. If you run read replicas with different workloads, add --stats-only</code> dumps from each node for higher-fidelity statistics across the cluster. Commit the resulting schema.json</code> to your repository so every developer and every AI agent starts the day with a current view.</p> The snapshot also serves as a lightweight audit trail. If someone adds a column through psql</code> without a migration, the next scheduled dump captures it, and the diff in git shows exactly what changed and when.</p> What dryrun is not</a> </h2> dryrun does not execute arbitrary SQL against your database. It does not manage migrations. It does not provision infrastructure. It is not an ORM, a query builder, or a database admin panel. It's read-only and narrow by design, trustworthy enough that you'd let an AI agent use it unsupervised.</p> What's next</a> </h2> dryrun ships today at v0.5.0. The next releases are focused on:</p> Shadow EXPLAIN</strong> (v0.6.0): inject production statistics into a local PostgreSQL instance so EXPLAIN</code> produces realistic plans from your laptop, without any production data. Currently explain_query</strong> runs against whatever local database you point it at, but if that database has 10 rows in every table, the planner gives you toy plans that tell you nothing about production behavior.</li> CI/CD Trust Gate</strong>: a single command that takes a migration file and runs safety analysis, schema diff, lint, and index suggestions in one pass. Designed to block unsafe migrations before they merge.</li> Improved migration support</strong>: deeper analysis of multi-statement migrations, version-specific advice, and integration with common migration frameworks.</li> </ul> Getting started</a> </h2> dryrun is MIT licensed and available on GitHub</a>. The tutorial</a> covers installation, offline and online workflows, multi-node setups, and MCP integration. It's part of the boringSQL</a> suite.</p> Good CTE, bad CTE 2026-03-29T14:47:00+02:00 The Common Table Expression</a>, or CTE, is often the first feature developers reach for beyond basic SQL, and often the only one. You write a subquery after WITH</code>, give it a name, and use it in the rest of your query. It only exists for the duration of that query.</p> But the popularity of CTEs usually has less to do with modernizing code and more to do with the promise of imperative logic. For many, CTE acts as an easy to understand remedy for 'scary queries' and way how to force execution order on the database. The way how many write queries is as if they tell optimizer "first do this, then do that".</p> This creates a problem. CTEs handle query decomposition, recursion and multi statement DDLs. Planner treats them differently depending how you write and use them though. For long time (prior PostgreSQL 12) CTEs acted as optimization fence. The planner couldn't push predicates into them, couldn't use indexes on the underlying tables. Couldn't do anything that materialize them and scan through the result.</p> PostgreSQL 12 changed this. CTEs now get inlined, materialized, or something in between, depending on how you write them.</p> Sample schema</a> </h2> We will use the same schema as in the article PostgreSQL Statistics: Why queries run slow</a>.</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> note </span>text</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> CREATE TABLE</span> orders_archive</span> (</span>LIKE</span> orders INCLUDING ALL EXCLUDING </span>IDENTITY</span>);</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span> SELECT</span></span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span> '2022-01-01'</span>::</span>date +</span> (random()</span> </span> 1095</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> ANALYZE customers;</span></span> ANALYZE orders;</span></span></code></pre> For recursive examples later on we'll also need an employees</code> table with a self-referencing hierarchy:</p> CREATE TABLE</span> employees</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span>,</span></span> manager_id </span>integer REFERENCES</span> employees(id),</span></span> department </span>text NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> employees (</span>name</span>, manager_id, department) </span>VALUES</span></span> (</span>'Alice'</span>, </span>NULL</span>, </span>'Engineering'</span>),</span></span> (</span>'Bob'</span>, </span>1</span>, </span>'Engineering'</span>),</span></span> (</span>'Charlie'</span>, </span>1</span>, </span>'Engineering'</span>),</span></span> (</span>'Diana'</span>, </span>2</span>, </span>'Engineering'</span>),</span></span> (</span>'Eve'</span>, </span>2</span>, </span>'Engineering'</span>),</span></span> (</span>'Frank'</span>, </span>3</span>, </span>'Sales'</span>),</span></span> (</span>'Grace'</span>, </span>3</span>, </span>'Sales'</span>),</span></span> (</span>'Hank'</span>, </span>6</span>, </span>'Sales'</span>),</span></span> (</span>'Ivy'</span>, </span>6</span>, </span>'Sales'</span>);</span></span> </span> ANALYZE employees;</span></span></code></pre>The optimization fence era (pre-PG 12)</a> </h2> As we already covered before PostgreSQL 12, every CTE was materialized. No exceptions. The planner would compute the CTE result set in full, store it in a temporary tuplestore, and then scan that tuplestore whenever the main query referenced the CTE. This made CTEs an optimization fence</strong> because the planner could not look through them.</p> Consider this simple query:</p> EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span> SELECT FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> On PostgreSQL 11 and earlier, the EXPLAIN</code> output would look something like this:</p> QUERY PLAN</span></span> -------------------------------------------------------------------</span></span> CTE Scan on filtered (cost=1840.00..2290.00 rows=2 width=58)</span></span> Filter: (status = 'pending')</span></span> CTE filtered</span></span> -> Seq Scan on orders (cost=0.00..1840.00 rows=10000 width=58)</span></span> Filter: (created_at > '2025-01-01'::date)</span></span></code></pre> Notice what happens here. The CTE runs a sequential scan on orders</code> with the date filter. It materializes all matching rows. Then the outer query applies the status = 'pending'</code> filter after</em> materialization. Even if a composite index on (created_at, status)</code> existed, the planner couldn't use it as it can't see through the CTE boundary to combine the predicates.</p> Why was it designed this way? Two reasons. First, reason was snapshot isolation. Materializing the CTE guaranteed that the result set was computed once, from a single snapshot, regardless of how many times it was referenced. Second, as protection for side-effect edge cases. If a CTE contained a data-modifying statement (INSERT</code>, UPDATE</code>, DELETE</code>), materialising ensured it executed exactly once.</p> The workaround was well-known in the community: rewrite CTEs as subqueries. Subqueries had always been subject to the planner's normal optimisation rules, including predicate pushdown and inlining. The same query written as SELECT * FROM (SELECT * FROM orders WHERE created_at > '2025-01-01') sub WHERE status = 'pending'</code> would produce a much better plan.</div> This led to an entire workaround culture. Developers would write queries with CTEs for readability during development, then rewrite them as nested subqueries for production. The community had a saying: CTEs are optimization fences</em>. It was repeated so often. Many developers still believe it today. But it hasn't been true since PostgreSQL 12.</p> PostgreSQL 12: CTE inlining</a> </h2> PostgreSQL 12 introduced automatic CTE inlining. Non-recursive, side-effect-free, singly-referenced CTEs are now inlined by default. The planner started treating them as subqueries and applies all its normal optimisations. Predicate pushdown, index usage, join reordering apply exactly as if the CTE syntax never existed.</p> The same query from the previous section now produces a completely different plan:</p> EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2355.00 rows=2 width=58)</span></span> Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span></code></pre> The CTE is gone from the plan entirely. Both predicates are merged into a single scan on orders</code>. If there's a suitable index, the planner can use it. The CTE syntax doesn't change the execution plan.</p> PostgreSQL 12 also introduced two new keywords that let you override the planner's decision:</p> MATERIALIZED</code> - forces the CTE to materialize, even when the planner would inline it</li> NOT MATERIALIZED</code> - forces inlining, even when the planner would materialize it</li> </ul> -- force materialization</span></span> EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span> </span> -- Force inlining</span></span> EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> This follows the same principle as VIEW inlining</a>.</p> When does a CTE get materialized?</a> </h2> Case 1: single reference, no side effects (INLINED)</a> </h3> The simplest and most common case. If you reference the CTE exactly once and it contains no side effects, the planner inlines it.</p> EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2355.00 rows=2 width=59)</span></span> Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span> (2 rows)</span></span></code></pre> Both predicates are merged. The planner considers all access paths on orders</code> directly.</p> Case 2: multiple references (MATERIALIZED)</a> </h3> When a CTE is referenced more than once, the planner materializes it. This is actually a feature, CTE is computed once and reused. Therefore avoiding redundant work.</p> EXPLAIN </span>WITH</span> summary </span>AS</span> (</span></span> SELECT status</span>, </span>count</span>(</span></span>) </span>AS</span> cnt </span>FROM</span> orders </span>GROUP BY status</span></span> )</span></span> SELECT</span> a</span>.</span>status</span>, </span>b</span>.</span>status</span></span> FROM</span> summary a, summary b</span></span> WHERE</span> a</span>.</span>cnt</span> ></span> b</span>.</span>cnt</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------</span></span> Nested Loop (cost=2355.04..2355.52 rows=5 width=64)</span></span> Join Filter: (a.cnt > b.cnt)</span></span> CTE summary</span></span> -> HashAggregate (cost=2355.00..2355.04 rows=4 width=17)</span></span> Group Key: orders.status</span></span> -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=9)</span></span> -> CTE Scan on summary a (cost=0.00..0.08 rows=4 width=40)</span></span> -> CTE Scan on summary b (cost=0.00..0.08 rows=4 width=40)</span></span> (8 rows)</span></span></code></pre> The CTE Scan</code> nodes appear twice, but the HashAggregate</code> runs only once. For expensive computations referenced multiple times, this is exactly what you want.</p> Case 3: recursive CTE (ALWAYS MATERIALIZED)</a> </h3> Recursive CTEs must maintain a working table between iterations. There's no way to inline them. We'll cover recursion in detail later in the article.</p> EXPLAIN </span>WITH RECURSIVE</span> subordinates </span>AS</span> (</span></span> SELECT</span> id, </span>name</span>, manager_id </span>FROM</span> employees </span>WHERE</span> id </span>=</span> 1</span></span> UNION ALL</span></span> SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span></span> FROM</span> employees e</span></span> JOIN</span> subordinates s </span>ON</span> e</span>.</span>manager_id</span> =</span> s</span>.</span>id</span></span> )</span></span> SELECT FROM</span> subordinates;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> CTE Scan on subordinates (cost=17.21..18.83 rows=81 width=40)</span></span> CTE subordinates</span></span> -> Recursive Union (cost=0.00..17.21 rows=81 width=13)</span></span> -> Seq Scan on employees (cost=0.00..1.11 rows=1 width=13)</span></span> Filter: (id = 1)</span></span> -> Hash Join (cost=0.33..1.53 rows=8 width=13)</span></span> Hash Cond: (e.manager_id = s.id)</span></span> -> Seq Scan on employees e (cost=0.00..1.09 rows=9 width=13)</span></span> -> Hash (cost=0.20..0.20 rows=10 width=4)</span></span> -> WorkTable Scan on subordinates s (cost=0.00..0.20 rows=10 width=4)</span></span> (10 rows)</span></span></code></pre>Case 4: data-modifying CTE (ALWAYS MATERIALIZED)</a> </h3> CTEs that contain INSERT</code>, UPDATE</code>, or DELETE</code> are always materialized. The side effects must execute exactly once, in a predictable order.</p> EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span> DELETE FROM</span> orders </span>WHERE status =</span> 'cancelled'</span> RETURNING </span></span></span> )</span></span> SELECT</span> count</span>(</span></span>) </span>FROM</span> deleted;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------</span></span> Aggregate (cost=2670.13..2670.14 rows=1 width=8)</span></span> CTE deleted</span></span> -> Delete on orders (cost=0.00..2105.00 rows=25117 width=6)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25117 width=6)</span></span> Filter: (status = 'cancelled'::text)</span></span> -> CTE Scan on deleted (cost=0.00..502.34 rows=25117 width=0)</span></span> (6 rows)</span></span></code></pre> The CTE Scan</code> is present because the DELETE</code> must be fully executed before the count()</code> can run.</p> Case 5: VOLATILE function (MATERIALIZED)</a> </h3> If a CTE contains a VOLATILE</code> function, the planner materializes it to prevent the function from being evaluated multiple times with potentially different results.</p> EXPLAIN </span>WITH</span> rand </span>AS</span> (</span></span> SELECT</span> id, random()</span> AS</span> r </span>FROM</span> orders</span></span> )</span></span> SELECT FROM</span> rand </span>WHERE</span> r </span><</span> 0</span>.</span>01</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------</span></span> CTE Scan on rand (cost=2105.00..4355.00 rows=33333 width=12)</span></span> Filter: (r < '0.01'::double precision)</span></span> CTE rand</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=100000 width=12)</span></span> (4 rows)</span></span></code></pre> Even though rand</code> is referenced only once, the CTE Scan</code> is there. random()</code> is VOLATILE</code>, which forces materialization.</p> Case 6: STABLE functions (INLINED)</a> </h3> STABLE</code> functions like now()</code> do not</strong> prevent inlining. The reason is that the time is frozen at transaction start.</p> EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span> SELECT * FROM</span> orders</span></span> WHERE</span> created_at </span>> now</span>()</span> -</span> interval </span>'7 days'</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2855.00 rows=2 width=59)</span></span> Filter: ((status = 'pending'::text) AND (created_at > (now() - '7 days'::interval)))</span></span> (2 rows)</span></span></code></pre> No CTE Scan</code>. The planner inlines the CTE and merges both predicates, just like Case 1. now()</code> is STABLE</code> as it returns the same value within a transaction - and the planner's inlining check only looks for VOLATILE</code> functions (via contain_volatile_functions()</code>). STABLE</code> passes that check.</p> Why do people think STABLE</code> blocks inlining? Because before PostgreSQL 12, all</strong> CTEs were materialized regardless of volatility. When PG 12 introduced inlining, the only function-level barrier was VOLATILE</code>. But the old "CTEs are optimization fences" mental model was so deeply ingrained that many developers assumed STABLE</code> was also a problem. It isn't.</p> The function that actually blocks inlining: clock_timestamp()</code>. Unlike now()</code>, it's VOLATILE</code> and returns a different value on every call. A CTE with clock_timestamp()</code> will be materialized. Similarly, random()</code> and nextval()</code> are VOLATILE</code> and force materialization (as shown in Case 5).</p> </div> If you see a CTE with now()</code> being materialized, the cause is something else. Either multiple references, a data-modifying statement, or an explicit MATERIALIZED</code> hint. Don't blame STABLE</code>.</p> Case 7: forcing behaviour with hints</a> </h3> You can always override the planner's decision.</p> -- force materialization on something that would normally be inlined</span></span> EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------</span></span> CTE Scan on filtered (cost=2105.00..2670.58 rows=5290 width=92)</span></span> Filter: (amount > '400'::numeric)</span></span> CTE filtered</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> (5 rows)</span></span></code></pre>-- force inlining on something that would normally be materialized</span></span> EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span> )</span></span> SELECT * FROM</span> filtered a</span></span> JOIN</span> filtered b </span>ON</span> a</span>.</span>customer_id</span> =</span> b</span>.</span>customer_id</span>;</span></span></code></pre> QUERY PLAN</span></span> --------------------------------------------------------------------------------</span></span> Hash Join (cost=2419.21..10686.65 rows=317742 width=118)</span></span> Hash Cond: (orders.customer_id = orders_1.customer_id)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> -> Hash (cost=2105.00..2105.00 rows=25137 width=59)</span></span> -> Seq Scan on orders orders_1 (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> (7 rows) </span></span></code></pre> If the CTE were materialized, you would see a single scan of the orders</code> table, followed by two CTE Scans on the result. Instead, the planner has treated your query as if you had written a standard join between two subqueries.</p> Be careful with NOT MATERIALIZED</code> on multiply-referenced CTEs.</strong> When you force inlining, the subquery runs once for each reference. In the example above, the orders</code> table is scanned twice. Once for a</code> and once for b</code>. For small result sets this might be fine. For large ones, you're doing double the work. Measure before using.</p> </div> Case 8: FOR UPDATE / FOR SHARE (MATERIALIZED)</a> </h3> Row-locking clauses force materialization even on a singly-referenced, side-effect-free CTE. Internally, the planner's contain_dml()</code> check treats FOR UPDATE</code> and FOR SHARE</code> the same as data-modifying statements.</p> EXPLAIN </span>WITH</span> locked </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span> FOR UPDATE</span></span> )</span></span> SELECT * FROM</span> locked </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------</span></span> CTE Scan on locked (cost=2356.37..2921.95 rows=5290 width=92)</span></span> Filter: (amount > '400'::numeric)</span></span> CTE locked</span></span> -> LockRows (cost=0.00..2356.37 rows=25137 width=65)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=65)</span></span> Filter: (status = 'pending'::text)</span></span> (6 rows)</span></span></code></pre> Without FOR UPDATE</code>, this CTE would be inlined. The LockRows</code> node and CTE Scan</code> confirm materialization.</p> Decision matrix</a> </h3> Here's the full picture across PostgreSQL versions:</p> Condition</th> PG ≤ 11</th> PG 12–16</th> PG 17–18</th></tr></thead> Single ref, pure SELECT</td> Materialized</td> Inlined</td> Inlined</td></tr> Multiple refs, pure SELECT</td> Materialized</td> Materialized</td> Materialized (better stats)</td></tr> VOLATILE function</td> Materialized</td> Materialized</td> Materialized</td></tr> STABLE function</td> Materialized</td> Inlined</td> Inlined</td></tr> Data-modifying (DML)</td> Materialized</td> Materialized</td> Materialized</td></tr> FOR UPDATE / FOR SHARE</td> Materialized</td> Materialized</td> Materialized</td></tr> Recursive</td> Materialized</td> Materialized</td> Materialized</td></tr> Explicit MATERIALIZED</code></td> -</td> Materialized</td> Materialized</td></tr> Explicit NOT MATERIALIZED</code></td> -</td> Inlined</td> Inlined</td></tr> </tbody></table> The statistics black hole</a> </h2> As mentioned in PostgreSQL Statistics: Why queries run slow</a>, materialized CTEs are one of the places "where no statistics go." This is arguably the biggest practical problem with CTE materialization.</p> When the planner materializes a CTE, the result set is stored in a temporary tuplestore. This tuplestore has no pg_statistic</code> entries - no histograms, no MCVs, no correlation data. The planner has to estimate row counts and value distributions using hardcoded defaults.</p> Let's see this in action. Here's a CTE over 10,000 rows:</p> EXPLAIN </span>WITH</span> all_orders </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> all_orders </span>WHERE status =</span> 'pending'</span> AND</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------</span></span> CTE Scan on all_orders (cost=1855.00..4355.00 rows=5290 width=92)</span></span> Filter: ((amount > '400'::numeric) AND (status = 'pending'::text))</span></span> CTE all_orders</span></span> -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=59)</span></span> (4 rows)</span></span></code></pre> The planner estimated 5,290 rows. Where does that number come from? The planner has no MCV list for status</code> inside the CTE, no histogram for amount</code>. It falls back to default selectivities of 0.3333</code> for the range comparison on amount</code> and a rough guess for the equality on status</code>, and multiplies them against the 100,000 input rows.</p> If this CTE were inlined, the planner would read the actual statistics from pg_statistic</code> for the orders</code> table and produce estimates based on real data distribution, not defaults.</div> In a simple query this might not matter much. But when a materialized CTE feeds into a join, default-based estimates can cascade. The planner might choose a nested loop where a hash join would be better, or vice versa. It might underestimate memory needs and spill to disk unexpectedly.</p> PG 17: Statistics propagation</a> </h3> PostgreSQL 17 brought two significant improvements to materialized CTEs:</p> Column statistics propagation.</strong> When the planner creates a CTE Scan</code> node, it now propagates column statistics from the underlying query into the scan node. This means n_distinct</code>, MCV lists, and histograms from the source table can inform estimates on the CTE scan.</p> Path key propagation.</strong> Materialized CTEs now preserve sort order information. If the CTE's subquery produces sorted output, the planner knows about it and can skip redundant sorts downstream.</p> These improvements significantly reduce the estimation gap, but they don't eliminate it. Inlined CTEs are still strictly better for planning accuracy, because the planner works directly with the base table statistics rather than propagated copies. If your CTE doesn't need to be materialized, don't force it.</p> When materialization helps</a> </h2> Materialization isn't always bad.</p> Multiple references.</strong> If the CTE result is used in multiple places, materialization computes it once. Without it, the subquery runs once per reference.</p> EXPLAIN </span>WITH</span> monthly_totals </span>AS</span> (</span></span> SELECT</span> date_trunc(</span>'month'</span>, created_at) </span>AS month</span>,</span></span> status</span>,</span></span> sum</span>(amount) </span>AS</span> total</span></span> FROM</span> orders</span></span> GROUP BY</span> 1</span>, </span>2</span></span> )</span></span> SELECT</span> cur</span>.</span>month</span>, </span>cur</span>.</span>status</span>, </span>cur</span>.</span>total</span>,</span></span> prev</span>.</span>total</span> AS</span> prev_month_total,</span></span> cur</span>.</span>total</span> -</span> prev</span>.</span>total</span> AS</span> delta</span></span> FROM</span> monthly_totals cur</span></span> LEFT JOIN</span> monthly_totals prev</span></span> ON</span> cur</span>.</span>month</span> =</span> prev</span>.</span>month</span> +</span> interval </span>'1 month'</span></span> AND</span> cur</span>.</span>status</span> =</span> prev</span>.</span>status</span>; </span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> Merge Left Join (cost=3887.46..3990.90 rows=4384 width=136)</span></span> Merge Cond: ((cur.month = ((prev.month + '1 mon'::interval))) AND (cur.status = prev.status))</span></span> CTE monthly_totals</span></span> -> HashAggregate (cost=3105.00..3181.72 rows=4384 width=49)</span></span> Group Key: date_trunc('month'::text, (orders.created_at)::timestamp with time zone), orders.status</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=100000 width=23)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: cur.month, cur.status</span></span> -> CTE Scan on monthly_totals cur (cost=0.00..87.68 rows=4384 width=72)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: ((prev.month + '1 mon'::interval)), prev.status</span></span> -> CTE Scan on monthly_totals prev (cost=0.00..87.68 rows=4384 width=72)</span></span> (12 rows)</span></span></code></pre> The aggregation runs once. Both cur</code> and prev</code> read from the materialized result. Without materialization, the entire aggregation would run twice.</p> Expensive VOLATILE expressions.</strong> If a CTE contains calls to volatile functions or expensive computations, materialization ensures they execute exactly once.</p> Data-modifying operations.</strong> The whole point of writable CTEs is that the side effects happen once and the RETURNING</code> data is available downstream. Materialization is not optional here.</p> When inlining isn't enough</a> </h2> The imperative mindset from the introduction, "first do this, then do that", doesn't go away just because the planner inlines your CTEs. And this one is my personal favourite, and source that never stop delivering query refactoring.</p> Developers still structure queries as sequential pipelines, and that structure itself can create performance problems that have nothing to do with materialization.</p> A common pattern is building queries as an assembly line: one CTE filters rows, the next LEFT JOINs related tables and aggregates metadata with GROUP BY</code>, the next filters on the aggregated results. It reads like a clean pipeline, but the GROUP BY</code> in the middle creates a wall the planner can't optimize past.</p> WITH</span> recent_orders </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2024-01-01'</span></span> ),</span></span> order_metadata </span>AS</span> (</span></span> SELECT</span></span> o</span>.</span>id</span>,</span></span> bool_or(</span>oa</span>.</span>id</span> IS NOT NULL</span>) </span>AS</span> was_archived,</span></span> count</span>(</span>o2</span>.</span>id</span>) </span>AS</span> related_count</span></span> FROM</span> recent_orders o</span></span> LEFT JOIN</span> orders_archive oa </span>ON</span> o</span>.</span>id</span> =</span> oa</span>.</span>id</span></span> LEFT JOIN</span> orders o2 </span>ON</span> o</span>.</span>customer_id</span> =</span> o2</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> GROUP BY</span> o</span>.</span>id</span></span> )</span></span> SELECT</span> o.</span></span>, </span>m</span>.</span>was_archived</span>, </span>m</span>.</span>related_count</span></span> FROM</span> recent_orders o</span></span> JOIN</span> order_metadata m </span>ON</span> o</span>.</span>id</span> =</span> m</span>.</span>id</span></span> WHERE</span> m</span>.</span>was_archived</span> =</span> false</span></span> AND</span> m</span>.</span>related_count</span> ></span> 0</span>;</span></span></code></pre> Each CTE here is referenced once, so they all get inlined. No materialization, no optimization fence. The planner sees the full query. So what's the problem?</p> The GROUP BY</code> in order_metadata</code>. Even after inlining, the planner cannot push the was_archived = false</code> predicate past the aggregation. It must first LEFT JOIN every filtered order against orders_archive</code> and self-join orders</code>, compute the aggregates for all of them, and only then discard the rows that don't match. If recent_orders</code> returns 50,000 rows but only 200 were ever archived, you're joining and aggregating 49,800 rows for nothing.</p> The fix is to replace the aggregation-then-filter pattern with correlated EXISTS</code> subqueries:</p> SELECT</span> o.</span></span></span> FROM</span> orders o</span></span> WHERE</span> o</span>.</span>created_at</span> ></span> '2024-01-01'</span></span> AND NOT EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders_archive oa </span>WHERE</span> oa</span>.</span>id</span> =</span> o</span>.</span>id</span></span> )</span></span> AND EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders o2</span></span> WHERE</span> o2</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> );</span></span></code></pre> EXISTS</code> short-circuits after finding the first matching row. The planner can push created_at > '2024-01-01'</code> all the way down to an index scan on orders</code>, then probe each related table per result. No aggregation, no wasted work.</p> The rule of thumb:</strong> if your CTE contains a GROUP BY</code> or a LEFT JOIN</code> just to compute a boolean ("does this row have related data?"), you've built a wall the planner can't see past. A correlated EXISTS</code> lets the planner push filters down and stop scanning early. This applies whether the CTE is materialized or inlined.</p> </div> Writable CTEs (the power and the traps)</a> </h2> Data-modifying CTEs let you INSERT</code>, UPDATE</code>, or DELETE</code> inside a WITH</code> clause and use the RETURNING</code> data in subsequent CTEs or the main query.</p> EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span> DELETE FROM</span> orders</span></span> WHERE status =</span> 'cancelled'</span></span> AND</span> created_at </span><</span> '2023-01-01'</span></span> RETURNING </span></span></span> ),</span></span> archived </span>AS</span> (</span></span> INSERT INTO</span> orders_archive</span></span> SELECT FROM</span> deleted</span></span> RETURNING id</span></span> )</span></span> SELECT</span> count</span>(</span></span>) </span>FROM</span> archived;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------</span></span> Aggregate (cost=2709.19..2709.20 rows=1 width=8)</span></span> CTE deleted</span></span> -> Delete on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> Filter: ((created_at < '2023-01-01'::date) AND (status = 'cancelled'::text))</span></span> CTE archived</span></span> -> Insert on orders_archive (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on deleted (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on archived (cost=0.00..166.68 rows=8334 width=0)</span></span> (9 rows) </span></span></code></pre> This deletes old cancelled orders, moves them to an archive table, and counts how many were archived - all in a single atomic statement, no application-level coordination needed.</p> But there are sharp edges.</p> You cannot read what you just wrote</a> </h3> All sub-statements in a data-modifying CTE see the same snapshot. This means the effects of one CTE are not visible</strong> to other CTEs or the main query when they read the target table</em>. Only the RETURNING</code> clause communicates data between CTE steps.</p> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre>WITH</span> ins </span>AS</span> (</span></span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, created_at)</span></span> VALUES</span> (</span>1</span>, </span>100</span>.</span>00</span>, </span>'pending'</span>, CURRENT_DATE)</span></span> RETURNING id</span></span> )</span></span> -- this does NOT see the row we just inserted</span></span> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre> The count(1)</code> query sees the pre-insert snapshot. If you need the inserted data, you must use the RETURNING</code> clause from the ins</code> CTE, not re-read the table.</p> Tuple shuffling</a> </h3> A common pattern is using writable CTEs to move rows between tables atomically:</p> WITH</span> moved </span>AS</span> (</span></span> DELETE FROM</span> orders_staging</span></span> RETURNING </span></span></span> )</span></span> INSERT INTO</span> orders</span></span> SELECT * FROM</span> moved;</span></span></code></pre> This deletes all rows from the staging table and inserts them into the production table in a single atomic operation. No window where data exists in both or neither table.</p> Data-modifying CTEs disable parallel query for the entire statement. If you have a complex query that mixes reads and writes, the write CTE prevents parallelism even for the read-only parts.</div> Recursive CTEs are always materialized</a> </h2> Recursive CTEs use an iterative working-table mechanism. Despite the name, they aren't truly recursive. PostgreSQL doesn't "call itself" by creating a nested stack of unfinished queries. Instead, it operates in loops.</p> Execute the non-recursive term (the "seed"). Put results in the working table.</li> Execute the recursive term using the working table as input. New rows become the next</em> working table.</li> Repeat until the recursive term returns no new rows.</li> Return the union of all iterations.</li> </ol> WITH RECURSIVE</span> org_chart </span>AS</span> (</span></span> -- Seed: start from the CEO</span></span> SELECT</span> id, </span>name</span>, manager_id, </span>1</span> AS</span> depth</span></span> FROM</span> employees</span></span> WHERE</span> manager_id </span>IS NULL</span></span> </span> UNION ALL</span></span> </span> -- Recursive term: find direct reports</span></span> SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span>, </span>oc</span>.</span>depth</span> +</span> 1</span></span> FROM</span> employees e</span></span> JOIN</span> org_chart oc </span>ON</span> e</span>.</span>manager_id</span> =</span> oc</span>.</span>id</span></span> )</span></span> SELECT * FROM</span> org_chart </span>ORDER BY</span> depth, </span>name</span>;</span></span></code></pre> id \| name \| manager_id \| depth</span></span> ----+---------+------------+-------</span></span> 1 \| Alice \| \| 1</span></span> 2 \| Bob \| 1 \| 2</span></span> 3 \| Charlie \| 1 \| 2</span></span> 4 \| Diana \| 2 \| 3</span></span> 5 \| Eve \| 2 \| 3</span></span> 6 \| Frank \| 3 \| 3</span></span> 7 \| Grace \| 3 \| 3</span></span> 8 \| Hank \| 6 \| 4</span></span> 9 \| Ivy \| 6 \| 4</span></span> (9 rows)</span></span></code></pre>UNION vs UNION ALL</a> </h3> The choice between UNION</code> and UNION ALL</code> in recursive CTEs matters more than in regular queries.</p> UNION ALL</code> keeps all rows, including duplicates. This is faster but dangerous: if your graph has cycles, the recursion never terminates. PostgreSQL will run until you cancel the query or memory is exhausted.</p> UNION</code> deduplicates at each iteration. This prevents infinite loops in cyclic graphs but adds the cost of hashing and comparing rows at every step.</p> PostgreSQL 14 added SQL-standard SEARCH</code> and CYCLE</code> clauses that replace the manual patterns for controlling traversal order (breadth-first vs. depth-first) and detecting cycles. SEARCH BREADTH FIRST BY</code> / SEARCH DEPTH FIRST BY</code> control the order, while CYCLE</code> automatically detects and marks cycles, which is far cleaner than the old pattern of accumulating an array of visited IDs.</p> For pure hierarchical data (trees without cycles), consider the ltree</code> extension as an alternative to recursive CTEs. It stores the full path as a label tree and supports efficient ancestor/descendant queries with GiST indexes. The trade-off is denormalized storage vs. on-the-fly recursion.</p> </div> The exotic edge cases</a> </h2> Partition pruning lost</a> </h3> When you materialize a CTE over a partitioned table, partition pruning cannot happen on the CTE scan side. The materialized result is a flat tuplestore disconnected from the partition metadata.</p> -- assume orders is range-partitioned by created_at</span></span> WITH</span> recent </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE</span> created_at </span>></span> '2025-06-01'</span>;</span></span></code></pre> The created_at > '2025-06-01'</code> predicate is applied after</em> materialization. All partitions are scanned to build the CTE, even though only one or two would be needed. Use NOT MATERIALIZED</code> (or simply let the planner inline it) to preserve partition pruning.</p> Prepared statements and plan caching</a> </h3> PostgreSQL generates custom plans for the first 5 executions of a prepared statement. After that, it may switch to a generic plan. CTE inlining decisions can differ between custom and generic plans, because generic plans don't know the actual parameter values.</p> This means a CTE that gets inlined during your first 5 calls might start materializing on the 6th or vice versa. If you see sudden plan changes with prepared statements, check whether the CTE inlining behaviour has shifted.</p> work_mem spilling</a> </h3> Materialized CTEs store their results in memory, bounded by work_mem</code>. When the result set exceeds this limit, it silently spills to disk as a temporary file. This is not an error - it just gets slower.</p> Monitor with log_temp_files = 0</code> (logs all temp files) or check EXPLAIN (ANALYZE, BUFFERS)</code> for temp read/write counts.</p> PostgreSQL 18: EXPLAIN now shows memory/disk usage</strong> Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> reports memory and disk usage for Material nodes, including CTE materialization. You can see exactly how much memory a materialized CTE consumed and whether it spilled to disk.</p> </div> CTE and security barrier views</a> </h3> A `security_barrier` view is a "black box" that forces the database to fully resolve the view's internal logic before any outer filters are applied.</div> Security-barrier views already prevent subquery flattening as a security measure (to stop user-defined functions from seeing rows they shouldn't). When you combine a security-barrier view with a CTE, you compound the optimisation barriers. The planner can neither inline the view nor the CTE. If performance matters in this scenario, consider materializing the security-sensitive filtering into a temporary table first. CTE vs. subquery vs. temporary table</a> </h2> CTE, referenced once</strong> the planner inlines it on PG 12+, so it doesn't affect the execution plan. This is the default choice for breaking up complex queries.</p> CTE, referenced multiple times (small result)</strong> represents acceptable cost. Materialization means the subquery runs once. For aggregations or filtered subsets that produce a few hundred rows, the overhead is minimal.</p> As Henrietta Dombrovskaya</a> emphasizes, "The best temporary table is the one you didn't create". Always exhaust your indexing and query-rewriting options before reaching for a temp table, as the DDL overhead often outweighs the execution gains.</div> CTE, referenced multiple times (large result) ss case when you might consider a temporary table instead. A materialized CTE has no indexes and no statistics. A temporary table can have both, and as covered in [Introduction to Buffers](/posts/introduction-to-buffers/), temporary tables use local buffers with simpler locking and no WAL overhead. If you're joining against 100k+ rows from a CTE, create a temp table, add an index, and `ANALYZE` it. Data-modifying operations</strong> with writable CTE. No alternative gives you the atomic, single-statement behavior.</p> Recursive CTEs</strong>. There's no alternative in pure SQL.</p> Large intermediate results needing indexes/statistics</strong>. If you really need them use temporary table. As discussed in Reading Buffer statistics</a>, temporary tables offer full planner support - indexes, statistics, and buffer management - that materialized CTEs simply don't have.</p> Scenario</th> Recommendation</th></tr></thead> Readability, single reference</td> CTE (inlined, free)</td></tr> Compute once, use many times (small)</td> CTE (materialized)</td></tr> Compute once, use many times (large)</td> Temporary table</td></tr> Atomic data modification</td> Writable CTE</td></tr> Hierarchy / graph traversal</td> Recursive CTE</td></tr> Need indexes on intermediate data</td> Temporary table</td></tr> </tbody></table> The PG 18 state of affairs</a> </h2> PostgreSQL 18 continues to refine CTE handling without any revolutionary changes:</p> EXPLAIN shows memory/disk usage</strong> for CTE materialization nodes. You can finally see whether your CTE fit in work_mem</code> or spilled to disk.</li> Better query plans for CTEs on the same table.</strong> The planner is smarter about eliminating redundant scans when multiple CTEs reference the same underlying table.</li> Data-modifying CTE fix for updatable views with rules.</strong> An edge case where writable CTEs interacted incorrectly with views defined using rules has been resolved.</li> CTE inlining is mature.</strong> The core inlining logic hasn't changed since PG 12. What has improved is everything around it - better statistics propagation (PG 17), better materialisation diagnostics (PG 18), better cost estimation.</li> </ul> CTEs are a good tool. Just know when you're holding the sharp end.</p> pg_regresql: truly portable PostgreSQL statistics 2026-03-21T14:32:00+00:00 The previous article</a> showed that PostgreSQL 18 makes optimizer statistics portable, but left one gap open:</p> It's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportionally.</p> </blockquote> The planner doesn't trust pg_class.relpages</code>. It calls smgrnblocks()</code> to read the actual number of 8KB pages from disk. Your table is 74 pages on disk but pg_class.relpages</code> says 123,513? The planner uses the ratio to scale reltuples</code> down to match the actual file size. The selectivity ratios stay correct, plan shapes mostly survive, but the absolute cost estimates are off.</p> For debugging a single query, that's usually fine. For automated regression testing where you compare EXPLAIN</code> costs across runs, it breaks things. A cost threshold of 2× means something different when the baseline was computed from fake-scaled numbers.</p> As part of my work on RegreSQL</a> I'm happy to announce pg_regresql</a> extension which fixes this by hooking directly into the planner.</p> Why the planner ignores relpages</a> </h2> When PostgreSQL's planner calls get_relation_info()</code> in plancat.c</code>, it delegates to estimate_rel_size()</code> which ends up in table_block_relation_estimate_size()</code> in tableam.c</code>. There, the actual page count comes from the storage manager:</p> curpages </span>=</span> RelationGetNumberOfBlocks</span>(rel);</span></span></code></pre> The function then computes a tuple density from pg_class</code> (reltuples / relpages</code>) and multiplies it by curpages</code> to estimate tuples. So pg_class.reltuples</code> isn't ignored, it's scaled to match the real file size. The reasoning is sound for normal operation: the catalog might be stale, but the file system is always current.</p> The same applies to indexes. The planner reads their actual sizes from disk too.</p> What pg_regresql does</a> </h2> The extension hooks into get_relation_info_hook</code>, a planner callback that runs after PostgreSQL reads the physical file stats. The hook replaces the file-based numbers with the values stored in pg_class</code>:</p> rel->pages</code> ← pg_class.relpages</code></li> rel->tuples</code> ← pg_class.reltuples</code></li> rel->allvisfrac</code> ← pg_class.relallvisible / pg_class.relpages</code></li> </ul> It does the same for every index in rel->indexlist</code>. Pages and tuples for each index are overridden from the index's own pg_class</code> entry.</p> The guard conditions are simple: skip the override if relpages == 0</code> (empty or never analyzed) or reltuples == -1</code> (never analyzed). The hook only activates for tables that have been ANALYZE</code>d or had statistics injected.</p> Installation</a> </h2> Build from source using PGXS:</p> cd</span> pg_ext</span></span> make</span></span> make</span> install</span></span></code></pre> It requires no GUCs, background workers, or shared memory.</p> Usage</a> </h2> Load the extension in your session:</p> LOAD</span> 'pg_regresql'</span>;</span></span></code></pre> That's it. Every EXPLAIN</code> in this session will now use catalog statistics instead of file sizes. There are no functions to call, no tables to configure.</p> You can also load it per-database by adding it to session_preload_libraries</code> in postgresql.conf</code> or via ALTER DATABASE</code>:</p> ALTER DATABASE</span> test_db </span>SET</span> session_preload_libraries </span>=</span> 'pg_regresql'</span>;</span></span></code></pre>The difference it makes</a> </h2> Using the same test_orders</code> example from the previous article</a>: 10,000 actual rows, injected with production statistics claiming 50 million rows across 123,513 pages.</p> Without pg_regresql</code>:</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span></code></pre> The plan shape is correct (index scan thanks to the histogram), but the row estimate is 6,340. For a 50-million-row table where the filter covers roughly 10% of the histogram range, the expected estimate should be in the millions. The planner saw 74 real pages on disk, scaled reltuples</code> down to ~30,000, then applied selectivity. The ratio is preserved but the absolute number is wrong.</p> With pg_regresql</code>:</p> LOAD</span> 'pg_regresql'</span>;</span></span> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153212.27 rows=10791836 width=27)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span> (2 rows)</span></span></code></pre> Cost numbers now reflect the full 50 million rows.</p> Where this matters</a> </h2> Cost-based regression testing.</strong> If you're comparing EXPLAIN</code> costs between schema versions (which is what RegreSQL</a> does), you need the absolute numbers to be stable and realistic. With the scaling behavior, your baseline costs are proportional to your test database size, not production. A migration that doubles a cost in production might show a 1.3× increase in CI because the scaled-down numbers compress the range.</p> Reproducing production plans on a laptop.</strong> Sometimes the plan shape itself changes depending on the absolute numbers. A query with multiple joins might get a different join order when the planner sees 50 million rows vs. 30,000 rows, because the cost crossover between hash join and nested loop depends on the absolute row count, not just the ratio.</p> Index-only scans.</strong> The allvisfrac</code> (fraction of all-visible pages) matters for index-only scan costing. Without the hook, allvisfrac</code> is computed from the real relallvisible</code> catalog value divided by the real page count. With injected stats, relallvisible</code> might be 120,000 but the real page count is 74, so the fraction clamps to 1.0 and the planner overestimates how cheap index-only scans are. The hook fixes this by using the injected relpages</code> as the denominator.</p> What it doesn't do</a> </h2> Column-level statistics and ANALYZE</code> behavior are unchanged. The extension only affects how the planner reads table and index sizes. One thing worth noting: EXPLAIN ANALYZE</code> will still show actual row counts from the real (small) data. The extension changes the planner's cost estimates, not query execution.</p> The full workflow</a> </h2> Combining PostgreSQL 18's portable statistics with pg_regresql</code>, the full workflow looks like this:</p> # 1. dump schema and statistics from production</span></span> pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span> pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span> </span> # 2. create test database</span></span> createdb</span> test_db</span></span> psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span> </span> # 3. load minimal fixture data (optional)</span></span> psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span> </span> # 4. inject production statistics</span></span> psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span> </span> # 5. install pg_regresql and prevent stats from being overwritten</span></span> psql</span> -d</span> test_db</span> <<</span>SQL</span></span> ALTER DATABASE test_db SET session_preload_libraries = 'pg_regresql';</span></span> ALTER TABLE orders SET (autovacuum_enabled = false);</span></span> -- repeat for other tables</span></span> SQL</span></span> </span> # 6. reconnect and verify (plans now match production)</span></span> psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM orders WHERE status = 'pending'"</span></span></code></pre>Compatibility</a> </h2> The extension works with PostgreSQL 13 through 18. The portable statistics functions (pg_restore_relation_stats</code>, pg_restore_attribute_stats</code>) require PostgreSQL 18, but pg_regresql</code> works with any method of writing to pg_class</code>, including direct catalog updates on older versions.</p> PostgreSQL 19 will need a small update: the get_relation_info_hook</code> used by pg_regresql has been replaced</a> with build_simple_rel_hook</code>. The new hook runs slightly later with different arguments, but the override logic stays the same.</p> Don't use this in production.</strong> The extension makes the planner ignore reality. That's exactly what you want for testing with injected statistics. In production, you want the planner to see the actual file sizes so it can adapt to data growth, bloat, and vacuum activity. Keep pg_regresql</code> in your dev/test and CI databases. </div> Production query plans without production data 2026-03-08T22:05:00+00:00 In the previous article</a> we covered how the PostgreSQL planner reads pg_class</code> and pg_statistic</code> to estimate row counts, choose join strategies, and decide whether an index scan is worth it. The message was clear: when statistics are wrong, everything else goes with it.</p> Streaming replication provides bit-to-bit replication, so all replicas share the same statistics with primary server.</div> But there was one thing we didn't talk about. Statistics are specific to the database cluster that generated them. The primary way to populate them is `ANALYZE` which requires the actual data. PostgreSQL 18 changed that. Two new functions: pg_restore_relation_stats</code> and pg_restore_attribute_stats</code> write numbers directly into the catalog tables. Combined with pg_dump --statistics-only</code>, you can treat optimizer statistics as a deployable artifact. Compact, portable, plain SQL.</p> The feature was driven by the upgrade use case</a>. In the past, major version upgrades used to leave pg_statistic</code> empty, forcing you to run ANALYZE</code>. Which might take hours on large clusters. With PostgreSQL 18 upgrades now transfer statistics automatically. But that's just the beginning. The same logic lets you export statistics from production and inject them anywhere - test database, local debugging, or as part of CI pipelines.</p> The problem</a> </h2> Your CI database has 1,000 rows. Production has 50 million. The planner makes completely different decisions for each. Running EXPLAIN</code> in CI tells you nothing about the production plan. This is the core premise behind RegreSQL</a>. Catching query plan regressions in CI is far more reliable when the planner sees production-scale statistics.</p> Same applies to debugging</strong>. A query is slow in production and you want to reproduce the plan locally, but your database has different statistics, and planner chooses the predictable path. Porting production stats can provide you that snapshot of thinking planner has to do in production, without actually going to production.</p> pg_restore_relation_stats</a> </h2> The first of function behind portable PostgreSQL statistics is pg_restore_relation_stats</code>. It writes table-level data directly into pg_class</code> in form of variadic name/value pairs.</p> SELECT</span> pg_restore_relation_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'orders'</span>,</span></span> 'relpages'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'reltuples'</span>, </span>50000000</span>::</span>real</span>,</span></span> 'relallvisible'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'relallfrozen'</span>, </span>120000</span>::</span>integer</span></span> );</span></span></code></pre> But that's just an example. Let's modify some real statistics to see the full value. We will create a small table, inject fake production-like statistics and watch the planner to change its mind.</p> CREATE TABLE</span> test_orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL</span>,</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> test_orders (customer_id, amount, </span>status</span>, created_at)</span></span> SELECT</span></span> (random()</span> </span> 9999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 5000</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> '2024-01-01'</span>::</span>date +</span> (random()</span> </span> 365</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10000</span>);</span></span> </span> CREATE INDEX</span> ON</span> test_orders (created_at);</span></span> CREATE INDEX</span> ON</span> test_orders (</span>status</span>);</span></span> ANALYZE test_orders;</span></span></code></pre> When you check the current statistics, it has predictable data.</p> SELECT</span> relname, relpages, reltuples</span></span> FROM</span> pg_class </span>WHERE</span> relname </span>=</span> 'test_orders'</span>;</span></span></code></pre> relname \| relpages \| reltuples</span></span> -------------+----------+-----------</span></span> test_orders \| 74 \| 10000</span></span> (1 row)</span></span></code></pre> With 10,000 rows across 74 pages, the planner picks a sequential scan.</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..199.00 rows=5891 width=26)</span></span> Filter: (created_at > '2024-06-01'::date)</span></span> (2 rows)</span></span></code></pre> Now inject production-scale table stats:</p> SELECT</span> pg_restore_relation_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'relpages'</span>, </span>123513</span>::</span>integer</span>,</span></span> 'reltuples'</span>, </span>50000000</span>::</span>real</span>,</span></span> 'relallvisible'</span>, </span>123513</span>::</span>integer</span></span> );</span></span></code></pre> And you might be surprised by the result.</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..448.45 rows=17649 width=26)</span></span> Filter: (created_at > '2024-06-01'::date)</span></span></code></pre> The planner is still using the sequential plan. Only the estimated number of rows has changed. Why? If you remember from previous article, it's where column level statistics come into play. Histogram bounds still match the original 10,000 rows we inserted.</p> pg_restore_attribute_stats</a> </h2> This function writes column-level statistics into pg_statistic</code> the same catalog that ANALYZE populates</a> with MCVs, histograms, and correlation</a>.</p> In previous section, we left the planner stuck on a sequential scan despite believing the table has 50 million rows. The missing piece is column-level statistics. Let's pick up where we left off and inject histogram bounds for created_at</code>.</p> SELECT</span> pg_restore_attribute_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'attname'</span>, </span>'created_at'</span>,</span></span> 'inherited'</span>, false::</span>boolean</span>,</span></span> 'null_frac'</span>, </span>0</span>.</span>0</span>::</span>real</span>,</span></span> 'avg_width'</span>, </span>4</span>::</span>integer</span>,</span></span> 'n_distinct'</span>, </span>-</span>0</span>.</span>05</span>::</span>real</span>,</span></span> 'histogram_bounds'</span>, </span>'{2019-01-01,2019-07-01,2020-01-01,2020-07-01,2021-01-01,2021-07-01,2022-01-01,2022-07-01,2023-01-01,2023-07-01,2024-01-01}'</span>::</span>text</span>,</span></span> 'correlation'</span>, </span>0</span>.</span>98</span>::</span>real</span></span> );</span></span></code></pre> Now the planner knows the data spans 5 years. A query filtering on the last 6 months of 2024 covers a narrow slice.</p> EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------------</span></span> Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26)</span></span> Index Cond: (created_at > '2024-06-01'::date)</span></span></code></pre> Histogram bounds divide the non-MCV portion of the data into equal-population buckets. If most_common_vals</code> accounts for most of the data, the histogram covers only the remaining tail. The number of buckets is controlled by default_statistics_target</code> (default 100, meaning 101 bounds). </div> And that's a plan flip! The histogram tells the planner the data spans 2019–2024, so > '2024-06-01'</code> matches a narrow tail. A small fraction of 50 million rows. The index scan that was ignored before is now the obvious choice. Table-level stats set the scale, column-level stats shaped the selectivity, and together they changed the plan.</p> The correlation</code> statistic tells the planner how closely the physical row order matches the column's sort order. A value near 1.0 means sequential access patterns - making index scan cheaper</a> because the next row is likely on the same or adjacent page. For time-series data like created_at</code> where rows are inserted chronologically, correlation is typically very high. </div> Injecting a skewed distribution</a> </h2> The same function handles MCV lists</a>. In production, your status</code> column isn't uniform, 95% of orders are delivered, 1.5% are pending.</p> SELECT</span> pg_restore_attribute_stats(</span></span> 'schemaname'</span>, </span>'public'</span>,</span></span> 'relname'</span>, </span>'test_orders'</span>,</span></span> 'attname'</span>, </span>'status'</span>,</span></span> 'inherited'</span>, false::</span>boolean</span>,</span></span> 'null_frac'</span>, </span>0</span>.</span>0</span>::</span>real</span>,</span></span> 'avg_width'</span>, </span>9</span>::</span>integer</span>,</span></span> 'n_distinct'</span>, </span>5</span>::</span>real</span>,</span></span> 'most_common_vals'</span>, </span>'{delivered,shipped,cancelled,pending,returned}'</span>::</span>text</span>,</span></span> 'most_common_freqs'</span>, </span>'{0.95,0.015,0.015,0.015,0.005}'</span>::</span>real</span>[]</span></span> );</span></span></code></pre> You can see</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------------------</span></span> Bitmap Heap Scan on test_orders (cost=8.93..90.42 rows=599 width=27)</span></span> Recheck Cond: (status = 'pending'::text)</span></span> -> Bitmap Index Scan on test_orders_status_idx (cost=0.00..8.78 rows=599 width=0)</span></span> Index Cond: (status = 'pending'::text)</span></span> (4 rows)</span></span></code></pre> and compare it with</p> EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE status =</span> 'delivered'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------</span></span> Seq Scan on test_orders (cost=0.00..448.45 rows=28458 width=27)</span></span> Filter: (status = 'delivered'::text)</span></span> (2 rows)</span></span></code></pre> Same column, same operator, different plans. The planner uses a bitmap index scan for pending</code> (1.5% rare enough to justify the index) and a sequential scan for delivered</code> (95% being most of the table). The selectivity ratios from the MCV list drive the plan choice.</p> You might have noticed the row estimates (599 and 28,458) are lower than you'd expect for a 50-million-row table. The planner checks the actual physical file size. Our table is only 74 pages on disk, not the 123,513 we injected. Hence the planner scales reltuples</code> and relpages</code> down proportionally. The absolute numbers shrink, but the ratios</i> between them stay correct, and it's the ratios that determine plan shape. When you use pg_dump --statistics-only</code> in practice, you're typically restoring into a database with comparable data volume, so the estimates align naturally. </div> </div> </div> </div> </div> pg_regresql extension</strong> The pg_regresql</code> extension fixes this scaling problem. It hooks into the planner to trust the injected relpages</code> value instead of reading the physical file size, so cost estimates match production even when your test database is tiny.</p> Read more</a> </div> </div> pg_dump</a> </h2> The functions we covered are the mechanics. For operational use pg_dump</code> provides everything you need. PostgreSQL 18 added three flags.</p> Flag</th> Effect</th></tr></thead> --statistics</code></td> dump the statistics (you have to request it explicitely)</td></tr> --statistics-only</code></td> dump only the statistics, not schema or data</td></tr> --no-statistics</code></td> do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database</p> pg_dump --statistics-only -d production_db > stats.sql</span></span> </span></code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above.</p> The full workflow to turn your production data into testable plans might look like this:</p> # 1. dump schema from production</span></span> pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span> </span> # 2. dump statistics from production</span></span> pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span> </span> # 3. create test database with schema</span></span> createdb</span> test_db</span></span> psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span> </span> # 4. load fixture data (optional; masked, minimal)</span></span> psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span> </span> # 5. inject production statistics</span></span> psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span> </span> # 6. query plans now match production</span></span> psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</span></span></code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started.</p> To prevent this, disable autovacuum analyze on the tables you've injected.</p> -- disable autovacuum</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_enabled </span>=</span> false);</span></span> </span> -- or set analyze threshold so high it nevers kicks-in</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_analyze_threshold </span>=</span> 2147483647</span>);</span></span></code></pre> Be careful here.</strong> If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. </p> For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run.</p> And please, never ever do this in production!</p> </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups).</p> It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>.</p> Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>.</p> The easiest way to grant it for automation:</p> GRANT</span> pg_maintain </span>TO</span> ci_service_account;</span></span></code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser.</p> PostgreSQL Statistics: Why queries run slow 2026-02-26T23:35:00+00:00 Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work? PostgreSQL doesn't run the query to find out - it estimates the cost. It reads pre-computed data from pg_class</code> and pg_statistic</code> and does the maths to figure out the cheapest path to your data.</p> In ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they are stale, the situation gets out of control. Planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed as optimal plan turns into a cascading failure.</p> How do statistics get stale? It can be either bulk load, a schema migration, faster-than-expected growth, or simply VACUUM</code> not keeping up. Whatever the cause, the result is the same. The planner is flying blind. Choosing paths based on reality that no longer exists.</p> In this post we will go inside the two catalogs the planner depends on, understand what ANALYZE</code> actually gets for you from a 30,000-row table, and see how those numbers determine whether your query takes milliseconds or minutes.</p> Sample schema</a> </h2> For demonstration purposes we will use the same schema as in the article Reading Buffer statistics in EXPLAIN output</a>.</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> note </span>text</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span> SELECT</span></span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span> '2022-01-01'</span>::</span>date +</span> (random()</span> </span> 1095</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> ANALYZE customers;</span></span> ANALYZE orders;</span></span></code></pre>What the Planner Reads</a> </h2> As mentioned above, every decision the planner makes is based on two sources:</p> table-level metadata from pg_class</code></li> column-level metadata from pg_statistic</code>.</li> </ul> pg_class - relational-level stats</a> </h3> It actually tracks all relations</i>. Not just tables and indexes, but also partitions, TOAST tables, sequences, composite types, and views. </div>Every table, index and materialized view has a row in `pg_class`. Before it even looks at column-level statistics, the planner reads four values from it: Column</th> Meaning</th></tr></thead> relpages</code></td> Number of 8KB pages representing the table on the disk</td></tr> reltuples</code></td> Estimated number of live rows in the table</td></tr> relallvisible</code></td> Pages where all tuples are visible to all transactions</td></tr> </tbody></table> For our sample table it looks like this.</p> SELECT</span> relname, relpages, reltuples, relallvisible</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>=</span> 'orders'</span>;</span></span></code></pre> relname \| relpages \| reltuples \| relallvisible</span></span> ---------+----------+-----------+---------------</span></span> orders \| 856 \| 100000 \| 856</span></span></code></pre>Compared to reading page from disk, handling single tuple represents tiny overhead. Per-row overhead is configured using cpu_tuple_cost</code>, which defaults to 0.01</code></div> The planner sees 100,000 rows spread across 856 pages. Every cost estimate starts from those two numbers. relpages</code> drives sequential scan cost - each page is one unit of I/O work as configured via seq_page_cost</code>. reltuples</code> controls estimates for joins, aggregations, and pretty much everything else.</p> The reltuples</code> value is only an estimate, not a live count. It's updated by ANALYZE</code> (and autovacuum), not by individual INSERTs or DELETEs. Between ANALYZE runs, PostgreSQL scales reltuples</code> proportionally when relpages</code> value changes - if the table grows by 10% in pages, the planner assumes 10% more rows too.</p> This works well enough for normal growth, but breaks down with bloat. If dead tuples are inflating the number of pages used, without adding real rows, the planner overestimates the table size.</p> </div> The other column mentioned above matters for specific operations. relallvisible</code> tells the planner how much of the table can be read with an index-only scan. Meaning an index-only scan can return results using the index without checking the heap for visibility.</p> pg_statistic (via pg_stats) - column-level stats</a> </h3> Knowing the size of a table is only half the picture. To estimate how many rows might match, the planner needs to understand the data inside each column. PostgreSQL maintains statistics in pg_statistic</code> catalog. Which you're most likely never going to use directly. In practice you will use the view pg_stats</code> which presents the human-friendly data behind it.</p> The most interesting values it exposes are:</p> Statistic</th> What it tells the planner</th></tr></thead> null_frac</code></td> Fraction of entries that are NULL values</td></tr> avg_width</code></td> Average width in bytes</td></tr> n_distinct</code></td> Number of distinct values (negative means fraction of rows)</td></tr> most_common_vals</code></td> Most frequent values</td></tr> most_common_freqs</code></td> Frequencies of those values</td></tr> histogram_bounds</code></td> Values dividing the remaining data into equal-population buckets (most_common_vals are excluded)</td></tr> correlation</code></td> Statistical correlation between physical row ordering and logical ordering of the column values</td></tr> </tbody></table> SELECT</span> attname, null_frac, avg_width, n_distinct,</span></span> most_common_vals, most_common_freqs, histogram_bounds, correlation</span></span> FROM</span> pg_stats</span></span> WHERE</span> tablename </span>=</span> 'orders'</span> AND</span> attname </span>=</span> 'status'</span>;</span></span></code></pre>-[ RECORD 1 ]-----+--------------------------------------</span></span> attname \| status</span></span> null_frac \| 0</span></span> avg_width \| 8</span></span> n_distinct \| 4</span></span> most_common_vals \| {pending,shipped,delivered,cancelled}</span></span> most_common_freqs \| {0.25396666,0.25,0.24973333,0.2463}</span></span> histogram_bounds \|</span></span> correlation \| 0.2524199</span></span></code></pre> From this planner knows there are exactly 4 distinct values, more or less equally distributed, without any NULL values. When you write a predicate WHERE status = 'pending'</code> it will estimate ~25% of rows are going to match. All that by not running a query, but reading the catalog row.</p> Different situation is when checking column note</code>.</p> SELECT</span> attname, null_frac, avg_width, n_distinct,</span></span> array_length(most_common_vals, </span>1</span>) </span>AS</span> mcv_count,</span></span> array_length(histogram_bounds, </span>1</span>) </span>AS</span> histogram_buckets</span></span> FROM</span> pg_stats</span></span> WHERE</span> tablename </span>=</span> 'orders'</span> AND</span> attname </span>=</span> 'note'</span>;</span></span></code></pre>-[ RECORD 1 ]-----+-------</span></span> attname \| note</span></span> null_frac \| 0.6982</span></span> avg_width \| 32</span></span> n_distinct \| 1</span></span> mcv_count \| 1</span></span> histogram_buckets \|</span></span></code></pre> For this column there's ~70% of NULLs, and only one distinct value. Therefore WHERE note IS NOT NULL</code> will estimate 30% of rows.</p> Selectivity in Action</a> </h2> Now that we have covered what data the planner has available, let's have a look at how it's used to estimate how many rows a certain part of a query will read. This "guess" is called Selectivity</strong>. And it's defined as a floating-point number between 0 and 1.</p> The formula is pretty simple:</p> Estimated Rows = Total Rows * Selectivity</span></span></code></pre> But the way Selectivity is calculated depends entirely on the operator you use (=</code>, <</code>, ></code> or LIKE</code>).</p> The equality (Most Common Values)</a> </h3> Equality is easiest to start with. When you use WHERE status = 'shipped'</code> the planner first checks most_common_vals</code> (MCV) list. If there's a match, the selectivity is same as the one provided by most_common_freqs</code>.</p> If the value isn't in the list, the planner assumes the value is part of the "remaining" population. It subtracts all MCV frequencies from 1.0 and divides the remainder by the number of other distinct values.</p> (1.0 - (SELECT sum(s) FROM unnest(most_common_freqs) s))</span></span> /</span></span> (n_distinct - array_length(most_common_vals, 1))</span></span></code></pre>The Range lookup (The Histogram)</a> </h3> Life would be easy if we would be looking only for exact values. Most of the time we need to utilise range lookup. For example WHERE amount > 400</code>.</p> MCVs are useless in this case as there might be thousands or millions of unique constants. This is where histogram_bounds</code> comes in. PostgreSQL divides the column values into a number of buckets, where each bucket contains equal number of rows (not values).</p> The Selectivity in this case is determined by how many buckets your query covers. In our example, if we have bounds (100, 200, 300, 400, 500, 600)</code> the planner will establish it covers 2 full buckets. Since there are 5 buckets in total the Selectivity is going to be 0.4 (2/5).</p> If you are now wondering where 2 and 5 come from? The histogram_bounds</code> array defines boundaries between buckets. With bounds (100, 200, 300, 400, 500, 600)</code> there are 5 buckets in total.</p> (100-200), (200-300), (300-400), (400-500), (500-600)</span></span></code></pre> And the same logic applies to matching. The condition amount > 400</code> matches on (400-500)</code> and (500-600)</code> buckets (2 in total).</p> The histogram's biggest weakness is linear interpolation. If your data has massive spikes in distribution the planner will always assume a perfect distribution.</div> Slightly more complex situation happens in the cases where your value falls inside a bucket. If your query has WHERE amount > 350</code> the planner locates the bucket containing value 350 (in our example (300-400)</code>), assumes the data is linearly distributed and calculates the ratio within that bucket.</p> Search and pattern matching</a> </h3> This is probably the most treacherous territory for the planner. For substring matching patterns like WHERE note LIKE '%middle%'</code>, there's no histogram or list of values to rely on. The planner must fall back to "magic constants" hardcoded in the PostgreSQL source code.</p> The default for generic patterns is 0.5% of the total rows, defined as</p> #define DEFAULT_MATCH_SEL 0.005</span></span></code></pre> Slightly better situation comes for prefixed matches like WHERE note LIKE 'boringSQL%'</code> where PostgreSQL can fall back to range conditions and use histogram bounds. While this is a subtle difference, it makes a night and day difference.</p> Correlation and index scan cost</a> </h3> Remember correlation</code> from the pg_stats</code>? It says how closely the physical order of rows on disk matches the logical order of column values. Values close to 1.0 mean high correlation, values near 0 mean data is laid out randomly across pages.</p> If you recall how data is organized in 8KB page</a> the row locality matters. This matters because it determines whether an index scan is worth it. The planner assumes a random page read costs 4× more than a sequential one (random_page_cost = 4.0</code> vs seq_page_cost = 1.0</code>). When correlation is high, the rows an index points to are physically adjacent. The planner expects sequential I/O and costs the scan cheaply. When correlation is low, each lookup likely hits a different page, and the planner costs each of those reads at the higher random rate. That difference alone can be enough to make a sequential scan cheaper than an index scan.</p> n_distinct and join estimation</a> </h3> Value n_distinct</code> plays an important role in joins. This further stresses the importance of running ANALYZE</code> after bulk data changes.</p> Let's imagine this simplified logic for the equality join:</p> In reality, Postgres also accounts for null_frac</code> (NULLs don't join) and MCVs on both sides. If both sides have MCVs, it calculates the "inner product" of the frequencies.</div> estimated_rows = (rows_left × rows_right) / max(n_distinct_left, n_distinct_right)</span></span></code></pre> Let's say you're trying to join two tables, both having roughly 2,000 distinct values for the join key. Outdated n_distinct</code> will cause significant estimation drifts and the planner may pick a wrong join strategy altogether.</p> This is the heap we are talking about</a> </h3> Please, keep in mind this describes how the planner estimates rows for a basic heap scan. Indexes, join selectivity, and complex types like JSONB each come with their own estimation logic, operator handling and quirks. The fundamentals of estimation are the same nevertheless.</p> What if there are no statistics?</a> </h2> So far we have touched on why up-to-date statistics are a must. But what if there are no statistics at all? For example for a new table or new column when ANALYZE</code> has not yet run.</p> In those cases PostgreSQL falls back to hardcoded defaults.</p> Condition type</th> Default selectivity</th> Constant</th></tr></thead> Equality (=</code>)</td> 0.5%</td> DEFAULT_EQ_SEL = 0.005</code></td></tr> Range (></code>, <</code>)</td> 33.3%</td> DEFAULT_INEQ_SEL = 0.3333</code></td></tr> Range (BETWEEN</code>)</td> 0.5%</td> DEFAULT_RANGE_INEQ_SEL = 0.005</code></td></tr> Pattern matching (LIKE</code>)</td> 0.5%</td> DEFAULT_MATCH_SEL = 0.005</code></td></tr> IS NULL</code></td> 0.5%</td> DEFAULT_UNK_SEL = 0.005</code></td></tr> IS NOT NULL</code></td> 99.5%</td> DEFAULT_NOT_UNK_SEL = 0.995</code></td></tr> </tbody></table> Nothing that a quick ANALYZE</code> can't fix, correct? Or maybe not.</p> Where No Statistics Go</strong> While this article focuses on statistics and getting them right, there are situations where no statistics will be available (never or not predictably). If you believe it won't affect you, please, think twice. CTEs and subqueries</strong> when not inlined/materialized have no statistics. See Good CTE, Bad CTE</a> for when exactly this happens.</li> Temporary tables</strong> are not touched by autovacuum, so no automatic ANALYZE</code>.</li> Foreign tables</strong> do not guarantee stats are propagated.</li> And, to a big surprise, computed expressions in WHERE</strong> like WHERE amount * 1.1 > 500</code> or lower(email) = 'hello@example.com'</code>; unless you create an expression index or extended statistics.</li> </ul> And btw, do you know about TRUNCATE</code> - it's fast way to get rid of the data, but it leaves statistics behind...</p> </div> How ANALYZE</code> Works</a> </h2> As we have already mentioned several times, ANALYZE</code> is the only mechanism that populates pg_class</code> and pg_statistic</code> with fresh data. Understanding what it samples, what it computes, and what it misses, is key to understanding why statistics are sometimes wrong.</p> The process itself consists of 6 separate steps (as reported by pg_stat_progress_analyze</code>):</p> initializing</li> acquiring sample rows</li> acquiring inherited sample rows (child/partitioned tables)</li> computing statistics (MCVs, histograms, correlation, etc.)</li> computing extended statistics (see below)</li> finalizing and writing to pg_statistic</code></li> </ol> For our purposes we will only cover sampling, computing statistics and writing to pg_statistic</code>.</p> The sampling mechanism</a> </h3> ANALYZE actually doesn't read the entire table. It samples what is considered to be a statistically justified minimum sample size (defined as 300 for the reservoir sampling</a> algorithm). For PostgreSQL that means 300 × default_statistics_target</code> rows.</p> SHOW default_statistics_target;</span></span></code></pre> default_statistics_target</span></span> ---------------------------</span></span> 100</span></span> (1 row)</span></span></code></pre> With the default target of 100, that's 30,000 rows. For our 100,000-row orders table, ANALYZE</code> reads about 30% of the data. For a 50-million-row table, it reads 0.06%. The same target also controls the size of the MCV list and histogram. Up to 100 entries each.</p> With a target of 100, don't be surprised that histogram_bounds</code> contains 101 values. 100 buckets need 101 boundaries to close them.</div> The sampling is two-stage. First, ANALYZE</code> selects a random set of pages. Then it reads all live rows from those pages. This gives a representative cross-section without reading every page.</p> Computing statistics</a> </h3> Once ANALYZE has its 30,000 sample rows (considering the default values), it processes each column independently. The pipeline for a single column looks roughly like this:</p> First, it counts NULLs and calculates null_frac</code> and avg_width</code> - the cheapest statistics to compute. Then it sorts the non-null values and builds the MCV list by counting duplicates. Values that appear frequently enough make the cut; the rest are passed to the histogram builder, which divides them into equal-population buckets. Finally, because the values are already sorted, ANALYZE compares the logical sort order against the physical tuple positions (which page each row came from) to compute correlation</code>.</p> The key detail here is that MCVs and histograms are not built from the same pool. Values that land in most_common_vals</code> are excluded from histogram_bounds</code>. This is why you'll sometimes see a column with MCVs but no histogram, or a histogram but no MCVs. They represent different slices of the same data.</p> Writing to pg_statistic</a> </h3> Once all columns are processed, ANALYZE writes the results into pg_statistic</code> - one row per column. If a row for that column already exists, it's updated in place. This is a regular heap update, which means the old row becomes a dead tuple. On tables with many columns or frequent ANALYZE runs, this can cause pg_statistic</code> itself to bloat.</p> After pg_statistic</code> is updated, ANALYZE refreshes relpages</code>, reltuples</code>, and relallvisible</code> in pg_class</code>. These values are recalculated from the sampling, not from a full table scan (they are estimates too).</p> Controlling statistics quality</a> </h2> default_statistics_target</a> </h3> The default target of 100 works well for most columns. It means up to 100 MCVs, 101 histogram bounds, and a 30,000-row sample. Increasing it helps when:</p> A column has many distinct values and the top 100 don't cover enough of the distribution</li> Range queries on skewed data produce bad estimates because histogram buckets are too coarse</li> Join estimates are off because n_distinct</code> is inaccurate</li> </ul> The cost scales linearly. Setting it to 1000 means 300,000 sampled rows, up to 1000 MCVs, more catalog storage, and slower planning from larger arrays to search. The maximum is 10,000.</p> You don't have to raise it globally. For a single problematic column:</p> ALTER TABLE</span> orders </span>ALTER</span> COLUMN </span>status SET STATISTICS</span> 500</span>;</span></span> ANALYZE orders;</span></span></code></pre> Now status</code> gets up to 500 MCVs and 501 histogram bounds, while every other column stays at 100.</p> Extended statistics</a> </h3> Standard statistics treat each column independently. This means the planner can't know that city = 'Edinburgh'</code> and country = 'UK'</code> are correlated. It multiplies their selectivities independently, potentially underestimating by orders of magnitude.</p> Extended statistics solve this for specific column combinations:</p> CREATE STATISTICS</span> orders_status_date (dependencies, ndistinct, mcv)</span></span> ON status</span>, created_at </span>FROM</span> orders;</span></span> ANALYZE orders;</span></span></code></pre> This tells ANALYZE to compute functional dependencies, combined distinct counts, and combined MCVs between the columns. The planner can then use these to avoid the independence assumption for queries filtering on both columns.</p> The three types of extended statistics serve different purposes:</p> dependencies</strong> capture functional dependencies between columns. Helps when knowing one column's value determines or narrows another's (e.g. zip_code</code> largely determines city</code>).</li> ndistinct</strong> tracks the number of distinct value combinations across columns. Helps with GROUP BY</code> on multiple columns where the planner would otherwise multiply distinct counts independently.</li> mcv</strong> builds a combined most-common-values list for column tuples. The most powerful but most expensive option. Helps with multi-column WHERE conditions on correlated values.</li> </ul> You can create extended statistics with any combination of these types. Start with dependencies</code> as it's cheapest, add mcv</code> when multi-column filter estimates are consistently wrong.</p> Extended statistics are worth creating when you see EXPLAIN</code> estimates that are consistently wrong on multi-column filters, and the columns are logically correlated. They are computed during step 5 of the ANALYZE process and stored in pg_statistic_ext_data</code>.</p> Diagnosing bad estimates</a> </h2> When a query is slow, the first question should always be: did the planner estimate correctly? Compare the estimate to reality with EXPLAIN ANALYZE</code>.</p> Estimate off by handful of rows means statistics are fine. But when you see estimates off by 10x or more, that's where planning goes wrong. A nested loop that looks cheap for 100 rows becomes a disaster at 10,000.</p> The statistics tell you what the planner believed, and comparing that with reality tells you what to do next. Either run ANALYZE</code>, consider tuning the statistics target for a specific column, or creating extended statistics if multiple columns are involved.</p> The planner is only as good as what it reads from the catalog. When estimates go wrong, don't blame the planner. Check the data it's working with.</p> Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment?</p> When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden.</p> You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run</p> CREATE DATABASE dbname;</span></span></code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use</p> CREATE DATABASE dbname TEMPLATE template1;</span></span></code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>.</p> In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine.</p> CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic.</p> Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time.</p> The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options.</p> FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files.</p> When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>.</p> With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space.</p> Quick setup checklist:</strong></p> Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database.</p> CREATE DATABASE</span> source_db</span>;</span></span> \c source_db</span></span> </span> CREATE TABLE</span> boring_data</span> (</span></span> id </span>serial PRIMARY KEY</span>,</span></span> payload </span>text</span></span> );</span></span> </span> -- generate 50m rows</span></span> INSERT INTO</span> boring_data (payload)</span></span> SELECT</span> md5(random()::</span>text</span>) </span>\|\|</span> md5(random()::</span>text</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>50000000</span>);</span></span> </span> -- force a checkpoint</span></span> CHECKPOINT</span>;</span></span></code></pre> You can verify the database now has roughly 6GB of data.</p> Name \| source_db</span></span> Owner \| postgres</span></span> Encoding \| UTF8</span></span> Locale Provider \| libc</span></span> Collate \| en_US.UTF-8</span></span> Ctype \| en_US.UTF-8</span></span> Locale \|</span></span> ICU Rules \|</span></span> Access privileges \|</span></span> Size \| 6289 MB</span></span> Tablespace \| pg_default</span></span> Description \|</span></span></code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get</p> CREATE DATABASE slow_copy TEMPLATE source_db;</span></span> CREATE DATABASE</span></span> Time: 67000.615 ms (01:07.001)</span></span></code></pre> Now, let's verify our configuration is set for speed:</p> show file_copy_method;</span></span> file_copy_method</span></span> ------------------</span></span> clone</span></span> (1 row)</span></span></code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time.</p> CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY;</span></span> CREATE DATABASE</span></span> Time: 212.053 ms</span></span></code></pre> That's a quite an improvement, isn't it?</p> Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes?</p> When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage.</p> This can create some initial confusion. If you ask PostgreSQL for the size:</p> SELECT</span> pg_database_size(</span>'source_db'</span>) </span>as</span> source,</span></span> pg_database_size(</span>'fast_clone'</span>) </span>as</span> clone;</span></span></code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size.</p> -[ RECORD 1 ]------</span></span> source \| 6594041535</span></span> clone \| 6594041535</span></span></code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page.</p> A single UPDATE will therefore trigger copy-on-write on multiple pages:</p> the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage.</p> XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</span></span></code></pre> All it takes is to update some rows using</p> update</span> boring_data </span>set</span> payload </span>=</span> 'new value'</span> \|\|</span> id </span>where</span> id </span>IN</span> (</span>select</span> id </span>from</span> boring_data </span>limit</span> 20</span>);</span></span></code></pre> and the situation will start to change.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10471550.. 10471589: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10297326.. 10297365: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 8 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span></code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared.</p> Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections.</p> Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy.</p> Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it.</p> VACUUM Is a Lie (About Your Indexes) 2025-12-11T23:34:00+00:00 There is common misconception that troubles most developers using PostgreSQL: tune VACUUM or run VACUUM, and your database will stay healthy. Dead tuples will get cleaned up. Transaction IDs recycled. Space reclaimed. Your database will live happily ever after.</p> But there are couple of dirty "secrets" people are not aware of. First of them being VACUUM is lying to you about your indexes</strong>.</p> The anatomy of storage</a> </h2> When you delete a row in PostgreSQL, it is just marked as a 'dead tuple'. Invisible for new transactions but still physically present. Only when all transactions referencing the row are finished, VACUUM can come along and actually remove them - reclamining the space in the heap (table) space.</p> To understand why this matters differently for tables versus indexes, you need to picture how PostgreSQL actually stores your data.</p> Your table data lives in the heap - a collection of 8 KB pages where rows are stored wherever they fit. There's no inherent order. When you INSERT a row, PostgreSQL finds a page with enough free space and slots the row in. Delete a row, and there's a gap. Insert another, and it might fill that gap - or not - they might fit somewhere else entirely.</p> This is why SELECT * FROM users</code> without an ORDER BY can return rows in order initially, and after some updates in seemingly random order, and that order can change over time. The heap is like Tetris. Rows drop into whatever space is available, leaving gaps when deleted.</p> </p> When VACUUM runs, it removes those dead tuples and compacts the remaining rows within each page. If an entire page becomes empty, PostgreSQL can reclaim it entirely.</p> And while indexes are on surface the same collection of 8KB pages, they are different. A B-tree index must maintain sorted order - that's the whole point of their existence and the reason why WHERE id = 12345</code> is so fast — though even indexes can be ignored</a> under certain conditions. PostgreSQL can binary-search down the tree instead of scanning every possible row. You can learn more about the fundamentals of B-Tree Indexes and what makes them fast</a>.</p> But if the design of the indexes is what makes them fast, it's also their biggest responsibility. While PostgreSQL can fit rows into whatever space is available, it can't move the entries in index pages to fit as much as possible.</p> </p> VACUUM can remove dead index entries. But it doesn't restructure the B-tree. When VACUUM processes the heap, it can compact rows within a page and reclaim empty pages. The heap has no ordering constraint - rows can be anywhere. But B-tree pages? They're locked into a structure. VACUUM can remove dead index entries, yes.</p> Many developers assume VACUUM treats all pages same. No matter whether they are heap or index pages. VACUUM is supposed to remove the dead entries, right?</p> Yes. But here's what it doesn't do - it doesn't restructure the B-tree</strong>.</p> What VACUUM actually does</strong></p> Removes dead tuple pointers from index pages</li> Marks completely empty pages as reusable</li> Updates the free space map</li> </ul> What VACUUM cannot do</strong>:</p> Merge sparse pages together (can do it for empty pages)</li> Reduce tree depth</li> Deallocate empty-but-still-linked pages</li> Change the physical structure of the B-tree</li> </ul> Your heap is Tetris, gaps can get filled. Your B-tree is a sorted bookshelf. VACUUM can pull books out, but can't slide the remaining ones together. You're left walking past empty slots every time you scan.</p> The experiment</a> </h2> Let's get hands-on and create a table, fill it, delete most of it and watch what happens.</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> pgstattuple;</span></span> CREATE TABLE</span> demo</span> (id </span>integer PRIMARY KEY</span>, </span>data text</span>);</span></span> </span> -- insert 100,000 rows</span></span> INSERT INTO</span> demo (id, </span>data</span>)</span></span> SELECT</span> g, </span>'Row number '</span> \|\|</span> g </span>\|\|</span> ' with some extra data'</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>) g;</span></span> </span> ANALYZE demo;</span></span></code></pre> At this point, our index is healthy. Let's capture the baseline:</p> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(</span>oid</span>)) </span>as</span> file_size,</span></span> pg_size_pretty((pgstattuple(</span>oid</span>)).tuple_len) </span>as</span> actual_data</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 6434 kB</span></span> demo_pkey \| 2208 kB \| 1563 kB</span></span></code></pre> Now remove some data, 80% to be precise - somewhere in the middle:</p> DELETE FROM</span> demo </span>WHERE</span> id </span>BETWEEN</span> 10001</span> AND</span> 90000</span>;</span></span></code></pre> The goal is to simulate a common real-world pattern: data retention policies, bulk cleanup operations, or the aftermath of a data migration gone wrong</a>.</p> VACUUM demo;</span></span> </span> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(oid)) as file_size,</span></span> pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data</span></span> FROM pg_class</span></span> WHERE relname IN ('demo', 'demo_pkey');</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 1278 kB</span></span> demo_pkey \| 2208 kB \| 313 kB</span></span></code></pre> The table shrunk significantly, while index remained unchanged You now have 20,000 rows indexed by a structure built to handle 100,000.</p> But notice something important: even though actual_data</code> dropped to ~1.3 MB (reflecting our 20,000 remaining rows), the file_size</code> for the index is still 2208 kB. VACUUM cleaned out the dead tuple data, but it doesn't return space to the OS or compact index structures - it only marks pages as reusable within PostgreSQL.</p> EDIT: the original table had a size without running VACCUM, which I must left out during what's sometime cumbersome process of authoring blog posts. Big thanks come to Creston Jamison of Scaling PostgreSQL</a> to notice it, and actually take what must have been a lot of time to analyze my posts.</p> </blockquote> This experiment is really an extreme case, but demonstrates the problem.</p> Understanding page states</a> </h2> Leaf pages have several states:</p> Full page (>80% density)</strong>, when the page contains many index entries, efficiently utilizing space. Each 8KB page</a> read returns substantial useful data. This is optimal state.</p> Partial page (40-80% density)</strong> with some wasted space, but still reasonably efficient. Common at tree edges or after light churn. Nothing to be worried about.</p> Sparse page (<40% density)</strong> is mostly empty. You're reading an 8KB page to find a handful of entries. The I/O cost is the same as a full page, but you get far less value.</p> Empty page (0% density)</strong> with zero live entries, but the page still exists in the tree structure. Pure overhead. You might read this page during a range scan and find absolutely nothing useful.</p> A note on fillfactor</a> </h3> You might be wondering how can fillfactor help with this? It's the setting you can apply both for heap and leaf pages, and controls how full PostgreSQL packs the pages during the data storage. The default value for B-tree indexes is 90%</strong>. This leaves 10% of free space on each leaf page for future insertions.</p> CREATE INDEX</span> demo_index</span> ON</span> demo(id) </span>WITH</span> (fillfactor </span>=</span> 70</span>);</span></span></code></pre> A lower fillfactor (like 70%) leaves more room, which can reduce page splits when you're inserting into the middle of an index - useful for tables random index column inserts or those with heavily updated index columns.</p> But if you followed carefully the anatomy of storage section, it doesn't help with the bloat problem. Quite the opposite. If you set lower fillfactor and then delete majority of your rows, you actually start with more pages, and bigger chance to end up with more sparse pages than partial pages.</p> Leaf page fillfactor is about optimizing for updates and inserts. It's not a solution for deletion or index-column update bloat.</p> Why the planner gets fooled</a> </h2> PostgreSQL's query planner estimates costs based on physical statistics</a>, including the number of pages in an index.</p> EXPLAIN ANALYZE </span>SELECT * FROM</span> demo </span>WHERE</span> id </span>BETWEEN</span> 10001</span> AND</span> 90000</span>;</span></span></code></pre>QUERY PLAN</span></span> --------------------------------------------------------------------------------------------------------------------</span></span> Index Scan using demo_pkey on demo (cost=0.29..29.29 rows=200 width=41) (actual time=0.111..0.112 rows=0 loops=1)</span></span> Index Cond: ((id >= 10001) AND (id <= 90000))</span></span> Planning Time: 1.701 ms</span></span> Execution Time: 0.240 ms</span></span> (4 rows)</span></span></code></pre> While the execution is almost instant, you need to look behind the scenes. The planner estimated 200 rows and got zero. It traversed the B-tree structure expecting data that doesn't exist. On a single query with warm cache, this is trivial. Under production load with thousands of queries and cold pages, you're paying I/O cost for nothing. Again and again.</p> If you dig further you discover much bigger problem.</p> SELECT</span> relname, reltuples::</span>bigint as</span> row_estimate, relpages </span>as</span> page_estimate</span></span> FROM</span> pg_class </span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| row_estimate \| page_estimate</span></span> -----------+--------------+---------------</span></span> demo \| 20000 \| 934</span></span> demo_pkey \| 20000 \| 276</span></span></code></pre> The relpages</code> value comes from the physical file size divided by the 8 KB page size</a>. PostgreSQL updates it during VACUUM and ANALYZE, but it reflects the actual file on disk - not how much useful data is inside. Our index file is still 2.2 MB (276 pages × 8 KB), even though most pages are empty.</p> The planner sees 276 pages for 20,000 rows and calculates a very low rows-per-page ratio. This is when planner can come to conclusion - this index is very sparse - let's do a sequential scan instead</em>. Oops.</p> "But wait," you say, "doesn't ANALYZE fix statistics?"</p> Yes and no. ANALYZE</code> updates the row count estimate. It will no longer think you have 100,000 rows but 20,000. But it does not shrink relpages, because that reflects the physical file size on disk. ANALYZE</code> can't change that.</p> The planner now has accurate row estimates but wildly inaccurate page estimates. The useful data is packed into just ~57 pages worth of entries, but the planner doesn't know that.</p> cost = random_page_cost × pages + cpu_index_tuple_cost × tuples ([visible in EXPLAIN output](/posts/explain-buffers/))</span></span></code></pre> With a bloated index:</p> pages are oversized (276 instead of ~57)</li> The per-page cost gets multiplied by empty pages</li> Total estimated cost is artificially high</li> </ul> The hollow index</a> </h2> We can dig even more into the index problem when we look at internal stats:</p> SELECT * FROM</span> pgstatindex(</span>'demo_pkey'</span>);</span></span></code></pre>-[ RECORD 1 ]------+--------</span></span> version \| 4</span></span> tree_level \| 1</span></span> index_size \| 2260992</span></span> root_block_no \| 3</span></span> internal_pages \| 1</span></span> leaf_pages \| 57</span></span> empty_pages \| 0</span></span> deleted_pages \| 217</span></span> avg_leaf_density \| 86.37</span></span> leaf_fragmentation \| 0</span></span></code></pre> Wait, what? The avg_leaf_density is 86% and it looks perfectly healthy. That's a trap. Due to the hollow index (we removed 80% right in the middle) we have 57 well-packed leaf pages, but the index still contains 217 deleted pages.</p> This is why avg_leaf_density</code> alone is misleading. The density of used pages looks great, but 79% of your index file is dead weight.</p> The simplest way to spot index bloat is comparing actual size to expected size.</p> SELECT</span></span> c</span>.</span>relname</span> as</span> index_name,</span></span> pg_size_pretty(pg_relation_size(</span>c</span>.</span>oid</span>)) </span>as</span> actual_size,</span></span> pg_size_pretty((</span>c</span>.</span>reltuples</span> </span> 40</span>)::</span>bigint</span>) </span>as</span> expected_size,</span></span> round</span>((pg_relation_size(</span>c</span>.</span>oid</span>) </span>/</span> nullif</span>(</span>c</span>.</span>reltuples</span> </span> 40</span>, </span>0</span>))::</span>numeric</span>, </span>1</span>) </span>as</span> bloat_ratio</span></span> FROM</span> pg_class c</span></span> JOIN</span> pg_index i </span>ON</span> c</span>.</span>oid</span> =</span> i</span>.</span>indexrelid</span></span> WHERE</span> c</span>.</span>relkind</span> =</span> 'i'</span> </span></span> AND</span> c</span>.</span>reltuples</span> ></span> 0</span></span> AND</span> c</span>.</span>relname</span> NOT LIKE</span> 'pg_%'</span></span> AND</span> pg_relation_size(</span>c</span>.</span>oid</span>) </span>></span> 1024</span> </span> 1024</span> -- only indexes > 1 MB</span></span> ORDER BY</span> bloat_ratio </span>DESC NULLS LAST</span>;</span></span></code></pre>index_name \| actual_size \| expected_size \| bloat_ratio</span></span> ------------+-------------+---------------+-------------</span></span> demo_pkey \| 2208 kB \| 781 kB \| 2.8</span></span></code></pre> A bloat_ratio</code> of 2.8 means the index is nearly 3x larger than expected. Anything above 1.8 - 2.0 deserves investigation.</p> We filter to indexes over 1 MB - bloat on tiny indexes doesn't matter that much. Please, adjust the threshold based on your environment; for large databases, you might only care about indexes over 100 MB.</p> But here comes BIG WARNING</strong>: pgstatindex() we used earlier physically reads the entire index. On a 10 GB index, that's 10 GB of I/O. Don't run it against all indexes on a production server - unless you know what you are doing!</p> REINDEX</a> </h2> How to actually fix index bloat problem? REINDEX</code> is s straightforward solution as it rebuilds the index from scratch.</p> REINDEX </span>INDEX</span> CONCURRENTLY demo_pkey ;</span></span></code></pre> After which we can check the index health:</p> SELECT FROM</span> pgstatindex(</span>'demo_pkey'</span>);</span></span></code></pre>-[ RECORD 1 ]------+-------</span></span> version \| 4</span></span> tree_level \| 1</span></span> index_size \| 466944</span></span> root_block_no \| 3</span></span> internal_pages \| 1</span></span> leaf_pages \| 55</span></span> empty_pages \| 0</span></span> deleted_pages \| 0</span></span> avg_leaf_density \| 89.5</span></span> leaf_fragmentation \| 0</span></span></code></pre> And</p> SELECT</span></span> relname,</span></span> pg_size_pretty(pg_relation_size(</span>oid</span>)) </span>as</span> file_size,</span></span> pg_size_pretty((pgstattuple(</span>oid</span>)).tuple_len) </span>as</span> actual_data</span></span> FROM</span> pg_class</span></span> WHERE</span> relname </span>IN</span> (</span>'demo'</span>, </span>'demo_pkey'</span>);</span></span></code></pre>relname \| file_size \| actual_data</span></span> -----------+-----------+-------------</span></span> demo \| 7472 kB \| 1278 kB</span></span> demo_pkey \| 456 kB \| 313 kB</span></span></code></pre> Our index shrunk from 2.2 MB to 456 KB - 79% reduction (not a big surprise though).</p> As you might have noticed we have used CONCURRENTLY</code> to avoid using ACCESS EXCLUSIVE lock. This is available since PostgreSQL 12+, and while there's an option to omit it - the pretty much only reason to do so is during planned maintenance to speed up the index rebuild time.</p> pg_squeeze</a> </h2> If you look above at the file_size of our relations, we have managed to reclaim the disk space for the affected index (it was REINDEX</code> after all), but the table space was not returned back to the operating system.</p> That's where pg_squeeze</a> shines. Unlike trigger-based alternatives, pg_squeeze uses logical decoding, resulting in lower impact on your running system. It rebuilds both the table and all its indexes online, with minimal locking:</p> CREATE</span> EXTENSION pg_squeeze;</span></span> </span> SELECT</span> squeeze</span>.</span>squeeze_table</span>(</span>'public'</span>, </span>'demo'</span>);</span></span></code></pre> The exclusive lock is only needed during the final swap phase, and its duration can be configured. Even better, pg_squeeze is designed for regular automated processing - you can register tables and let it handle maintenance whenever bloat thresholds are met.</p> pg_squeeze makes sense when both table and indexes are bloated, or when you want automated management. REINDEX CONCURRENTLY is simpler when only indexes need work.</p> There's also older tool pg_repack - for a deeper comparison of bloat-busting tools, see article The Bloat Busters: pg_repack vs pg_squeeze</a>.</p> VACUUM FULL (The nuclear option)</a> </h2> VACUUM FULL</code> rewrites the entire table and all indexes. While it fixes everything it comes with a big but - it requires an ACCESS EXCLUSIVE lock - completely blocking all reads and writes for the entire duration. For a large table, this could mean hours of downtime.</p> Generally avoid this in production</strong>. Use pg_squeeze instead for the same result without the downtime.</p> When to act, and when to chill</a> </h2> Before you now go and REINDEX</code> everything in sight, let's talk about when index bloat actually matters.</p> B-trees expand and contract with your data</strong>. With random insertions affecting index columns - UUIDs, hash keys, etc. the page splits happen constantly. Index efficiency might get hit at occassion and also settle around 70 - 80% over different natural cycles of your system usage. That's not bloat. That's the tree finding its natural shape for your data.</p> The bloat we demonstrated - 57 useful pages drowning in 217 deleted ones - is extreme. It came from deleting 80% of contiguous data. You won't see this from normal day to day operations.</p> When do you need to act immediately:</p> after a massive DELETE (retention policy, GDPR purge, failed migration cleanup)</li> bloat_ratio</code> exceeds 2.0 and keeps climbing</li> query plans suddenly prefer sequential scans on indexed columns</li> index size is wildly disproportionate to row count</li> </ul> But in most cases you don't have to panic. Monitor weekly and when indexes bloat ratio continously grow above warning levels, schedule a REINDEX CONCURRENTLY</code> during low traffic period.</p> Index bloat isn't an emergency until it is. Know the signs, have the tools ready, and don't let VACUUM's silence fool you into thinking everything's fine.</p> Conclusion</a> </h2> VACUUM is essential for PostgreSQL. Run it. Let autovacuum do its job. But understand its limitations: it cleans up dead tuples, not index structure.</p> The truth about PostgreSQL maintenance is that VACUUM handles heap bloat reasonably well, but index bloat requires explicit intervention. Know when your indexes are actually sick versus just breathing normally - and when to reach for REINDEX.</p> VACUUM handles heap bloat. Index bloat is your problem. Know the difference.</p> RegreSQL: Regression Testing for PostgreSQL Queries 2025-11-13T22:47:00+00:00 TL;DR</strong> - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production.</em></p> As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification.</p> For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</em></a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries.</p> I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level.</p> Introduction</a> </h2> The RegreSQL</a></strong> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later).</p> But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere?</p> This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations.</p> The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management.</p> Basic regression testing</a> </h2> Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like</p> $ regresql text</span></span> Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> </span> Results: 8 passed, 0 failed, 8 skipped (0.00s)</span></span></code></pre> In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for .sql</code> files and attempts to run all queries against the configured PostgreSQL connection.</p> The individual files can contain either single or multiple sql queries. Like following example</p> -- name: top-artists-by-album</span></span> -- Get the list of the N artists with the most albums</span></span> SELECT</span></span> artist</span>.</span>name</span>,</span></span> count</span>(</span></span>) </span>AS</span> albums</span></span> FROM</span></span> artist</span></span> LEFT JOIN</span> album </span>USING</span> (artist_id)</span></span> GROUP BY</span></span> artist</span>.</span>name</span></span> ORDER BY</span></span> albums </span>DESC</span></span> LIMIT</span> :n;</span></span></code></pre> The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now).</p> This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines.</p> Performance regression testing</a> </h2> Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts.</p> RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated</p> $ regresql baseline</span></span> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> Creating baselines directory: regresql/baselines</span></span> Creating directory 'regresql/baselines'</span></span> </span> Creating baselines for queries:</span></span> </span> ./</span></span> Created baseline: album-by-artist_list-albums-by-artist.1.json</span></span> Created baseline: album-by-artist_list-albums-by-artist.2.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.1.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.2.json</span></span> Created baseline: artist_top-artists-by-album.1.json</span></span> Created baseline: genre-topn_genre-top-n.top-1.json</span></span> Created baseline: genre-topn_genre-top-n.top-3.json</span></span> Created baseline: genre-tracks_tracks-by-genre.json</span></span> </span> Baselines have been created successfully!</span></span> Baseline files are stored in: regresql/baselines</span></span></code></pre> the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place.</p> To demonstrate this, let's review the test output with the baselines.</p> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> ✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: album, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: artist, genre</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> ✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, track</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> Results: 16 passed (0.00s)</span></span></code></pre> As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready".</p> In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below.</p> -- name: query_name</span></span> -- metadata: key1=value1, key2=value2</span></span> SELECT</span> ...;</span></span></code></pre> At this point RegreSQL recognizes</p> notest</code> to skip the query testing altogether (not just cost tracking)</li> nobaseline</code> to skip cost tracking</li> noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li> and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li> </ul> -- name: query_name</span></span> -- regresql: notest, nobaseline</span></span> -- regresql: noseqscanwarn</span></span> -- regresql: difffloattolerance:0.25</span></span> -- query that can vary in cost by 20% without being considered a failure</span></span> SELECT</span> ...;</span></span></code></pre>ORM enters the room</a> </h2> ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data:</p> orders</span> =</span> (</span></span> session.query(Order)</span></span> .filter(Order.user_id</span> ==</span> user_id)</span></span> .options(</span></span> joinedload(Order.user),</span></span> joinedload(Order.shipping_address),</span></span> selectinload(Order.items)</span> # NEW: Load order items</span></span> )</span></span> .all()</span></span> )</span></span></code></pre> That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan.</p> RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system:</p> @event.listens_for</span>(engine,</span> "before_cursor_execute"</span>)</span></span> def</span> capture_sql</span>(conn, cursor, statement,</span> </span>args):</span></span> captured_queries.append(statement)</span></span></code></pre> Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development.</p> Test Data Management</a> </h2> Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky.</p> Traditional approach to create test data might involve</p> Database dumps</strong> become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li> SQL scripts</strong> might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li> Factories in application code</strong> might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li> Shared test database</strong> is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li> </ul> What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000).</p> This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale.</p> RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example</p> fixture</span>:</span> basic_users</span></span> description</span>:</span> a handful of test users</span></span> cleanup</span>:</span> rollback</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> alice@example.com</span></span> name</span>:</span> Alice Anderson</span></span> created_at</span>:</span> 2024-01-15</span></span> -</span> id</span>:</span> 2</span></span> email</span>:</span> bob@example.com</span></span> name</span>:</span> Bob Builder</span></span> created_at</span>:</span> 2024-02-20</span></span></code></pre> To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture</p> fixtures</span>:</span></span> -</span> basic_users</span></span> </span> "1"</span>:</span></span> email</span>:</span> alice@example.com</span></span> </span> "2"</span>:</span></span> email</span>:</span> bob@example.com</span></span></code></pre> And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators</strong> including</p> sequences, random integer, decimal, string, uuid, email and name generators</li> date_between for generating random timestamps within a range</li> foreign key references to be able to reuse data from other table's fixtures</li> range to select value from predefined sources</li> Go template support</li> </ul> fixture</span>:</span> realistic_orders</span></span> generate</span>:</span></span> -</span> table</span>:</span> customers</span></span> count</span>:</span> 1000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> email</span>:</span></span> generator</span>:</span> email</span></span> domain</span>:</span> shop.example.com</span></span> name</span>:</span></span> generator</span>:</span> name</span></span> type</span>:</span> full</span></span> created_at</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span> </span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 5000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> customer_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1000</span></span> amount</span>:</span></span> generator</span>:</span> decimal</span></span> min</span>:</span> 10.00</span></span> max</span>:</span> 999.99</span></span> precision</span>:</span> 2</span></span> order_date</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span></code></pre> This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like.</p> The fixtures are also stackable</strong> and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse.</p> fixture</span>:</span> orders_with_shipping</span></span> depends_on</span>:</span></span> -</span> basic_users</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> orders</span></span> rows</span>:</span></span> -</span> id</span>:</span> 101</span></span> user_id</span>:</span> 1</span> # References Alice from basic_users</span></span> total</span>:</span> 99.99</span></span> status</span>:</span> shipped</span></span></code></pre> Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation.</p> fixture</span>:</span> mixed_setup</span></span> description</span>:</span> Combine SQL with YAML and generated data</span></span> cleanup</span>:</span> rollback</span></span> </span> # SQL executes first (either as file or inline)</span></span> sql</span>:</span></span> -</span> file</span>:</span> sql/setup_schema.sql</span></span> -</span> inline</span>:</span> "INSERT INTO config (key, value) VALUES ('version', '1.0');"</span></span> </span> # followed YAML data</span></span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> admin@example.com</span></span> </span> # and finally generated data</span></span> generate</span>:</span></span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 100</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> user_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1</span></span></code></pre> RegreSQL provides commands to inspect and validate your fixtures</p> # List all available fixtures</span></span> regresql</span> fixtures list</span></span> </span> # Show fixture details and dependencies</span></span> regresql</span> fixtures show realistic_orders</span></span> </span> # Validate fixture definitions</span></span> regresql</span> fixtures validate</span></span> </span> # Show dependency graph</span></span> regresql</span> fixtures deps</span></span> </span> # Apply fixture manually (for debugging)</span></span> regresql</span> fixtures apply basic_users</span></span></code></pre> The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves.</p> (EDIT 2025-11-20)</strong> The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use</p> Fixed generated fixtures re-used between tests</li> Use RegreSQL to test fixtures before/after migration</li> </ul> I'm working on next release of RegreSQL to address DX of fixtures usage.</p> RegreSQL future</a> </h2> Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL</strong> brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development.</p> At the same time RegreSQL</strong> is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable.</p> RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>.</p> Beyond Start and End: PostgreSQL Range Types 2025-11-02T21:40:00+00:00 One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them.</p> Bug Not Invented Here</a> </h2> But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets.</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> </span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> -- define the hold period explicitly</span></span> hold_started_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP,</span></span> hold_expires_at </span>TIMESTAMPTZ NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span> </span> CREATE INDEX</span> seat_holds_on_seat_id</span> ON</span> seat_holds(seat_id);</span></span> CREATE INDEX</span> seat_holds_on_expiration</span> ON</span> seat_holds(hold_expires_at);</span></span></code></pre> While the table design looks perfectly reasonable, it has one serious flaw</strong> - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee.</p> If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain.</p> Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention</strong> situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.).</p> And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level.</p> -- needed to add GiST support to equality of integers (for seat_id)</span></span> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> btree_gist;</span></span> </span> ALTER TABLE</span> seat_holds </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> tsrange(hold_started_at, hold_expires_at) </span>WITH</span> &&</span></span> );</span></span></code></pre> Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - you're making it impossible to have invalid state in your database, not just unlikely. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load.</p> But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box.</p> Introducing the Data Range Types</a> </h2> You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in?</p> This brings us to the core of the matter: PostgreSQL's native Range Types</strong>.</p> PostgreSQL provides a set of built-in range types, all following the pattern of type and range:</p> int4range</code> for integer</li> int8range</code> for bigint</li> numrange</code> for numeric</li> tsrange</code> for timestamp without time zone</li> tstzrange</code> for timestamp with time zone (which we briefly saw above)</li> daterange</code> for date</li> </ul> And it does not stop there. You can easily define your own custom range types</strong> over any basic data type.</p> First win: cleaner schema</a> </h2> When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code.</p> When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct. The application code no longer needs to manage two separate boundaries.</p> CREATE TABLE</span> seat_holds_native</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span></code></pre> This is the power of a first-class data type</strong>. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly.</p> Second win: Atomicity guaranteed</a> </h2> While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint</strong> as demonstrated previously.</p> ALTER TABLE</span> seat_holds_native </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> hold_period </span>WITH</span> &&</span></span> );</span></span></code></pre> This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once:</p> seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li> hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li> </ol> Finally EXCLUDE USING gist</code> is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint. This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.).</p> Range boundaries: A quick math refresher</a> </h2> Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics.</p> PostgreSQL ranges can have four different boundary types:</p> Inclusive boundaries -</strong> [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to".</p> -- [10, 20] includes both 10 and 20</span></span> -- represents 10 ≤ x ≤ 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>);</span></span></code></pre> Exclusive boundaries -</strong> (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality.</p> -- (10, 20) excludes both 10 and 20</span></span> -- represents 10 < x < 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'()'</span>);</span></span> </span></code></pre> Mixed boundaries -</strong> [)</code> or (]</code> You can mix and match. The most common and default pattern</strong> in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next".</p> As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods.</p> -- these ranges are adjacent, not overlapping</span></span> -- Week 1</span></span> SELECT</span> '[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Week 2</span></span> SELECT</span> '[2025-11-08, 2025-11-15)'</span>::tstzrange;</span></span></code></pre> With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments.</p> Canonicalization and Range Set Operations</a> </h2> Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges.</p> For discrete</strong> range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper).</p> PostgreSQL uses a canonicalization function</strong> to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing.</p> -- [10, 20] includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span> </span> -- (9, 21) includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>9</span>, </span>21</span>, </span>'()'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span></code></pre> Exception are continuous ranges</strong> (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning.</p> The operator toolkit</a> </h2> The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier.</p> Overlap operator - &&</code></strong> As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points.</p> -- find holds active at any point between 10:00 and 11:00</span></span> SELECT FROM</span> seat_holds_native </span>WHERE</span> hold_period && </span>'[2025-12-25 10:00, 2025-12-25 11:00)'</span>::tstzrange;</span></span></code></pre> Contains operator - @></code></strong> For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type).</p> -- find holds that are active at the specific momement</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '2025-11-05 15:00'</span>::</span>timestamptz</span>;</span></span> </span> -- find holds that are active at the specific time range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '[2025-12-25 10:00, 2025-12-25 10:15)'</span>::tstzrange;</span></span></code></pre> Contained By operator - <@</code></strong> In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right.</p> -- find holds that are within November '25</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span><</span>@ </span>'[2025-11-01, 2025-12-01)'</span>::tstzrange;</span></span></code></pre> Strictly Before/After operators - <<</code> and >></code></strong> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries).</p> -- find holds that finished strictly before 10 November</span></span> SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</span></span></code></pre> Boundary Extension operators - &<</code> and &></code></strong> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range).</p> -- find holds that end before or at the same time as reference range ends</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span><</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span> </span> -- find holds that start at or after reference range starts</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span>></span>'[2025-11-08 09:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre> Finally the Adjecent operator - -\|-</code></strong> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict.</p> -- find holds that are immediately adjacent (touching) to given range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>-</span>\|</span>-</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre>To Infinity and Beyond</a> </h2> Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>.</p> Let's start with infinite bounds</strong>, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time).</p> -- range that never expires (upper bound is infinite)</span></span> SELECT</span> '[2025-11-01 10:00, infinity)'</span>::tstzrange;</span></span> </span> -- range that has always been valid (lower bound is infinite)</span></span> SELECT</span> '[-infinity, 2025-11-01 10:00)'</span>::tstzrange;</span></span> </span> -- range covering all time</span></span> SELECT</span> '[-infinity, infinity)'</span>::tstzrange;</span></span></code></pre> This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription.</p> -- lifetime subscription that never expires</span></span> INSERT INTO</span> subscriptions (user_id, plan, active_period)</span></span> VALUES</span> (</span>42</span>, </span>'lifetime'</span>, </span>'[2025-11-01, infinity)'</span>);</span></span> </span> -- all active subscriptions right now</span></span> SELECT * FROM</span> subscriptions</span></span> WHERE</span> active_period @</span>> NOW</span>();</span></span></code></pre> Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent.</p> At any point you can validate whatever range has infinite bounds:</p> SELECT</span></span> lower_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> lower_is_infinite,</span></span> upper_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> upper_is_infinite;</span></span></code></pre>Understanding NULL vs empty: Schrödinger's Range</a> </h2> Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty.</p> Let's see this in practice:</p> -- NULL range: we don't know what the period is</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>42</span>, </span>'abc-123'</span>, </span>NULL</span>);</span></span> </span> -- empty range: we know the period is explicitly "nothing"</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>43</span>, </span>'def-456'</span>, </span>'empty'</span>);</span></span></code></pre> The main difference between them is when it comes to handling.</p> SELECT NULL</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: NULL (not true, not false—we don't know)</span></span> </span> SELECT</span> 'empty'</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: false (we know it doesn't overlap)</span></span></code></pre> And you can check for them in your queries using built-in function isempty</code>.</p> -- check for NULL (like any column)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>IS NULL</span>;</span></span> </span> -- check for empty (special function)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> isempty(hold_period);</span></span></code></pre> In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail.</p> Practical Integer ranges for Tiered pricing</a> </h2> While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing.</p> CREATE TABLE</span> quantity_discounts</span> (</span></span> discount_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> product_id </span>INTEGER NOT NULL</span>,</span></span> quantity_range INT4RANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- no overlapping tiers</span></span> EXCLUDE </span>USING</span> GIST (</span></span> product_id </span>WITH =</span>,</span></span> quantity_range </span>WITH</span> &&</span></span> )</span></span> );</span></span> </span> INSERT INTO</span> quantity_discounts (product_id, quantity_range, discount_percentage) </span>VALUES</span></span> -- 1-9 units: no discount</span></span> (</span>1</span>, </span>'[1,10)'</span>, </span>0</span>.</span>00</span>),</span></span> -- 10-49 units: 5% off</span></span> (</span>1</span>, </span>'[10,50)'</span>, </span>5</span>.</span>00</span>),</span></span> -- 50-99 units: 10% off</span></span> (</span>1</span>, </span>'[50,100)'</span>, </span>10</span>.</span>00</span>),</span></span> -- 100+ units: 15% off</span></span> (</span>1</span>, </span>'[100,1000)'</span>, </span>15</span>.</span>00</span>);</span></span> </span> -- verify what discount we offer for ordering 75 units?</span></span> SELECT</span> discount_percentage</span></span> FROM</span> quantity_discounts</span></span> WHERE</span> product_id </span>=</span> 1</span></span> AND</span> quantity_range @</span>></span> 75</span>;</span></span> -- Result: 10.00</span></span></code></pre>Making Bad Data Impossible</a> </h2> If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database.</p> CREATE TABLE</span> promotional_campaigns</span> (</span></span> campaign_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span>,</span></span> budget_range NUMRANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- campaigns must be at least 1 days long</span></span> CONSTRAINT</span> campaigns_minimum_duration</span></span> CHECK</span> (</span>upper</span>(active_period) </span>-</span> lower</span>(active_period) </span>>=</span> INTERVAL </span>'1 days'</span>),</span></span> </span> -- budget must be between $1000 and $100000</span></span> CONSTRAINT</span> campaigns_valid_budget</span></span> CHECK</span> (budget_range </span><</span>@ numrange(</span>1000</span>, </span>100000</span>, </span>'[]'</span>)),</span></span> </span> -- active period must not be empty</span></span> CONSTRAINT</span> campaigns_valid_period</span></span> CHECK</span> (</span>NOT</span> isempty(active_period))</span></span> );</span></span></code></pre> While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions).</p> Multiranges: When one range is not enough</a> </h2> Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column.</p> PostgreSQL 14 introduced multirange types for all the built-in range types:</p> int4multirange</code>, int8multirange</code>, nummultirange</code></li> tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li> </ul> The real power of multiranges lies in schema density</strong> and query efficiency</strong>. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas.</p> Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data.</p> CREATE TABLE</span> user_periods_single_range</span> (</span></span> period_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> user_id </span>INTEGER NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_single_range_gist_idx</span></span> ON</span> user_periods_single_range</span></span> USING</span> gist (active_period);</span></span> </span> -- 20 fragmented periods for user_id 42 (20 rows total)</span></span> INSERT INTO</span> user_periods_single_range (user_id, active_period)</span></span> SELECT</span></span> 42</span>,</span></span> tstzrange(</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> \|\|</span> ' days'</span>)::interval,</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> +</span> 1</span> \|\|</span> ' days'</span>)::interval</span></span> )</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>20</span>) </span>AS</span> s(i);</span></span></code></pre> compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data.</p> CREATE TABLE</span> user_periods_multirange</span> (</span></span> user_id </span>INTEGER PRIMARY KEY</span>,</span></span> active_periods TSTZMULTIRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_multirange_gist_idx</span></span> ON</span> user_periods_multirange</span></span> USING</span> gist (active_periods);</span></span> </span> -- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row</span></span> INSERT INTO</span> user_periods_multirange (user_id, active_periods)</span></span> SELECT</span></span> user_id,</span></span> -- range_agg function automatically handles merging adjacent ranges</span></span> range_agg(active_period)::TSTZMULTIRANGE</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span> user_id </span>=</span> 42</span></span> GROUP BY</span> user_id;</span></span></code></pre> consider now the difference between</p> EXPLAIN</span></span> SELECT</span> period_id, user_id</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_period @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------</span></span> Bitmap Heap Scan on user_periods_single_range (cost=4.19..13.67 rows=1 width=12)</span></span> Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span> Filter: (user_id = 42)</span></span> -> Bitmap Index Scan on user_periods_single_range_gist_idx (cost=0.00..4.19 rows=6 width=0)</span></span> Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> and same version of the consolidated data</p> EXPLAIN ANALYZE</span></span> SELECT</span> user_id, active_periods</span></span> FROM</span> user_periods_multirange</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_periods @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------------------</span></span> Index Scan using user_periods_multirange_pkey on user_periods_multirange (cost=0.15..8.17 rows=1 width=36)</span></span> Index Cond: (user_id = 42)</span></span> Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements.</p> One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps.</p> Creating custom range types</a> </h2> While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges.</p> To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values:</p> -- function to calculate difference between two IP addresses using bigint</span></span> CREATE OR REPLACE FUNCTION</span> inet_diff</span>(x </span>INET</span>, y </span>INET</span>)</span></span> RETURNS</span> FLOAT8 </span>AS</span> $$</span></span> SELECT</span> (</span></span> (host(x)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>) </span>-</span></span> (host(y)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>)</span></span> )::float8;</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span> </span> -- custom inetrange type</span></span> CREATE TYPE</span> inetrange</span> AS RANGE</span> (</span></span> subtype </span>= inet</span>,</span></span> subtype_diff </span>=</span> inet_diff</span></span> );</span></span> </span> -- convert CIDR ranges to inetranges</span></span> CREATE OR REPLACE FUNCTION</span> cidr_to_inetrange</span>(</span>cidr CIDR</span>)</span></span> RETURNS</span> inetrange </span>AS</span> $$</span></span> SELECT</span> inetrange(</span></span> host(network($</span>1</span>))::</span>inet</span>,</span></span> host(broadcast($</span>1</span>))::</span>inet</span>,</span></span> '[]'</span></span> );</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span></code></pre> Now you can use these custom types just like the built-in ones:</p> CREATE TABLE</span> ip_blocklists</span> (</span></span> blocklist_name </span>TEXT PRIMARY KEY</span>,</span></span> blocked_range inetrange </span>NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> ip_blocklists (blocklist_name, blocked_range) </span>VALUES</span></span> (</span>'attack #434401'</span>, cidr_to_inetrange(</span>'192.168.1.0/24'</span>)),</span></span> (</span>'attack #434401 (1)'</span>, </span>'[203.0.113.50,203.0.113.99]'</span>),</span></span> (</span>'attack #434401 (2)'</span>, </span>'[203.0.113.143,203.0.113.159]'</span>);</span></span></code></pre> and locate which attack has been assigned to particular malicious IP address.</p> SELECT</span> blocklist_name, blocked_range</span></span> FROM</span> ip_blocklists</span></span> WHERE</span> blocked_range @</span>></span> '192.168.1.25'</span>::</span>INET</span>;</span></span></code></pre> blocklist_name \| blocked_range</span></span> ----------------+-----------------------------</span></span> attack #434401 \| [192.168.1.0,192.168.1.255]</span></span></code></pre> But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block.</p> Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges.</p> SELECT</span> typname </span>FROM</span> pg_type </span>WHERE</span> typname </span>LIKE</span> 'inet%range'</span>;</span></span></code></pre> typname</span></span> ----------------</span></span> inetmultirange</span></span> inetrange</span></span></code></pre> Danger zone! When creating a custom range type the subtype_diff</strong> function is more than just simple helper. It plays important role in indexing and query performance</strong>. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges.</p> In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans.</p> Performance deep dive: GiST vs GIN</a> </h2> Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt.</p> Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree)</strong> is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index)</strong> would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space.</p> The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with</strong> EXCLUDE constraints.</strong> GiST is mandatory there.</p> Therefore while GiST index is going to be preferred for cases like</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span></span> );</span></span></code></pre> the GIN index is actually preferred for</p> CREATE TABLE</span> venue_blackouts</span> (</span></span> venue_id </span>INTEGER</span>,</span></span> blocked_dates DATERANGE </span>NOT NULL</span></span> );</span></span></code></pre> The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day.</p> The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available.</p> Conclusion</a> </h2> From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design.</p> But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables</strong>. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article.</p> For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes. Your future self - the one debugging a problems in least convenient time - will thank you.</p> PostgreSQL maintenance without superuser 2025-09-13T20:55:00+00:00 How many people/services have superuser access to your PostgreSQL cluster(s)? Did you ever ask why your software engineers might need it? Or your BI team? Why those use cases require same privileges as someone who can drop your databases?</p> The answer isn't because these operations are inherently dangerous - it's because PostgreSQL historically offered limited options for operational access or simply because not enough people are aware of the options. So the common practice is to either got basic permissions or handover the keys to the kingdom.</p> PostgreSQL's built-in predefined roles</strong> solve this problem by providing purpose-built privileges for common maintenance tasks. Instead of granting superuser access for routine operations, you can delegate specific capabilities - monitoring teams get comprehensive observability access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need, nothing more.</p> What are Predefined Roles?</a> </h2> PostgreSQL's built-in administrative roles are purpose-built permission sets that solve the superuser dilemma for common maintenance tasks. Out of the box, there are 15 predefined roles that provide granular access to specific operational capabilities without requiring full superuser privileges.</p> While you can view their list and description in official documentation</a>, in this article we will explore them bit more thoroughly and at the same time look into system catalogs to understand them better. The individual roles can be grouped by their functionality and most of them are quite easy to grasp, ranging from simple monitoring access to powerful filesystem operations that require careful consideration.</p> Data Access Role</strong></p> pg_database_owner</code> - Database-specific ownership (special case)</li> pg_read_all_data</code> - Read access to all tables, views, sequences</li> pg_write_all_data</code> - Write access to all tables, views, sequences</li> </ul> Monitoring & Observability</strong></p> pg_monitor</code> - Which is actually monitoring meta role that contains 3 roles listed below</li> pg_read_all_settings</code> - Configuration access</li> pg_read_all_stats</code> - Statistics views</li> pg_stat_scan_tables</code> - Table scanning for stats</li> </ul> System Operations</strong></p> pg_signal_backend</code> - Cancel queries, terminate sessions</li> pg_checkpoint</code> - Run CHECKPOINT command</li> pg_maintain</code> - VACUUM, ANALYZE, REINDEX operations (PostgreSQL 17+)</li> pg_signal_autovacuum_worker</code> - Signal autovacuum worker (PostgreSQL 18+)</li> </ul> File System Access</strong></p> pg_read_server_files</code> - Read files from server filesystem</li> pg_write_server_files</code> - Write files to server filesystem</li> pg_execute_server_program</code> - Execute programs on server</li> </ul> And specialised use cases</strong></p> pg_create_subscription</code> - Logical replication management</li> pg_use_reserved_connections</code> - Connection reservation (PostgreSQL 16+)</li> </ul> Why Use Predefined Roles?</a> </h2> The primary benefit of predefined roles is expanding the pool of users who can safely manage PostgreSQL databases</strong>. Traditional PostgreSQL administration created an artificial binary choice: either users had basic access with limited capabilities, or they required full superuser privileges for operational tasks. This forced many organizations to grant excessive permissions simply to perform routine operations like monitoring, backups, or maintenance.</p> Predefined roles break this limitation by allowing more granular control over operational tasks without distributing superuser privileges. Instead of having a small number of highly privileged superusers handling all administrative work, organizations can now delegate specific capabilities to appropriate operational roles - monitoring teams get monitoring access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need.</p> The deeper benefit is abstraction</strong>. Not only do you switch from manually managing permissions on individual system objects, you switch to managing logical capability sets. This creates a clear separation between what</em> you want to allow and how</em> it's actually implemented.</p> The advantage is clearly demonstrated in the following example</p> -- granting select on each schema separately</span></span> GRANT</span> USAGE </span>ON SCHEMA</span> finance, hr, app, </span>audit TO</span> analytics_team;</span></span> GRANT SELECT ON</span> ALL TABLES </span>IN SCHEMA</span> finance </span>TO</span> analytics_team;</span></span> GRANT SELECT ON</span> ALL SEQUENCES </span>IN SCHEMA</span> finance </span>TO</span> analytics_team;</span></span> </span> -- versus one time setup</span></span> GRANT</span> pg_read_all_data </span>TO</span> analytics_team;</span></span></code></pre> The predefined roles approach automatically covers:</p> All current tables, views and sequences</li> All schemas</li> Future objects created after the GRANT</code></li> </ul> And the good news is the benefits don't end here. The real thing on operational level comes with the scope of those predefined roles - they apply at cluster level</strong> (instead on database level). Only notable exception is role pg_database_owner</code> which we will cover next.</p> Before going there, let's briefly mention second benefit, and that's the fact any new operational features in PostgreSQL will be covered by those predefined roles automatically.</p> The Evolution of Predefined Roles</a> </h2> Understanding when and why predefined roles were introduced provides important insight into PostgreSQL's operational priorities and helps explain some of the design decisions. The journey from a single role in PostgreSQL 9.6 to today's comprehensive set reflects real-world pain points that PostgreSQL developers encountered in production environments.</p> PostgreSQL 9.6 (2016)</strong> introduced first predefined role pg_signal_backend</code> to address common frustration - inability to cancel running query without giving away superuser rights.</p> -- This became possible in 9.6 without superuser privileges</span></span> SELECT</span> pg_cancel_backend(</span>12345</span>); </span>-- Cancel a query</span></span> SELECT</span> pg_terminate_backend(</span>12345</span>); </span>-- Terminate a session</span></span></code></pre> PostgreSQL 10 (2017)</strong> brought biggest expansion of predefined roles with, adding four monitoring specific roles to improve database observability, and support growing importance of database monitoring in production environments.</p> -- Create the monitoring user</span></span> CREATE USER</span> postgres_exporter</span> WITH PASSWORD</span> 'monitoring_password'</span>;</span></span> </span> -- Grant comprehensive monitoring access</span></span> GRANT</span> pg_monitor </span>TO</span> postgres_exporter;</span></span></code></pre> Giving access to variety of metrics and settings. With single GRANT</code> you can access data:</p> From pg_read_all_settings (inherited via pg_monitor) to SELECT * FROM pg_settings</code> - Configuration parameters for alerting on misconfigurations and configuration metrics like shared_buffers, work_mem, max_connections</li> </ul> From pg_read_all_stats (inherited via pg_monitor):</p> -- Database-wide statistics (connections, transactions, blocks read/written)</span></span> SELECT * FROM</span> pg_stat_database</span></span> -- Current query activity and connection states</span></span> SELECT * FROM</span> pg_stat_activity</span></span> -- Replication lag and status</span></span> SELECT * FROM</span> pg_stat_replication</span></span> -- Background writer performance</span></span> SELECT * FROM</span> pg_stat_bgwriter</span></span> -- Lock contention monitoring</span></span> SELECT * FROM</span> pg_locks</span></span></code></pre> From pg_stat_scan_tables (inherited via pg_monitor):</p> Enhanced table statistics collection with sequential scan information</li> More detailed I/O statistics for table access patterns</li> </ul> The pg_monitor</code> role is particularly clever in its design. Rather than being a single monolithic permission set, it's actually a composite of the other three roles. This allows for granular access when needed - you can grant pg_read_all_stats for basic monitoring without including configuration access, or grant the full pg_monitor bundle for comprehensive monitoring capabilities.</p> PostgreSQL 11 (2018)</strong> laid the foundation for secure file system operations with the introduction of three powerful roles that fundamentally changed how PostgreSQL handles server-side file access:</p> pg_read_server_files</code></li> pg_write_server_files</code></li> pg_execute_server_program</code></li> </ul> Addressing many long-standing issues with providing file system access, like ETL processes that needs to read CSV files and data processing pipelines often involving external programs.</p> The file system roles represented a significant security advancement. Before PostgreSQL 11, operations like server-side COPY FROM file were superuser-only, forcing administrators to either grant excessive privileges or find workarounds.</p> PostgreSQL 14 (2021)</strong> introduced three significant roles that addressed different data access patterns:</p> pg_read_all_data</code></li> pg_write_all_data</code></li> pg_database_owner</code></li> </ul> The first two solved a common backup and analytics challenge. The special case is pg_database_owner</code> which we will cover later.</p> -- create ETL extraction user</span></span> CREATE USER</span> etl_extractor</span> WITH PASSWORD</span> 'extract_password'</span>;</span></span> GRANT</span> pg_read_all_data </span>TO</span> etl_extractor;</span></span> </span> -- create ETL loader with write access</span></span> CREATE USER</span> etl_loader</span> WITH PASSWORD</span> 'loader_password'</span>;</span></span> GRANT</span> pg_write_all_data </span>TO</span> etl_loader;</span></span></code></pre> One caveat that comes with those ALL data roles is that they still (despite what the name might suggest) respect Row Level Security policies (RLS). This is a design choice of PostgreSQL which prioritises security by default, even for users with broad data access. The only way to avoid RLS is to explicitly grant the BYPASSRLS</code> role attribute.</p> PostgreSQL 15 (2022)</strong> expanded set with pg_checkpoint</code>, reflecting the reality that checkpoint operations are sometimes needed for maintenance but don't require full superuser privileges.</p> PostgreSQL 16 (2023)</strong> expanded the list by introduction of</p> pg_use_reserved_connections</code> - resolving a critical problem with high-traffic databases, where previously only superusers could access reserved connections. Introduction of this roles expanded the pool of potential users who might be able to resolve incidents or high-load situations. For example with pg_signal_backend</code> grant.</li> pg_create_subscription</code> - with ability to create Logical Replication</a> subscriptions and confirming the place of PostgreSQL in the distributed database scenarios.</li> </ul> PostgreSQL 17 (2024)</strong> finally saw pg_maintain</code>, role that had a turbulent development history (initially committed for PostgreSQL 16), allowing non-superusers to perform crucial maintenance tasks like VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE.</p> The pg_maintain</code> role is particularly valuable for:</p> Automated maintenance scripts that can now run with minimal privileges, reducing the security footprint of scheduled maintenance operations.</li> Operational teams managing multiple databases who need consistent maintenance capabilities without requiring superuser access to each database.</li> Cloud and managed environments where maintenance operations need to be delegated to operations staff without granting broader system access.</li> </ul> PostgreSQL 18 (2025)</strong> continues the trend with pg_signal_autovacuum_worker</code> expanding the ability for non-superusers to signal autovacuum workers specifically, enabling them to cancel vacuum operations on specific tables or terminate autovacuum sessions that may be causing performance issues during critical periods.</p> What "all data" actually means</a> </h2> Per Stephen Frost's introducing commit</a>, the design intent of pg_read_all_data</code> is "running pg_dump as a non-superuser without explicitly granting access to all tables, schemas, et al."</em> — and the role's reach follows from that, going beyond the "tables, views, sequences" the official documentation</a> calls out.</p> A complete pg_dumpall</code> includes more than application data: stored password material for every role, foreign-server credentials, subscription connection strings, and the contents of large objects. The role permits reading all of it, regardless of the ordinary REVOKEs and view-layer masks PostgreSQL applies elsewhere. This is intentional, since each is something a complete dump genuinely needs to ship.</p> pg_read_all_data</code> is closer to a "complete cluster dump" role than a "general read-only user" role.</strong> For analytics, BI, monitoring, or audit identities, narrower mechanisms (pg_monitor</code>, explicit per-schema GRANT SELECT</code>, application-specific roles) are usually what you actually want. </div> The Magic of pg_database_owner</a> </h2> If you paid attention, there's one role that stands apart from all others - pg_database_owner</code>. While it does not provide any specific privileges that wouldn't be possible without superuser access, it serves as a marker for the owner of the database. This role is crucial for maintaining database ownership and ensuring that the database is managed by the correct user.</p> Before PostgreSQL 15 you would have public</code> schema owned by postgres</code> user.</p> demo=> \dn</span></span> List of schemas</span></span> Name \| Owner</span></span> --------+----------</span></span> public \| postgres</span></span></code></pre> It came with the baggage of relaxed default permission, allowing every user (PUBLIC) to create objects. Making this a maintenance nightmare.</p> Starting with PostgreSQL 15 and above, the public</code> schema is owned by pg_database_owner</code> role.</p> demo=# \dn</span></span> List of schemas</span></span> Name \| Owner</span></span> --------+-------------------</span></span> public \| pg_database_owner</span></span></code></pre> Unlike any other roles, pg_database_owner</code> has a unique behaviour - it membership changes with the current database. As the code snippet above shows the "shape-shifting nature" of this role fixed the public</code> schema and complexity of the permissions associated with it.</p> Another special behaviour is that the role itself does not come with any permissions. Let that sink in - zero permissions. Its power only comes from what you grant it or inherit. It opens up system for elaborate template inheritance</em>.</p> -- define functionality in template1</span></span> \c template1 postgres</span></span> CREATE OR REPLACE FUNCTION</span> db_owner_stats</span>()</span> RETURN</span> ... </span>SECURITY</span> DEFINER;</span></span> GRANT EXECUTE ON FUNCTION</span> db_owner_stats()</span> TO</span> pg_database_owner;</span></span> </span> -- and let it automatically apply to all new databases</span></span> CREATE DATABASE</span> app_prod</span> OWNER</span> first_app;</span></span> CREATE DATABASE</span> demo_prod</span> OWNER</span> second_app;</span></span> </span> -- both first_app and second_app automatically inherit db_owner_stats function on their databases</span></span></code></pre> And when we are talking about magic, the special status of the role prevents this role to be granted to any other role.</p> demo</span>=</span>#</span> GRANT</span> pg_database_owner </span>TO</span> labs_app;</span></span> ERROR: </span>role</span> "pg_database_owner"</span> cannot have </span>explicit</span> members</span></span></code></pre>Conclusion</a> </h2> Predefined roles transform PostgreSQL administration from ad-hoc permission management into systematic capability delegation. They solve the fundamental problem of needing operational access without operational trust.</p> The evolution continues - each PostgreSQL release adds new predefined roles addressing real-world operational challenges. The pattern is established: identify common pain points, create focused roles, eliminate the need for superuser privileges in routine operations.</p> The next time you reach for superuser access to solve an operational problem, check if PostgreSQL has already provided a predefined role for exactly that purpose. You'll likely find it already has one.</p> Update, 27 May 2026: Added What "all data" actually means</a> on pg_read_all_data</code>'s reach beyond tables, views, and sequences.</em></p> Beyond the Basics of Logical Replication 2025-06-24T07:10:00+00:00 With First Steps with Logical Replication</a> we set up a basic working replication between a publisher and a subscriber and were introduced to the fundamental concepts. This part covers the operational side: initial data copies, monitoring, evolving publications, and how logical decoding actually works.</p> Initial Data Copy</a> </h2> As we demonstrated in the first part, when setting up the subscriber, you can opt out of the initial data copy with WITH (copy_data = false)</code>. The default copy is convenient, but it has characteristics you should understand before relying on it in a production environment.</p> The mechanism effectively asks the publisher to copy the table data by taking a snapshot (courtesy of MVCC), sending it to the subscriber, and thanks to the replication slot "bookmark," seamlessly continues streaming the changes from the point the snapshot was taken.</p> Simplicity</strong> is the key feature here, as a single command handles the snapshot, transfer, and transition to ongoing streaming.</p> The trade-off is performance</strong>, because each table is copied by a single worker using COPY</code>. Small tables finish quickly, but tables with gigabytes of data will take noticeably long.</p> Although parallelism can be controlled by the max_sync_workers_per_subscription</code> configuration parameter, it still might leave you waiting for hours (and days) for any real-life database to get replicated. You can monitor whether the tables have already been synchronized or are still waiting/in progress using the pg_subscription_rel</code> catalog.</p> SELECT</span> srrelid::regclass </span>AS</span> table_name, srsubstate</span></span> FROM</span> pg_subscription_rel;</span></span></code></pre> Each table will have one of the following states:</p> i</code> not yet started</li> d</code> copy is in progress</li> f</code> finished the table copy</li> s</code> synchronized with the main apply stream</li> r</code> ready, replicating normally</li> </ul> Tables reach r</code> independently, so changes stream to already-synchronized tables even while others are still copying. Nevertheless, the replication slot retains the LSN position</strong>, which means that the publisher will retain WAL files</strong> until the subscriber has caught up.</p> Manual Synchronization for Production Workloads</a> </h2> As mentioned above, the implicit copy might bring performance trade-offs that are simply too much for production use, unless you consider it while designing your logical replication topology. In all other scenarios, going with manual synchronization is the way forward.</p> The entire process has one leading principle: the data you load manually must be consistent with the point in time from which the logical replication stream begins. This is achieved by creating a logical replication slot before (or synchronized with) the point at which you restore the data (PITR)</strong> and enabling it once the data is transferred. Only this will allow you to correctly apply all subsequent changes on the subscriber.</p> There are several ways to achieve this, and you will have to evaluate the mechanism based on the available constraints in your particular use case:</p> Use the backup & restore mechanism that supports Point-in-Time Recovery (PITR) if the amount of data to be synchronized approaches the total size of the backed-up data (i.e., most of the tables).</li> Orchestrate the data change ingestion on the publisher with the backup, by stopping the incoming changes (for example, during a planned maintenance window) and only restoring them when a consistent snapshot is available to a known LSN. This is for cases where you can control table ingestion and the amount of data being transferred fits the maintenance window.</li> If none of this is available, you might be able to manually advance replication slots to a predefined LSN. This is a last resort; advancing slots by hand is easy to get wrong.</li> </ol> The recommended way for the most reliable backup and restore handling is the use of pgBackRest</strong>, as it allows you to restore a particular backup in advance and apply only changes needed later to advance to the selected Point-in-Time, hence significantly reducing the time required for the initial sync. On the other hand, if you are only using a small portion of the data within the backup on your subscriber, it might create resource constraints not acceptable for the setup.</p> While pg_dump can't be considered a reliable backup tool</strong>, it can handle a consistent copy of a small subset of tables. The catch is where the snapshot comes from: a snapshot exported with pg_export_snapshot()</code> is not synchronized with a replication slot's start point. To get a slot-consistent snapshot, create the slot over a replication connection with CREATE_REPLICATION_SLOT ... EXPORT_SNAPSHOT</code>, keep that session open, and pass the returned snapshot name to pg_dump</code>:</p> pg_dump -d publisher_db --snapshot="00000004-000004A2-1" --data-only -Fc -f initial_data.dump</span></span></code></pre> The topic of consistent backups and Point-in-Time Recovery is beyond the scope of this guide.</p> Monitoring Logical Replication</a> </h2> Once you set up your publishers and subscribers and have the initial data in place, it's time to start thinking about how to keep that process running.</p> For day-to-day operations, the basic building block for monitoring is the pg_replication_slots</code> catalog. Sample query:</p> SELECT</span></span> slot_name,</span></span> plugin,</span></span> slot_type,</span></span> active,</span></span> pg_size_pretty(</span></span> pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)</span></span> ) </span>AS</span> wal_retained_size</span></span> FROM</span> pg_replication_slots</span></span> WHERE</span></span> slot_type </span>=</span> 'logical'</span>;</span></span></code></pre> giving you a result like:</p> -[ RECORD 1 ]-----+-----------------------------</span></span> slot_name \| my_sample_subscription</span></span> plugin \| pgoutput</span></span> slot_type \| logical</span></span> active \| t</span></span> wal_retained_size \| 49 MB</span></span></code></pre> You should look for two things:</p> active</code> indicates whether the slot is being used by any connection. An inactive slot is acceptable for short durations, but a slot that stays inactive is a sign that the subscriber is down or not configured properly.</li> The computed wal_retained_size</code> is a critical metric. An increasing value indicates problems consuming changes by the subscriber, and your cluster might be at risk of running out of disk space.</li> </ul> The active</code> flag also catches dangling slots, where the subscriber has been destroyed (or simply wiped) without properly dropping the subscription, leaving the replication slot behind. In such cases, the only direct response is dropping the slot by its name:</p> SELECT</span> pg_drop_replication_slot(</span>'my_dangling_slot'</span>);</span></span></code></pre> If you want to dig deeper, you can also use pg_stat_replication</code> that provides more (albeit volatile) data about the replication status. It can give you other metrics, like replay_lag</code>.</p> To wrap up the monitoring part, you can either design your custom checks or use a modified version of the query above:</p> SELECT</span></span> slot_name,</span></span> CASE WHEN</span> active</span></span> THEN</span> 1</span></span> ELSE</span> 0</span></span> END as</span> active_status,</span></span> pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) </span>as</span> wal_bytes_retained,</span></span> extract(epoch </span>from now</span>()) </span>as time</span></span> FROM</span> pg_replication_slots</span></span> WHERE</span> slot_type </span>=</span> 'logical'</span>;</span></span></code></pre> as part of a Grafana alerting rule:</p> Alert Rule</span>:</span> "Inactive Replication Slot"</span></span> Condition</span>:</span></span> -</span> Query</span>:</span> active_status</span></span> -</span> Reducer</span>:</span> last()</span></span> -</span> Evaluator</span>:</span> below 1</span></span> -</span> For</span>:</span> 30m</span> # if inactive for 30+ minutes</span></span> </span> Additional conditions (AND)</span>:</span></span> -</span> Query</span>:</span> wal_bytes_retained</span></span> -</span> Reducer</span>:</span> last()</span></span> -</span> Evaluator</span>:</span> above 1073741824</span> # 1GB limit</span></span> </span></code></pre> Nevertheless, you need to evaluate the monitoring details applicable for your particular use case(s). There might also be other options, for example, if your architecture allows for it, to use max_slot_wal_keep_size</a> and mark the replication slot invalid and release the WAL files when a particular slot falls behind the configured value (for example, 100GB). While this might sound reasonable, you must consider whether your application or use case can recover from such data loss.</p> Evolving Publications</a> </h2> Going back to the first part of our journey into logical replication, there's an important distinction to make. While we have described the initial use case, in real life your publisher schema will change as time goes by. Tables will be created and included implicitly (via FOR ALL TABLES</code>) or explicitly added/removed.</p> ALTER</span> PUBLICATION my_publication </span>ADD TABLE</span> public</span>.</span>new_table</span>;</span></span> ALTER</span> PUBLICATION my_publication</span> DROP TABLE</span> public</span>.old_table;</span></span></code></pre> Adding a new table to the publication changes nothing for an existing subscription on its own. The only way for the subscriber to reflect the changes in the underlying configuration is to refresh the subscription</strong>. It's a process where PostgreSQL compares the current subscription table list with the publication table list, (if configured to do so) starts the initial synchronization process and once finished goes to apply streamed changes.</p> As mentioned, you can refresh a subscription with or without copying the initial data. The copy_data</code> option controls this behavior and is currently the only supported setting.</p> ALTER</span> SUBSCRIPTION my_subscription</span></span> REFRESH PUBLICATION;</span></span> </span> ALTER</span> SUBSCRIPTION my_subscription</span></span> REFRESH PUBLICATION </span>WITH</span> (copy_data </span>=</span> false);</span></span></code></pre> When you remove a table from the publisher, the situation is much simpler as the changes for that particular table are no longer considered for logical decoding.</p> But what happens if you add a new table to the publication without refreshing the subscription? Even though the WAL includes changes for all tables and logical decoding processes all published table changes</strong>, there is no additional WAL retention to consider. The subscriber advances confirmed_flush_lsn</code> as before: the initial state for the newly added table has not yet been recorded, so its streamed changes are (for the moment) ignored.</p> Logical Decoding</a> </h2> If you paid attention throughout our journey or already worked a bit with logical replication, you might have come across something called the pgoutput</code> plugin. This is the built-in default mechanism responsible for logical decoding, the process that goes through the WAL stream and transforms it into a higher-level format.</p> Where physical replication ships "byte X at offset Y, in page Z", logical decoding translates the WAL stream into row-level changes. It uses the catalog of the source database to map object identifiers to schema, table, and column names, along with the respective values (both old and new), and reassembles the changes into the correct transaction flow.</p> The plugin's job is to format that output in a particular way and by itself is not aware of:</p> any transport layer details</li> replication slots</li> any retry logic</li> </ul> The built-in plugins are just a start, and developers can create custom output plugins. We can also look inside the decoded stream manually.</p> Consider this example:</p> -- create new publication</span></span> CREATE</span> PUBLICATION all_data </span>FOR</span> ALL TABLES;</span></span> </span> -- create new replication slot with pgoutput plugin</span></span> SELECT</span> pg_create_logical_replication_slot(</span>'plugin_demo'</span>, </span>'pgoutput'</span>);</span></span> </span> -- sample table</span></span> CREATE TABLE</span> IF</span> NOT EXISTS</span> demo_table (</span></span> id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT</span>,</span></span> email </span>TEXT</span></span> );</span></span> </span> -- generate some changes with insert/update/delete</span></span> INSERT INTO</span> demo_table (</span>name</span>, email) </span>VALUES</span> (</span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 1</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 1</span>;</span></span> </span> -- get binary changes</span></span> SELECT * FROM</span> pg_logical_slot_get_binary_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>, </span>'proto_version'</span>, </span>'1'</span>, </span>'publication_names'</span>, </span>'all_data'</span>);</span></span> </span> -- drop the replication slot and the publication</span></span> SELECT</span> pg_drop_replication_slot(</span>'plugin_demo'</span>);</span></span> DROP TABLE</span> demo_table;</span></span> DROP</span> PUBLICATION all_data;</span></span></code></pre> Giving us a peek into the recently performed changes (output is shortened):</p> lsn \| xid \| data</span></span> ------------+------+---------------------------------</span></span> 1/D6997380 \| 1038 \| \x4200000001d69974180</span></span> 1/D6997380 \| 1038 \| \x52000042057075626c696300646...</span></span> 1/D6997380 \| 1038 \| \x49000042054e000374000...</span></span> 1/D6997448 \| 1038 \| \x430000000001d6997...</span></span> 1/D6997448 \| 1039 \| \x4200000001d6997...</span></span> 1/D6997448 \| 1039 \| \x49000042054e0....</span></span> 1/D6997510 \| 1039 \| \x430000000001d6...</span></span></code></pre> This is correct, but won't help us demonstrate the flow: pgoutput</code> is the plugin used for actual replication, and its output is binary. PostgreSQL also ships the test_decoding</code> plugin, which decodes the changes into a human-readable text representation.</p> -- create replication slot with test_decoding plugin</span></span> -- (unlike pgoutput, test_decoding doesn't use publications, so none is needed)</span></span> SELECT</span> pg_create_logical_replication_slot(</span>'plugin_demo'</span>, </span>'test_decoding'</span>);</span></span> </span> CREATE TABLE</span> IF</span> NOT EXISTS</span> demo_table (</span></span> id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT</span>,</span></span> email </span>TEXT</span></span> );</span></span> </span> -- generate some changes with insert/update/delete</span></span> INSERT INTO</span> demo_table (</span>name</span>, email) </span>VALUES</span> (</span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 1</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 1</span>;</span></span> </span> -- get human readable changes</span></span> SELECT * FROM</span> pg_logical_slot_get_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>);</span></span> </span> -- drop the replication slot and the table</span></span> SELECT</span> pg_drop_replication_slot(</span>'plugin_demo'</span>);</span></span> DROP TABLE</span> demo_table;</span></span></code></pre> This time the output is actually going to be helpful.</p> lsn \| xid \| data</span></span> ------------+------+-----------------------------------------------------------------------------------------------------</span></span> 1/D69CCC30 \| 1048 \| BEGIN 1048</span></span> 1/D69CCC98 \| 1048 \| table public.demo_table: INSERT: id[integer]:1 name[text]:'John Doe' email[text]:'john@example.com'</span></span> 1/D69CCDC0 \| 1048 \| COMMIT 1048</span></span> 1/D69CCDC0 \| 1049 \| BEGIN 1049</span></span> 1/D69CCDC0 \| 1049 \| table public.demo_table: UPDATE: id[integer]:1 name[text]:'John Doe' email[text]:'doe@example.com'</span></span> 1/D69CCE50 \| 1049 \| COMMIT 1049</span></span> 1/D69CCE50 \| 1050 \| BEGIN 1050</span></span> 1/D69CCE50 \| 1050 \| table public.demo_table: DELETE: id[integer]:1</span></span> 1/D69CCEC0 \| 1050 \| COMMIT 1050</span></span></code></pre> Just reading it, you can easily follow the sequence of the transaction (auto-commit in psql) and track individual changes.</p> We can also reiterate the importance of REPLICA IDENTITY</code> with this example. The id</code> column of demo_table</code> is declared PRIMARY KEY</code>, and the default replica identity uses exactly that to identify rows. Switching the table to REPLICA IDENTITY FULL</code> shows, via test_decoding</code>, how much more verbose the decoded stream becomes.</p> (This continues the previous example; setup and teardown omitted.)</p> ALTER TABLE</span> demo_table </span>REPLICA IDENTITY</span> FULL;</span></span> </span> INSERT INTO</span> demo_table (id, </span>name</span>, email) OVERRIDING </span>SYSTEM VALUE VALUES</span> (</span>999</span>, </span>'John Doe'</span>, </span>'john@example.com'</span>);</span></span> UPDATE</span> demo_table </span>SET</span> email </span>=</span> 'doe@example.com'</span> WHERE</span> id </span>=</span> 999</span>;</span></span> DELETE FROM</span> demo_table </span>WHERE</span> id </span>=</span> 999</span>;</span></span> </span> SELECT * FROM</span> pg_logical_slot_get_changes(</span>'plugin_demo'</span>, </span>NULL</span>, </span>NULL</span>);</span></span></code></pre> And you can observe the changes. While by its nature INSERT</code> gives you the same output (all values are being sent), UPDATE</code> and DELETE</code> are no longer relying on the primary key. Instead, they have to provide all values, for the UPDATE</code> case both old and new data:</p> lsn \| 1/D69F8770</span></span> xid \| 1062</span></span> data \| table public.demo_table: UPDATE: old-key: id[integer]:999 name[text]:'John Doe' email[text]:'john@example.com' new-tuple: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</span></span></code></pre> and for DELETE</code> the old data.</p> lsn \| 1/D69F8828</span></span> xid \| 1063</span></span> data \| table public.demo_table: DELETE: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</span></span></code></pre> It's obvious how a misconfigured replica identity increases the amount of data decoded and replicated.</p> Fine-grained Publication Control</a> </h2> Before we introduce the advanced replication topologies in later parts of this series, it's worth knowing how precisely you can control what data reaches the subscribers. Filtering what you publish cuts replication overhead and network traffic, and keeps data the subscriber shouldn't see out of the stream.</p> The first option to filter the data is by explicit column_list</code>, allowing you to exclude sensitive (PII or similar) or unnecessary data. The column list must include the replica identity columns (typically the primary key) if the publication replicates UPDATE</code> or DELETE</code>; otherwise those operations fail on the publisher.</p> CREATE</span> PUBLICATION hr_analytics</span></span> FOR TABLE</span> hr</span>.</span>employees</span> (employee_id, first_name, last_name, department, </span>start_date</span>);</span></span></code></pre> Similar to column selection, you can control what operations a publication replicates. The available options are 'insert', 'update', 'delete', and 'truncate'.</p> CREATE</span> PUBLICATION hr_analytics</span></span> FOR TABLE</span> hr</span>.</span>employees</span> (employee_id, first_name, last_name, department, </span>start_date</span>)</span></span> WITH</span> (publish </span>=</span> 'insert,delete'</span>);</span></span></code></pre>Securing Logical Replication</a> </h2> Limiting what a publication carries only helps if access is limited too. A restricted publication means little when the replication role can read (or another publication can expose) the same data without restrictions. So far in this series, we have relied for simplicity on SUPERUSER</code> access. To design logical replication for real-life production use, you need a least-privilege model.</p> The permissions split across two roles. Creating a publication requires the CREATE</code> privilege on the database, adding a table to one requires owning that table, and FOR ALL TABLES</code> publications require a superuser. The role a subscription connects as is a separate concern: it needs the REPLICATION</code> attribute, CONNECT</code> on the database, USAGE</code> on the schema, and SELECT</code> on the published tables (or columns) for the initial data copy.</p> Row-level security deserves explicit attention: if the replication role lacks SUPERUSER</code> and BYPASSRLS</code>, publisher row security policies can execute during the initial copy. If the role does not trust all table owners, add options=-crow_security=off</code> to the connection string; replication then halts on tables with a row security policy instead of executing it.</p> Using the fine-grained example used above, we can demonstrate the setup. First, creating a role with LOGIN</code> and REPLICATION</code> clauses and granting database and schema access:</p> CREATE ROLE</span> hr_analytics_role </span>WITH LOGIN</span> REPLICATION </span>PASSWORD</span> 'a_strong_password'</span>;</span></span> </span> GRANT CONNECT ON DATABASE</span> my_publisher_db </span>TO</span> hr_analytics_role;</span></span> GRANT</span> USAGE </span>ON SCHEMA</span> hr </span>TO</span> hr_analytics_role;</span></span></code></pre> The crucial step is to correctly define the GRANT</code> for selecting the data.</p> GRANT SELECT</span></span> (employee_id, first_name, last_name, department, </span>start_date</span>)</span></span> ON</span> hr</span>.</span>employees</span> TO</span> hr_analytics_role;</span></span></code></pre> This way, even if someone gains access to the replication role, they cannot read data beyond what was explicitly granted.</p> First steps with Logical Replication in PostgreSQL 2025-06-11T00:00:00+00:00 PostgreSQL's logical replication streams row-level changes</strong> from one PostgreSQL instance to another using a publish-subscribe</strong> model. Unlike physical replication, which copies the whole cluster byte for byte, it lets you pick what to replicate and where to send it, which makes it a building block for scaling out, distributing load, and integrating PostgreSQL with the rest of your architecture.</p> Physical vs. Logical Replication</a> </h2> Both kinds of replication in PostgreSQL are built on top of the Write-Ahead Log (WAL)</strong>.</p> The WAL is a sequential, append-only log that records every change made to the cluster data. For durability purposes, all modifications are first written to the WAL before the modified data pages are flushed to disk. This allows PostgreSQL to recover from crashes by replaying logged changes.</p> Versioned changes, necessitated by concurrent transactions, are managed through Multi-Version Concurrency Control (MVCC)</strong>. Instead of overwriting data directly, MVCC creates multiple versions of rows, allowing each transaction to see a consistent snapshot of the database. It is the WAL that captures these versioned changes along with the transactional metadata to ensure data consistency at any given point in time.</p> Physical replication</strong> is built directly on the Write-Ahead Log. It enables streaming of the binary WAL data from the primary server to one or more standby servers (replicas), effectively creating a byte-for-byte copy of the entire cluster. This requirement makes the replicas read-only, making them ideal candidates for failover or scaling purposes.</p> Logical replication</strong>, while also built on top of the WAL data, takes a fundamentally different approach. Instead of streaming raw change data, it decodes the WAL into logical, row-level changes</strong> (INSERT</code>, UPDATE</code>, and DELETE</code>) and only then sends them to the subscribers using a publish-subscribe model. This allows selective replication and writable subscribers that are not strictly tied to a single publisher, at a cost: logical replication does not replicate DDL changes</strong>.</p> </th> Physical</th> Logical</th></tr></thead> Data Streamed</strong></td> Binary WAL segments</td> Row-level SQL changes</td></tr> Scope</strong></td> Byte-for-byte stream</td> Selected tables</td></tr> Node Type</strong></td> Read-only standby</td> Fully writable instance</td></tr> PostgreSQL Version</strong></td> All servers must match major version</td> Supports across versions</td></tr> Database Schema</strong></td> Changes automatically replicated</td> Changes must be applied on subscriber(s) separately</td></tr> Use Case</strong></td> Failover, high-availability, and read scaling</td> Integration, zero-downtime upgrades and schema migrations, complex topologies</td></tr> </tbody></table> Physical replication is your go-to for high availability</strong> and disaster recovery</strong>, where you want a fast, exact copy of the entire database cluster that can take over in case of failure. It’s simple to set up and very efficient but limited in flexibility.</p> Logical replication fits when you need fine-grained control over what data is replicated, require writable replicas, or want to integrate PostgreSQL with other systems or versions. Typical uses are zero-downtime major-version upgrades, multi-region deployments, and feeding downstream systems such as a search index or a warehouse.</p> Setting up Logical Replication</a> </h2> Enough of boring theory for now. To get started, you will need two instances of PostgreSQL, whether you provision two virtual machines or two clusters running on the same computer. We will call one publisher</strong> and the other subscriber</strong>.</p> Preparing the Publisher</a> </h3> First, you need to prepare the publisher to emit the logical changes. You will need to modify postgresql.conf</code> (or add particular conf.d</code> configuration) with the following parameters:</p> wal_level = logical</span></span> max_replication_slots = 10</span></span> max_wal_senders = 10</span></span></code></pre> Your publisher also needs to be reachable by the subscriber, in most cases via a TCP socket.</p> listen_addresses = ''</span></span></code></pre> Here, the configuration is:</p> wal_level</code></a> set to logical</code> is a crucial piece of the configuration. It tells PostgreSQL how much information to write to the WAL, in this case, to support logical decoding.</li> max_replication_slots</code></a> defines the maximum number of replication slots that can be created on the server. Each logical replication subscription needs its own slot.</li> max_wal_senders</code></a> should be set high enough to accommodate all expected concurrent replication connections from subscribers, and should be higher than or equal to max_replication_slots</code>. Each active replication subscription consumes a wal_sender slot.</li> </ul> You will also need to configure client authentication in pg_hba.conf</code> to allow the subscriber to connect (for simplification, we allow all users). Note that the replication</code> keyword in pg_hba.conf</code> matches only physical replication connections; a logical replication subscriber connects to a regular database, so the entry must name that database (or all</code>):</p> # TYPE DATABASE USER ADDRESS METHOD</span></span> host logical_demo_publisher all subscriber_ip/32 scram-sha-256</span></span></code></pre> While for the purposes of the article we will assume the use of a superuser, making it convenient:</p> CREATE USER</span> my_user_name</span> SUPERUSER </span>PASSWORD</span> 'my_secure_password'</span>;</span></span></code></pre> It is recommended to use a dedicated replication user for real-life deployments:</p> CREATE USER</span> replication_user</span> WITH</span> REPLICATION </span>ENCRYPTED PASSWORD</span> 'my_secure_password'</span>;</span></span></code></pre> Once configured, restart PostgreSQL for the configuration changes to take effect. After that, connect to the server and prepare the environment.</p> -- example for psql</span></span> </span> CREATE DATABASE</span> logical_demo_publisher</span>;</span></span> \c logical_demo_publisher;</span></span></code></pre> Create a sample table and seed initial data:</p> CREATE TABLE</span> products</span> (</span></span> id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> category </span>TEXT</span>,</span></span> price </span>DECIMAL</span>(</span>10</span>, </span>2</span>) </span>NOT NULL</span>,</span></span> stock_quantity </span>INT DEFAULT</span> 0</span>,</span></span> created_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> updated_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> description TEXT</span>,</span></span> is_active </span>BOOLEAN DEFAULT</span> TRUE</span></span> );</span></span> </span> -- seed some data (10 records)</span></span> INSERT INTO</span> products (</span></span> name</span>,</span></span> category,</span></span> price,</span></span> stock_quantity,</span></span> description</span>,</span></span> is_active</span></span> )</span></span> SELECT</span></span> 'Product Batch '</span> \|\|</span> s</span>.</span>id</span> AS name</span>,</span></span> CASE</span> (</span>s</span>.</span>id</span> % </span>5</span>)</span></span> WHEN</span> 0</span> THEN</span> 'Electronics'</span></span> WHEN</span> 1</span> THEN</span> 'Books'</span></span> WHEN</span> 2</span> THEN</span> 'Home Goods'</span></span> WHEN</span> 3</span> THEN</span> 'Apparel'</span></span> ELSE</span> 'Miscellaneous'</span></span> END AS</span> category,</span></span> ROUND</span>((RANDOM()</span> </span> 500</span> +</span> 10</span>)::</span>numeric</span>, </span>2</span>) </span>AS</span> price,</span></span> FLOOR</span>(RANDOM()</span> </span> 200</span>)::</span>int AS</span> stock_quantity,</span></span> 'Auto-generated description for product ID '</span> \|\|</span> s</span>.</span>id</span> \|\|</span> '. Lorem ipsum dolor sit amet, consectetur adipiscing elit.'</span> AS description</span>,</span></span> (</span>s</span>.</span>id</span> % </span>10</span> <></span> 0</span>) </span>AS</span> is_active</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10</span>) </span>AS</span> s(id);</span></span></code></pre> The final step for the publisher</strong> is to create a publication to define what data we want to publish.</p> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products;</span></span></code></pre>Preparing the Subscriber</a> </h3> Next, we will use our subscriber</strong> instance to receive the logical changes. There's no need to make any configuration changes just for subscribing, as it's not emitting changes itself.</p> Create the target database and schema. This step matters: logical replication does not create tables on the subscriber, so the target table must already exist and match the published one.</p> -- example for psql</span></span> </span> CREATE DATABASE</span> my_subscriber_db</span>;</span></span> \c my_subscriber_db;</span></span> </span> CREATE TABLE</span> products</span> (</span></span> id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> category </span>TEXT</span>,</span></span> price </span>DECIMAL</span>(</span>10</span>, </span>2</span>) </span>NOT NULL</span>,</span></span> stock_quantity </span>INT DEFAULT</span> 0</span>,</span></span> created_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> updated_at </span>TIMESTAMP WITH TIME ZONE DEFAULT</span> CURRENT_TIMESTAMP,</span></span> description TEXT</span>,</span></span> is_active </span>BOOLEAN DEFAULT</span> TRUE</span></span> );</span></span></code></pre> Now we have the foundation for testing logical replication. The simplest way is to create a subscription using connection details and the name of the publication to subscribe to.</p> CREATE</span> SUBSCRIPTION my_subscription</span></span> CONNECTION</span> 'host=publisher_ip_address port=5432 user=your_replication_user password=my_secure_password dbname=logical_demo_publisher'</span></span> PUBLICATION my_publication;</span></span></code></pre> my_subscription</code> should be a descriptive name for your subscription.</li> CONNECTION</code> defines how to connect to the publisher (which is a regular connection string and could also be a service</a>).</li> PUBLICATION</code> specifies the name of the publication you created earlier on the publisher.</li> </ul> If your connection was correct, the subscription will by default start with the initial data sync and listen for incoming changes. If you have used the queries above, you can validate that the data is now on the subscriber.</p> #</span> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> products;</span></span> count</span></span> ---------</span></span> 10</span></span> (</span>1</span> row</span>)</span></span></code></pre> The number of records should match the number of records created using generate_series</code> above (10 in our example). You can go ahead and insert a single row again on your publisher instance and validate the data being replicated to the subscriber.</p> That's your first logical replication setup working.</p> Core Concepts of Logical Replication</a> </h2> With a working setup in place, here are the moving parts that made it work.</p> Publication</a> </h3> As the name implies, a publication</strong> is where it all starts. It's essentially a catalogue of data you offer from the publisher. You can publish</a> a number of objects:</p> --- specific tables</span></span> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products, orders;</span></span> </span> --- everything in your database (make sure you really want to do this)</span></span> CREATE</span> PUBLICATION all_data </span>FOR</span> ALL TABLES;</span></span> </span> --- replicate specific columns only (PostgreSQL 15 and higher)</span></span> CREATE</span> PUBLICATION generic_data </span>FOR TABLE</span> products (id, </span>name</span>, price);</span></span> </span> --- filter published data (PostgreSQL 15 and higher)</span></span> CREATE</span> PUBLICATION active_products </span>FOR TABLE</span> products </span>WHERE</span> (is_active </span>=</span> true);</span></span> </span> --- filter different operations</span></span> CREATE</span> PUBLICATION my_publication </span>FOR TABLE</span> products, orders</span></span> WITH</span> (publish </span>=</span> 'insert'</span>);</span></span> </span> --- or you can mix & match it to get exactly what you need</span></span> CREATE</span> PUBLICATION eu_customers </span>FOR</span></span> TABLE</span> customers (id, email, country, created_at)</span></span> WHERE</span> (country </span>IN</span> (</span>'DE'</span>, </span>'FR'</span>, </span>'IT'</span>)),</span></span> TABLE</span> orders (id, customer_id, total_amount)</span></span> WHERE</span> (total_amount </span>></span> 100</span>)</span></span> WITH</span> (publish </span>=</span> 'insert, update'</span>);</span></span></code></pre> As you can see, logical replication really gives you quite a lot of options and is far from the rigid, byte-for-byte copying</strong> of physical replication. This is exactly the characteristic that allows you to build complex topologies.</p> Once you set up your publication(s), you can review it:</p> --- in psql</span></span> \dRp</span></span> \dRp+ my_publication</span></span> </span> --- using pg_catalog</span></span> SELECT FROM pg_publication_tables WHERE pubname = 'my_publication';</span></span></code></pre> From these options, it's easy to think of a publication as a customisable data feed.</p> Subscription</a> </h3> The other side of logical replication is a subscription</strong>. It defines what and how to consume events from a publisher. You subscribe to a publication using connection details and a number of options.</p> --- basic subscription with full connection detail</span></span> CREATE</span> SUBSCRIPTION my_subscription</span></span> CONNECTION</span> 'host=publisher_host port=5432 user=repl_user password=secret dbname=source_db'</span></span> PUBLICATION my_publication;</span></span> </span> -- subscription using service definition</span></span> CREATE</span> SUBSCRIPTION realtime_only</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication;</span></span></code></pre> The default behaviour of the subscription is to copy existing data</strong>, start immediately</strong>, and continue with streaming data changes. Once you start experimenting with logical replication, you can control that behaviour.</p> --- subscribe without initial copy of the data</span></span> CREATE</span> SUBSCRIPTION streaming_only</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication</span></span> WITH</span> (copy_data </span>=</span> false);</span></span> </span> --- or defer the start for later</span></span> CREATE</span> SUBSCRIPTION manual_start</span></span> CONNECTION</span> 'service=my_source_db'</span></span> PUBLICATION my_publication</span></span> WITH</span> (</span>enabled =</span> false);</span></span></code></pre> You can monitor your subscriptions:</p> --- in psql</span></span> \dRs</span></span> \dRs+ my_subscription</span></span> </span> --- or their status using pg_catalog</span></span> SELECT subname, received_lsn, latest_end_lsn, latest_end_time</span></span> FROM pg_stat_subscription;</span></span></code></pre>Replication Slot</a> </h3> Publications and subscriptions establish the data flow, but there's a critical piece missing: how does the publisher keep track of multiple subscribers reading the WAL at different speeds? Replication slots</strong> are the answer. They act as a persistent bookmark in the WAL stream that tracks exactly where each subscriber is (or was last time), ensuring no changes are lost</strong> even if the connection stops.</p> A replication slot can tell us quite a lot about itself.</p> #</span> SELECT * FROM</span> pg_replication_slots;</span></span> -</span>[ RECORD 1 ]</span>-------+---------------</span></span> slot_name \| my_subscription</span></span> plugin \| pgoutput</span></span> slot_type \| logical</span></span> datoid \| </span>16390</span></span> database</span> \| logical_demo_publisher</span></span> temporary \| f</span></span> active \| t</span></span> active_pid \| </span>3162</span></span> xmin \|</span></span> catalog_xmin \| </span>777</span></span> restart_lsn \| </span>0</span>/</span>1DEFBF0</span></span> confirmed_flush_lsn \| </span>0</span>/</span>1DEFC28</span></span> wal_status \| reserved</span></span> safe_wal_size \|</span></span> two_phase \| f</span></span> inactive_since \|</span></span> conflicting \| f</span></span> invalidation_reason \|</span></span> failover</span> \| f</span></span> synced \| f</span></span></code></pre> The most interesting attributes are:</p> slot_name</code></li> plugin</code> used for logical replication</li> slot_type</code> confirming it's for logical replication</li> active</code> indicates whether there's a subscriber reading from this slot</li> </ul> And most important of those being:</p> restart_lsn</code> identifying the LSN where this slot "holds" WAL files from being released</li> confirmed_flush_lsn</code> being the last LSN position the subscriber confirmed it has successfully processed (i.e., applied).</li> </ul> While we won't go into details about WAL, it's enough to say LSN (Log Sequence Number) is a unique address or position</strong> in the WAL that identifies the position in the stream of database changes.</p> In most cases, you don't have to create replication slots manually</strong>, as subscriptions create and manage them. The persistent nature of the slots guarantees they survive the restart of both publisher and subscriber.</p> If the subscription is not actively consuming the changes, the publisher won't release WAL files that contain changes a slot has not consumed. This prevents data loss, but can easily fill up disk if any of the subscribers fall too far behind.</strong></p> You can monitor the WAL retention per each replication slot.</p> #</span> SELECT</span></span> slot_name,</span></span> active,</span></span> pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) </span>AS</span> wal_retained</span></span> FROM</span> pg_replication_slots;</span></span> </span> -</span>[ RECORD 1 ]</span>+</span>--------------------</span></span> slot_name \| my_subscription</span></span> active \| t</span></span> wal_retained \| </span>265</span> MB</span></span></code></pre>Replication Identity</a> </h3> As we have already covered, logical replication works with row-level changes, such as INSERT</code>, UPDATE</code>, and DELETE</code>. And not all operations are equal. While INSERT</code> is relatively straightforward (a new row is sent to the subscriber), for both UPDATE</code> and DELETE</code> operations, PostgreSQL needs to be able to uniquely identify the target row to modify on the subscriber.</p> This is managed by the REPLICA IDENTITY</code> property of each published table. The default uses the table's primary key (it has one, right?). You can also point it at another unique index with USING INDEX</code>, or specify FULL</code>, which sends all the old column values of the row to find the target row (and should generally be avoided). PostgreSQL never falls back to FULL</code> on its own: without a primary key, the identity stays DEFAULT</code> but records nothing, and any UPDATE</code> or DELETE</code> on the published table fails with an error until you set an identity explicitly.</p> In practice:</p> Prefer REPLICA IDENTITY DEFAULT</code> (the primary key), the most efficient choice. If the table doesn't have a primary key, consider adding one (a surrogate key is always an option).</li> Use REPLICA IDENTITY USING INDEX index_name</code> if there is a better (or smaller) unique index matching the business logic or architecture of your database model.</li> </ul> There are special cases of what not to do when it comes to replication identity, but we will touch on those cases in the advanced section of this guide.</p> Schema Changes</a> </h3> Although it was already mentioned, it's vital to reiterate that logical replication, as it streams only row-level changes rather than full WAL segments, does not automatically replicate Data Definition Language (DDL) changes</strong>.</p> Any schema changes made on the publisher must be manually applied to all subscribers</strong> before data reflecting the change is replicated.</p> We will cover schema specifically in later parts of this guide, but to sum it up, this is the recommended order of operations when it comes to schema changes:</p> (Optional) but recommended for breaking changes, pause replication where applicable.</li> Apply and verify DDL changes to the subscribers.</li> Apply and verify DDL changes on the publisher.</li> Resume the replication if applicable.</li> </ol> While this might seem like a major drawback compared to physical replication, it's the characteristic that gives us the most flexibility.</p> Other Considerations</a> </h3> As logical replication reliably handles row-level changes, it's crucial to understand other specific limitations and behaviours. Specifically:</p> Sequences</strong> are not replicated. While they are used to generate the value on the publisher, and their value advances there, the corresponding sequence (if it exists) on the subscriber won't be updated.</li> Other database objects won't be replicated. While it might be understandable for views, stored procedures, triggers, and rules, you also can't rely on it for materialized views.</li> Special consideration (similar to schema changes) must be paid to user-defined data types. If a replicated table uses a user-defined type (e.g., a column using an ENUM</code>), the type must already be available on the subscriber(s) and have exactly the same name and structure. As ENUM</code>s are represented as ordered sets, even the order of the values matters.</li> </ul> Living with Logical Replication</a> </h2> In our previous example, we set up the publisher and subscriber, relied on the initial copy of the data, and let things run. Both of them will survive a restart, and thanks to the replication slot, they will keep going as soon as they come online.</p> But what if you need to perform maintenance or do regular things like a schema update on the subscriber or the publisher? Sometimes you might need to temporarily stop the replication</strong>. PostgreSQL makes this straightforward with subscription management, allowing you to stop consuming the changes.</p> ALTER</span> SUBSCRIPTION my_subscription </span>DISABLE</span>;</span></span></code></pre> As we hinted earlier, when disabled, the subscription stops consuming changes from the publisher, while keeping the replication slot. This means:</p> No new changes are applied on the subscriber.</li> WAL files will start to accumulate on the publisher (since the replication slot holds its position).</li> </ul> While disabled, you can't really check the status from the subscriber, as only the replication slot has the data. You can use the query we mentioned above on the publisher to check the status.</p> SELECT</span></span> slot_name,</span></span> active,</span></span> pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) </span>AS</span> wal_retained</span></span> FROM</span> pg_replication_slots;</span></span></code></pre> When ready, you can resume the subscription.</p> ALTER</span> SUBSCRIPTION my_subscription </span>ENABLE</span>;</span></span></code></pre>Coordinating Schema Changes</a> </h2> As we mentioned earlier, since logical replication does not automatically replicate DDL changes, you need to coordinate schema updates manually. Here's a basic overview of how to perform such a change, step by step.</p> First, disable the replication on the subscriber and apply your schema changes.</p> --- disable replication</span></span> ALTER</span> SUBSCRIPTION my_subscription </span>DISABLE</span>;</span></span> </span> --- apply changes on the subscriber</span></span> ALTER TABLE</span> products </span>ADD</span> COLUMN category_id </span>INTEGER</span>;</span></span> CREATE INDEX</span> products_by_category</span> ON</span> products(category_id);</span></span></code></pre> Only then can you apply the changes to the publisher.</p> ALTER TABLE</span> products </span>ADD</span> COLUMN category_id </span>INTEGER</span>;</span></span> CREATE INDEX</span> products_by_category</span> ON</span> products(category_id);</span></span></code></pre> And resume the replication.</p> ALTER</span> SUBSCRIPTION my_subscription </span>ENABLE</span>;</span></span></code></pre> If you applied the schema change to the publisher first, any new data using the new column would fail to replicate to the subscriber that doesn't have the column yet.</p> Handling Incompatible Changes</a> </h2> PostgreSQL's default conflict resolution is very simple. When a conflict occurs, logical replication stops</strong>, and the subscription enters an error state. For example, if you consider a product row already present on the subscriber, you will see something like this in the log files:</p> ERROR: duplicate key value violates unique constraint "products_pkey"</span></span> DETAIL: Key (id)=(452) already exists.</span></span> CONTEXT: processing remote data for replication origin "pg_16444" during message type "INSERT" for replication target relation "public.products" in transaction 783, finished at 0/1E020E8</span></span> LOG: background worker "logical replication apply worker" (PID 457) exited with exit code 1</span></span></code></pre> As you can see, logical replication stopped with an error code and won't continue until you manually resolve the conflict. To do so, you need to identify the conflicting data and decide what to do with it.</p> -- option 1: remove the conflicting data</span></span> DELETE FROM</span> products </span>WHERE</span> id </span>=</span> 452</span>;</span></span> </span> -- option 2: update the data to match the expected state</span></span> UPDATE</span> products </span>SET name =</span> 'New Product'</span>, price </span>=</span> 29</span>.</span>99</span> WHERE</span> id </span>=</span> 452</span>;</span></span></code></pre> Only after the conflict is resolved can the logical replication continue. This has been only a very simple example of a conflict that might arise during logical replication. Other examples might involve:</p> Data problems, constraint violations, or type mismatch</li> Schema conflicts (missing or renamed table/column)</li> Permissions issues or row-level security</li> </ul> In all those cases, you still need to go and fix the problem before logical replication can resume.</p>

boringSQL | Supercharge your SQL & PostgreSQL powers - postgresql

The tests passed. The plan didn't.

2026-07-12T20:15:00+00:00

TL;DR - RegreSQL 1.0 tested that your queries return the right rows. 2.0 tests that they return them the right way, and it does the checking against production's real statistics instead of your empty dev database, which lies.

A migration cleanup dropped an index nobody thought was load-bearing. Every test passed: same rows, same order, green. Three days later the API started timing out on a query that hadn't changed a character, because the planner had quietly switched it from an index scan to a sequential scan over a table that had kept growing.

The first version of RegreSQL</a> would have passed that change too. It tests what your queries return: run them, diff the rows against a committed expected file, go red when the output changes. That catches the query that now returns the wrong rows. It says nothing about the query that returns the right rows the wrong way, which is most of what takes a database down.

Version 2.0 tests that.

Test the plan, not just the rows</a> </h2>
Here is that failure on a laptop. An orders</code> table, an index on customer_id</code>, and a query that reads one customer's orders:
-- orders-by-customer.sql select id, total from orders where customer_id = :cid order by id;</code></pre> Baseline it and run the tests. Green:
$ regresql test ✓ 2 passing</code></pre> Now drop the index the way that migration did, and run the same tests again:
$ regresql test ✓ 1 passing ✗ 1 failing FAILING: orders-by-customer.1.buffers (3898 > 109 * 102%, +3476.1%) Expected buffers: 109 Actual buffers: 3898 (+3476.1%) Cost (info): 7962.41 (baseline: 421.03) ⚠️ Table 'orders': Bitmap Heap Scan → Seq Scan</code></pre> Here is the whole loop, start to failure, as it actually runs:
</a>
The output check still passes; the rows didn't change. The plan check catches what the output check can't: the same query, returning the same result, now runs a sequential scan instead of the bitmap index scan it used before, reading thirty-five times the buffers. That failure blocks the merge. And the diff names the table and the exact change, in the same shape as the row diffs you already read in code review. Plan drift stops being something you find in production and becomes something you review in a pull request.
An empty database is a liar </a> </h2>
That demo caught the regression because the table was big enough for the index to matter. On your dev database, it usually isn't. Run EXPLAIN</code> there and it says every query is fine, and it isn't lying on purpose: at a few hundred rows the cheapest plan really is a sequential scan, and it picks it. Production's planner reads different statistics and picks a different plan. A green suite on small data proves your queries are correct on small data, and nothing about the plan production will run. That gap, and why code review can't see it, is a story of its own.
You can't copy production data to your laptop. You don't need to. You need what the planner reads: the row counts, the histograms, the most-common-values. PostgreSQL 18 dumps and loads those on their own (pg_dump --statistics-only</code>), and RegreSQL 2.0 injects them so EXPLAIN</code>, against ten rows on your laptop, sees the real table's distribution:
regresql test --stats production-stats.sql</code></pre> Now the plan you baseline is the plan production would pick, and a plan regression fails on your laptop before it reaches a server. That is the difference between testing your query and testing it against reality.
Gate on what happened, not what was guessed </a> </h2>
Cost estimates are the planner's opinion. Opinions drift, and they drift most exactly when a plan is about to go wrong, because a bad plan is usually a bad estimate first. Gating a test on estimated cost means trusting the number that breaks first to tell you when it broke.
So 2.0 gates on measured behavior instead. regresql baseline --analyze</code> runs the query for real and records what happened: buffers actually read, whether a sort or hash spilled to disk, how many rows actually flowed through each node, and the cardinality error, the ratio between what the planner predicted and what it got (the "q-error"). A query whose estimates were off by 2x and are now off by 200x has a plan that is one data change away from collapse, and the row counts say so long before the cost does. The baseline stores actuals; the test compares actuals; the estimate is never on the witness stand.
Wall-clock time is the actual everyone wants and the one that lies most, because it moves with cache state, whatever else is on the box, and luck. 2.0 measures it and treats it as evidence, not a verdict: interleaved runs, a per-query median, and a noise threshold derived by permuting the samples, so a query counts as slower only when it beats its own measurement noise. Anything that can't clear the threshold goes in an unstable bucket instead of failing the build. Timing advises; it never gates the exit code.
Knowing when you can't tell</a> </h2> Here is the part that took the longest to get right, and the part I am most sure of. Point RegreSQL at the same corpus on two builds of PostgreSQL and it will show you every query that plans differently between them. Early on, one query looked like a clean regression: the newer build read nearly three times the buffers, deterministically, run after run. It had every mark of a real find. It was not one. Both builds had ANALYZE</code>'d their own copy of the data, and the query sat on a knife-edge join-order decision that a few unlucky sampled rows tip one way or the other. Inject one build's exact statistics into the other and they plan identically. Re-ANALYZE</code> and the "regression" moves to the other build. It was sampling noise, and a tool with less doubt would have reported it as a regression. So 2.0 will not make a cross-version claim on statistics it does not trust. It injects identical statistics into both builds, so any remaining difference is code and not luck, and it re-ANALYZE</code>s the baseline several times to throw out queries whose plan is a coin-flip to begin with. A difference is reported only when it survives both. On that investigation the honest output was zero regressions, and zero was the correct answer. A tool that only ever says "found something" is a tool you eventually stop believing. In a field full of confident dashboards, "I can't tell you that from this data" is the more useful sentence, and it is a first-class answer here, not an apology. Testing the planner itself</a> </h2> Chase these problems far enough and you end up at PostgreSQL's own test suite. pg_regress</code>, the harness that has guarded the database for decades, checks one thing: does the query return the expected text. It says nothing about the plan, the cost, or the cardinality error, because it folds all of them into a single .out</code> text file that either matches or does not. It is correctness-only by construction, and nothing in the standard toolchain measures whether a planner change made real queries better or worse. Everything above, separating results from plan-shape from measured actuals from cardinality error into layers that can each be judged on their own, is what you would need to answer that question. Which is why 2.0 also has the pieces that only make sense pointed at PostgreSQL rather than at your application: Cross-version scoreboards with the trust filter above, so a planner patch can be measured against a corpus without drowning in ANALYZE</code> noise.</li> A metamorphic check: flip an optimization that is supposed to preserve results (eager aggregation, memoize, incremental sort) and confirm the rows do not move. When they move, you have found a wrong-results bug in the optimizer, with no second database to compare against.</li> Determinism admission, which keeps only the queries whose result is stable across different plans, so an ambiguous LIMIT</code> over ties never masquerades as a regression.</li> </ul> This is a longer road, and an open one. But the same machinery that tells you your application query got slower can tell a PostgreSQL developer whether their patch made a thousand real queries slower, and that is a thing the project cannot currently see per-change. RegreSQL started as a way to test the queries you write. It turns out the honest way to do that is most of the way to a harness for the planner underneath them. Leave evidence</a> </h2> None of this is useful if it only runs on your laptop. regresql test</code> exits non-zero on a failure, emits JUnit, GitHub Actions, or JSON, and reads DATABASE_URL</code> so the same project runs against a CI database without editing committed config. A plan flip fails the build the way a wrong-rows change does, and the run leaves a machine-readable record of what was checked against which snapshot. The check that advised you in the editor is the same check that blocks the merge. You can go further and name the tables where a sequential scan is never acceptable, and have that fail the build outright. But which tables those are, and which of these signals is a release blocker versus a shrug, is a judgment the planner doesn't have and never will. That part is a longer story, and a separate one. What "verified" means</a> </h2> RegreSQL 2.0 is careful about the word. It does not promise your database is safe. It verifies what it can measure and says so plainly when it can't, which on the day it mattered meant reporting nothing at all. A green RegreSQL run is not "nothing will go wrong." It is "these queries return what they returned, plan the way they planned, against the statistics that actually apply." That is a smaller promise than most tools make, and one you can keep. It runs on your laptop, against a database you already have, no production access: regresql init postgres://localhost/yourdb</code></pre> From there the SQL passes through the harness the way your code already passes through the type checker. The getting-started guide</a> walks the whole loop, from zero to a plan check in CI, in about ten minutes.

Same rows, different SUM 2026-06-28T21:25:00+00:00 Everyone knows not to store money as a double precision</code>. One can hope. The rule is so well drilled that it has stopped being interesting, and it is also not where the trouble usually starts. The float is already in the schema before anyone weighs in on it: a measurement column someone later sums for a report, telemetry that drifts into a finance dashboard, a third-party feed ingested as double precision</code> because that is how it arrived. Here is the part the rule does not warn you about. Take a table of five million floating-point readings, sum the column, and run it three times in a row. Nothing else touches the table. Same connection, same data, same statement. SELECT sum(reading) FROM measurements;</code></pre> sum -------------------- 2500519211.7874823 sum ------------------- 2500519211.787477 sum -------------------- 2500519211.7874575</code></pre> Three runs, three different totals. No UPDATE</code>, no concurrent writer, no random seed. The rows did not change between runs, and the query is the same character for character. Yet the answer is not. This is not a Postgres bug, and it is not specific to Postgres. It is what happens when floating-point arithmetic meets parallel aggregation, and it has been generating "my dashboard total changed and I have no idea why" tickets for as long as databases have parallelized. The non-determinism does not wait for you to opt into bad practice; it shows up the moment a parallel plan runs over whatever floats you happen to have. The schema</a> </h2> One column of double precision</code>, five million rows. CREATE TABLE measurements ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, reading double precision NOT NULL ); INSERT INTO measurements (reading) SELECT random() * 1000 FROM generate_series(1, 5000000); ANALYZE measurements;</code></pre> Five million rows is enough that Postgres parallelizes the sum on its own. No settings forced, defaults all the way: EXPLAIN (COSTS OFF) SELECT sum(reading) FROM measurements;</code></pre> QUERY PLAN ----------------------------------------------------- Finalize Aggregate -> Gather Workers Planned: 2 -> Partial Aggregate -> Parallel Seq Scan on measurements</code></pre> Two workers each sum part of the table in parallel, and a final step folds the partial sums together. Everything that follows is in that split. Floating point does not associate</a> </h2> On paper, (a + b) + c</code> and a + (b + c)</code> are the same number. Postgres is not doing arithmetic on paper. Floating-point addition is not associative: change the grouping and you can change the answer, because every intermediate result is rounded to fit 64 bits. Move the parentheses and you move which roundings happen. The textbook demonstration takes three values and one subtraction: SELECT 1.0e20::float8 + 1.0::float8 - 1.0e20::float8;</code></pre> ?column? ---------- 0</code></pre> The answer should be 1</code>. Read the expression left to right and watch where it goes wrong: 1.0e20 + 1.0</code> should be 100000000000000000001</code>. But a double only carries about 16 significant digits, and that number needs 21. The trailing 1</code> falls off the end, below the smallest digit the number can represent. So the addition rounds back to exactly 1.0e20</code>.</li> 1.0e20 - 1.0e20</code> is then exactly 0</code>.</li> </ul> The 1</code> never made it past the first step. It was swallowed by the addition, before the subtraction even ran. This is absorption: a value too small relative to its neighbour to leave a mark. It is not a database phenomenon. The same expression evaluates to 0</code> in C, Python, JavaScript, and every other language that uses IEEE 754 doubles. A double precision</code> number keeps 52 bits for the mantissa, the digits that carry precision. This is around 15 to 17 decimal digits, no more. When the running total becomes big and the next number is small, the small one does not fit into those bits, so the rounding throws its tail away. Sum five million numbers and the total is, for most of its life, much bigger than any single number you add to it.</div> So the sum of a column depends on the order in which the rows are added. Not on which rows, on the order. That is the seam parallel query pries open. Why parallelism surfaces it</a> </h2> The plan spells out exactly how. Parallel Seq Scan</code> does not split the table into fixed slices upfront. It hands every worker a shared pointer into the same block iterator, and each worker grabs the next block, processes it, and comes back for another. Each worker runs its own Partial Aggregate</code>, summing only the rows it happened to grab, and by default the leader process pitches in as a third one (parallel_leader_participation</code>). Gather</code> collects those partial sums in whatever order they arrive, and Finalize Aggregate</code> folds them into a single total. Two things vary from run to run, and neither is under your control: Which rows each process sees. Because the blocks are pulled from a shared iterator on demand, the split is a race. A worker that gets a few microseconds ahead, for any reason the OS scheduler decides, grabs the next block, so the same worker ends up with a different set of rows on every run. The division is never guaranteed to be the same twice.</li> The order the partial sums come back. Finalize Aggregate</code> folds the partials into its total in arrival order. With only two values this would not matter, since a + b</code> equals b + a</code>. But there are three of them, the two workers and the leader, and combining three floats is once again an associativity question: which two get added first decides the rounding.</li> </ol> Both feed directly into the one thing float addition is sensitive to: grouping. Different row split, different partial sums; different arrival order, different final grouping of the additions. The total lands a few bits away from where it landed last time. Across five million readings that shows up in the sixth decimal place, exactly where the three runs above disagreed. sum(float8)</code> is marked parallel-safe, which is why the planner is free to do this. It is safe in the sense that it will not crash or corrupt anything. It is not safe in the sense people assume, that the answer is a function of the input. This is not unique to SUM</code>. AVG</code>, STDDEV</code>, VARIANCE</code>, and any other floating-point aggregate that accumulates across rows inherit the same order sensitivity. SUM</code> is just the easiest to demonstrate and the one most likely to land in a number a human reads. </div> It was never really parallelism</a> </h2> It is tempting to file this under "parallel query is flaky" and turn it off. That misses the point. Parallelism does not cause the non-determinism, it only exposes an order-dependence that was there the whole time. Turn parallelism off completely and the sum becomes stable: SET max_parallel_workers_per_gather = 0; SELECT sum(reading) FROM measurements; -- 2500519211.787516 SELECT sum(reading) FROM measurements; -- 2500519211.787516 SELECT sum(reading) FROM measurements; -- 2500519211.787516</code></pre> Stable, but only because a serial sequential scan visits the heap in the same physical order every time, so the additions group the same way every time. The order is fixed, so the rounding is fixed. Change the order and the serial sum moves too. We can force different orders without any parallelism at all, using an ordered-set aggregate: SET max_parallel_workers_per_gather = 0; -- serial, so ONLY order varies SELECT sum(reading ORDER BY id) FROM measurements; -- 2500519211.787516 SELECT sum(reading ORDER BY reading) FROM measurements; -- 2500519211.7876067 SELECT sum(reading ORDER BY reading DESC) FROM measurements; -- 2500519211.7875423</code></pre> Same rows, same single thread, three summation orders, three answers. Parallelism is one way to shuffle the order, but a plain ORDER BY</code> inside the aggregate does it just as well. The dependence is on order, full stop. Deterministic is not the same as correct</a> </h2> There is a subtler trap hiding in the "turn it off and it is stable" result. A stable answer is not necessarily the right answer. To see the true total, sum the same values as numeric</code>, which adds in exact decimal with no rounding until the very end: SELECT sum(reading::numeric) FROM measurements;</code></pre> sum ------------------------------- 2500519211.787503050078417764</code></pre> That is the exact sum of the stored values. Now line everything up: How the sum was computed</th> Result</th> Off by</th></tr></thead> Exact (numeric</code>)</td> 2500519211.787503050…</code></td> reference</td></tr> Serial float8</code>, physical order</td> 2500519211.787516</code></td> ~1.3e-5</td></tr> Parallel float8</code>, run A</td> 2500519211.7874823</code></td> ~2.1e-5</td></tr> Parallel float8</code>, run B</td> 2500519211.787477</code></td> ~2.6e-5</td></tr> Ordered by value ascending</td> 2500519211.7876067</code></td> ~1.0e-4</td></tr> </tbody></table> Every floating-point total is wrong, including the "stable" serial one. They differ from the exact value in the fifth and sixth decimal places. Parallelism did not introduce the error. It removed the one comforting property the serial sum had, which was getting the same wrong answer every time. That sameness is what people mistake for correctness. Summing smallest-to-largest is the classic textbook advice for reducing floating-point error, and here ascending order happens to land further from the exact value than physical order does. Order changes the error; it does not reliably shrink it. If you need a smaller error there are compensated-summation algorithms (Kahan and friends), but inside a SQL aggregate you do not get to choose them.</div> Where it actually bites</a> </h2> In the sixth decimal of a five-million-row random sum, nobody cares. The problem is the places where someone does. The obvious one is reconciliation. A finance team sums a revenue column stored as double precision</code>, exports it, and the number disagrees with the same query run an hour later, or with the same logic in a different tool running a different parallel degree. Nothing in the data changed, so the hunt goes to caching, to replicas, to "a dirty read somewhere" long before anyone suspects the aggregate itself. The same thing wrecks regression tests, just less visibly. Assert that a float total equals an exact literal, or compare two pipelines for byte-identical sums, and the test will flap. It passes on a small fixture that runs serially, then fails in CI on a larger one that crosses the parallel threshold, with no code change in between. The test is asserting a property the type does not have, and it will keep flapping until someone notices that. And anywhere a float sum feeds an equality check, a join, or a GROUP BY</code> key, you are building on sand. Two totals that "should" be equal can differ in their last bits, and now your join silently drops rows. Postgres already knows this is dangerous</a> </h2> The most convincing evidence that this is a real, known hazard and not a corner case is that Postgres deliberately gives up an optimization because of it. Window functions can compute a moving sum efficiently using an inverse transition function: as the frame slides forward, subtract the value leaving the window instead of re-summing the whole frame. It works for numeric</code> and integers. For float4</code> and float8</code>, Postgres refuses to do it. From the documentation on user-defined aggregates</a>: An example of an aggregate for which adding an inverse transition function seems easy at first, yet where this requirement cannot be met is sum</code> over float4</code> or float8</code> inputs. </blockquote> The docs then show why, with the same absorption we saw earlier. If a window holds 1e20</code> and 1</code>, subtracting 1e20</code> as it leaves the frame yields 0</code>, not the 1</code> that should remain. So the built-in float sum simply does not use the inverse optimization, and a moving sum(float8)</code> recomputes the frame to stay correct: SELECT n, sum(x) OVER (ORDER BY n ROWS BETWEEN CURRENT ROW AND 1 FOLLOWING) FROM (VALUES (1, 1.0e20::float8), (2, 1.0::float8)) AS v(n, x);</code></pre> n | sum ---+------- 1 | 1e+20 2 | 1</code></pre> The second row is 1</code>, correct, because Postgres declined the shortcut that would have made it 0</code>. The engine is quietly working around the imprecision of float sums on your behalf. The non-determinism in the parallel case is the same hazard showing up somewhere the engine cannot paper over it. What to do about it</a> </h2> The fix is almost always the type, not the query. Use numeric</code> for anything anyone reconciles, compares, or reads as a figure. Money, totals on reports, anything that has to match another system or itself across runs. numeric</code> sums in exact decimal and is order-independent: same rows, same answer, parallel or not, in any order. SET max_parallel_workers_per_gather = 4; SELECT sum(reading::numeric) FROM measurements; -- 2500519211.787503050078417764 SELECT sum(reading::numeric) FROM measurements; -- 2500519211.787503050078417764</code></pre> The cast above stabilizes a single query, but the durable fix is storing the column as numeric</code> in the first place, so the exact values are what live on disk. numeric</code> is not free. It is arbitrary-precision arithmetic done in software, so it is slower to compute and takes more space on disk than a hardware float the CPU adds in a single instruction. For anything that has to reconcile, that is the price of correctness, and it is almost always worth paying. For a column nobody ever sums into a figure a person reads, it usually is not. If the data is genuinely floating point (sensor readings, scientific measurements, anything where the input is already an approximation), then accept that aggregates over it are approximate and do not assert exact equality on them. Round to a sensible number of decimals before comparing or displaying, and never use a raw float sum as a join or grouping key. Pinning max_parallel_workers_per_gather = 0</code> makes a float sum repeatable, but does not make it correct, and it costs you parallelism on every other query in the session. It is a way to get a stable wrong answer, not a fix. Reach for the type instead. None of this is Postgres-specific. Any database that parallelizes aggregation, and they all do, has the same behavior over the same IEEE 754 floats. The trigger that makes it visible is simply crossing whatever threshold turns parallelism on, which is why it so often shows up when a table grows past a certain size and "nothing else changed." The NULL in your NOT IN 2026-06-14T23:30:00+00:00 A NOT IN</code> query can return the wrong answer without telling you. It is valid SQL, it runs without an error, and it hands back a perfectly well-formed result set that happens to be empty when it should not be. No warning, no hint, nothing in the logs: just zero rows where you expected hundreds, and a database that considers it correct. Almost always the cause is a single NULL</code> sitting somewhere you forgot to look, combined with two keywords you have typed a thousand times: NOT IN</code>. None of it is a Postgres bug. This is exactly what the SQL standard mandates, implemented faithfully. That is precisely what makes it so easy to walk into, and why the planner could not safely optimize around it for the better part of Postgres's history. It comes down to one if</code> statement in the parser. Sample schema</a> </h2> Nothing elaborate. A table of products, one of which has no category assigned yet, and a table of archived categories that happens to contain a NULL</code>: CREATE TABLE products (id int, category_id int); INSERT INTO products VALUES (1, 10), (2, 20), (3, NULL), (4, 10); CREATE TABLE archived (category_id int); INSERT INTO archived VALUES (20), (NULL);</code></pre> The NULL</code> in archived</code> is not contrived. The moment a column is nullable (and most are, by default), a NULL</code> can find its way into any subquery you point a NOT IN</code> at. That is the whole point: this is not an exotic data condition, it is the ordinary one. The query that returns nothing</a> </h2> Here is the request you have written a hundred times: give me the products whose category is not archived. SELECT id, category_id FROM products WHERE category_id NOT IN (SELECT category_id FROM archived);</code></pre> You expect products 1 and 4 (category 10, which is not in the archived set). What comes back is: id | category_id ----+------------- (0 rows)</code></pre> Every row gone. Not a subset, not an off-by-one: all of them. Drop the NULL</code> from archived</code> and the same query behaves: SELECT id, category_id FROM products WHERE category_id NOT IN (SELECT category_id FROM archived WHERE category_id IS NOT NULL);</code></pre> id | category_id ----+------------- 1 | 10 4 | 10 (2 rows)</code></pre> To understand why a single NULL</code> empties the entire result, we have to stop thinking of NOT IN</code> as a single thing and watch the parser take it apart. IN is an OR, NOT IN is an AND</a> </h2> IN</code> is not a primitive operator. It is shorthand that the parser rewrites into a chain of equality comparisons joined by OR</code>: x IN (a, b, c) -- becomes x = a OR x = b OR x = c</code></pre> NOT IN</code> is the logical negation of that, and by De Morgan's law negating an OR</code> of equalities gives you an AND</code> of inequalities: x NOT IN (a, b, c) -- becomes x <> a AND x <> b AND x <> c</code></pre> This is not an analogy. It is literally the expression Postgres builds, and you can read it straight off an EXPLAIN</code>. The literal-list forms collapse into array operators whose names give the whole game away: EXPLAIN (COSTS OFF) SELECT * FROM products WHERE category_id IN (1, 2, 3); -- Filter: (category_id = ANY ('{1,2,3}'::integer[])) EXPLAIN (COSTS OFF) SELECT * FROM products WHERE category_id NOT IN (1, 2, 3); -- Filter: (category_id <> ALL ('{1,2,3}'::integer[]))</code></pre> IN</code> is = ANY</code>: equal to any element, an OR</code>. NOT IN</code> is <> ALL</code>: different from all elements, an AND</code>. The actual node types matter here, because they are what you end up staring at when you dump a parse tree or read a normalized pg_stat_statements</code></a> entry. A literal list compiles to a single ScalarArrayOpExpr</code>: the scalar on the left, the array on the right, and a useOr</code> flag that is the entire difference between = ANY</code> and <> ALL</code>. The subquery forms are a different node altogether, a SubLink</code>. Recognising those two names on sight tells you immediately which path the planner is on. If "IN</code> and = ANY</code> are the same operator" is news: they compile to the same parse node and the same plan, with the spellings diverging only in plan-cache churn and selectivity estimates. The NOT IN</code> case in front of you here is the one corner where the choice is not cosmetic but a matter of correctness.</div> Three-valued logic does the rest</a> </h2> SQL does not have two truth values, it has three: true, false, and unknown. Any comparison against NULL</code> yields unknown</code>, because NULL</code> means "no value here" and you cannot ask whether an absent value is different from 20: -- not false. unknown (displayed as a blank) SELECT 10 <> NULL;</code></pre> Now walk the NOT IN</code> expansion for product 1 (category 10) against the archived set of 20 and NULL</code>: 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> </div> true AND unknown</code> is unknown</code>, not true</code>. A WHERE</code> clause keeps a row only when its predicate evaluates to true. Both false</code> and unknown</code> cause the row to be discarded. So product 1 is dropped. Run the same arithmetic for product 4 and you land on unknown</code> again. The mechanism in one sentence: the instant a single NULL</code> enters the right-hand side, the trailing AND unknown</code> term can never be true</code>, so the whole NOT IN</code> can never be true</code>, so every row is discarded, regardless of how many million rows you have or what they contain. </div> NULLs on the left side too</a> </h2> Keeping NULL</code>s out of the subquery is not enough. The same unknown</code> arises from NULL</code>s on the left: product 3 (whose category_id</code> is NULL</code>) evaluates to unknown AND unknown</code>, so it is dropped even against a spotless right-hand set. IN</code> and NOT IN</code> are not complements: a row can fail both tests simultaneously. There is a NULL</code>-shaped gap between them that belongs to neither. The seam, in the source</a> </h2> All of this reduces to one branch in one function. Open src/backend/parser/parse_expr.c</code> and find transformAExprIn</code>, the routine that turns both IN</code> and NOT IN</code> list expressions into something the planner can chew on. The very first thing it decides is whether it is building an OR</code> or an AND</code>: /* * If the operator is <>, combine with AND not OR. */ if (strcmp(strVal(linitial(a->name)), "<>") == 0) useOr = false; else useOr = true;</code></pre> That is the entire fork. IN</code> arrives carrying the operator =</code> and gets useOr = true</code>; NOT IN</code> arrives carrying <></code> and gets useOr = false</code>. The flag rides all the way down to where the boolean tree is finally assembled, several hundred lines later: result = (Node *) makeBoolExpr(useOr ? OR_EXPR : AND_EXPR, list_make2(result, cmp), a->location);</code></pre> OR_EXPR</code> for IN</code>, AND_EXPR</code> for NOT IN</code>. There is no special-casing of NULL</code> anywhere in this function, and there does not need to be: the three-valued behavior is an emergent property of having chosen AND</code>. The parser does the obvious, correct thing, and the NULL</code> semantics fall straight out of standard boolean logic. The "bug", if you insist on the word, belongs to the SQL standard, which Postgres implements faithfully. A grammar asymmetry: list vs. subquery</a> </h3> A list and a subquery are built differently. The list form is the <> ALL</code> chain of inequalities you just saw. The subquery form is not a <> ALL</code> at all: it becomes NOT (foo = ANY (subquery))</code>. Different shapes, same truth table, a NULL</code> in the comparison makes the result unknown</code>, and unknown</code> loses. That NOT (... = ANY ...)</code> shape is the one the planner sees. Why the planner won't save you</a> </h2> The correctness problem has a plan-shape twin. Watch what the planner does with three sibling queries on a larger schema of 200,000 orders</code> and 1,000 vip</code> rows. First, the positive case, IN</code> against a subquery: EXPLAIN (COSTS OFF) SELECT * FROM orders WHERE customer_id IN (SELECT customer_id FROM vip);</code></pre> Hash Semi Join Hash Cond: (orders.customer_id = vip.customer_id) -> Seq Scan on orders -> Hash -> Seq Scan on vip</code></pre> A clean semi-join. The planner promotes the subquery to a first-class relation, picks a hash join, and is free to reorder it against the rest of the query. Now the natural mirror, orders whose customer is not a VIP, written first with NOT EXISTS</code>: EXPLAIN (COSTS OFF) SELECT * FROM orders o WHERE NOT EXISTS (SELECT 1 FROM vip v WHERE v.customer_id = o.customer_id);</code></pre> QUERY PLAN ---------------------------------------------- Hash Anti Join Hash Cond: (o.customer_id = v.customer_id) -> Seq Scan on orders o -> Hash -> Seq Scan on vip v</code></pre> A hash anti-join, the efficient and symmetric counterpart to the semi-join. Now the same intent expressed with NOT IN</code>: EXPLAIN (COSTS OFF) SELECT * FROM orders WHERE customer_id NOT IN (SELECT customer_id FROM vip);</code></pre> QUERY PLAN --------------------------------------------------------------- Seq Scan on orders Filter: (NOT (ANY (customer_id = (hashed SubPlan 1).col1))) SubPlan 1 -> Seq Scan on vip (4 rows)</code></pre> No join at all. The subquery collapses into an opaque SubPlan</code> filter bolted onto a sequential scan, the NOT (... = ANY ...)</code> shape from the grammar is right there in the filter. The wording of that filter line is version-specific. PostgreSQL 16 prints the terser Filter: (NOT (hashed SubPlan 1))</code>; a later version changed EXPLAIN</code> to expose the inner comparison, which is the NOT (ANY (... = (hashed SubPlan 1).col1))</code> form shown here on 18.4. The plan underneath is the same opaque subplan in every released version.</div> The difference between those last two plans is structural, not a tuning detail. A Hash Anti Join</code> builds one hash table from the inner relation and streams the outer relation through it exactly once, and when the inner side outgrows work_mem</code> it partitions into batches and spills to disk without changing the result. A SubPlan</code> gives the planner none of that. Even the hashed</code> variant you see here is evaluated in place as a filter on the scan: it cannot be reordered during the global join search, it cannot have the outer query's join clauses pushed into it to drive an index scan, and it has no multi-batch spill logic to fall back on if the hashed set turns out larger than the estimate. The subquery stops being a relation the optimizer can plan and becomes an opaque function the optimizer has to call, once per outer row. This is the same optimization-fence effect as an uninlined CTE</a>: the planner simply cannot see inside. On small inputs nobody notices; on large ones it is the difference between milliseconds and minutes. The reason the planner is stuck with that shape is precisely the NULL</code> semantics from earlier. An anti-join keeps a row when it finds no match. But NOT IN</code> must discard a row when the comparison goes unknown</code>, and "unknown" is not the same as "no match". Those two behaviors diverge exactly when a NULL</code> is in play, so the planner historically could not prove the rewrite safe and never attempted it. Declaring the columns NOT NULL</code> does not rescue you on released Postgres. I constrained both the outer and inner columns NOT NULL</code> on 18.4 and re-ran the query, and the plan was still the opaque SubPlan</code> filter. The released planner does not even look. So NOT IN (subquery)</code> has been a correctness trap for as long as Postgres has existed, and a planner pessimization ever since anti-joins arrived in 8.4 (2009) and every other negated subquery learned to use them — neither of which the released optimizer rescues you from. </div> The fix is landing in PostgreSQL 19</a> </h2> That last paragraph is finally going out of date. This is a performance fix, not a semantic one: it rescues the case that was already correct but badly planned (NOT NULL</code> columns stuck behind the opaque SubPlan</code> from the previous section) and leaves nullable columns exactly where the SQL standard puts them. Tracing this through a checkout of the development branch, I found a function that exists in no released Postgres: sublink_testexpr_is_not_nullable</code>, in src/backend/optimizer/plan/subselect.c</code>. It guards a brand-new branch inside convert_ANY_sublink_to_join</code>: /* * Per SQL spec, NOT IN is not ordinarily equivalent to an anti-join, so * that by default we have to fail when under_not. However, if we can * prove that neither the outer query's expressions nor the sub-select's * output columns can be NULL, and further that the operator itself cannot * return NULL for non-null inputs, then the logic is identical and it's * safe to convert NOT IN to an anti-join. */ if (under_not && (!sublink_testexpr_is_not_nullable(root, sublink) || !query_outputs_are_not_nullable(subselect))) return NULL;</code></pre> git blame</code> dates it to commit 383eb21ebff</code>, "Convert NOT IN sublinks to anti-joins when safe", merged in March 2026 by Richard Guo. It lives on master</code>, bound for PostgreSQL 19. It appears in no REL_18</code> tag, which is exactly why my 18.4 box still produced the SubPlan</code> even with NOT NULL</code> columns. The commit message states the bargain plainly: if we can prove that neither side of the comparison can yield NULL values, and further that the operator itself cannot return NULL for non-null inputs, the behavior of NOT IN and anti-join becomes identical. </blockquote> The proof has to establish three things: Both operands provably non-NULL</code>. Established from schema NOT NULL</code> constraints (via a NOT-NULL-attnums hash table), from outer-join nullability tracking (so a Var</code> from the nullable side of an outer join does not qualify), and from qual clauses that force a Var</code> non-null.</li> The subquery's output columns provably non-NULL</code>, the query_outputs_are_not_nullable</code> half of the guard.</li> A NULL</code>-safe operator. The operator must belong to a B-tree or Hash operator family. That is a proxy for "behaves like a normal boolean comparison and won't return NULL</code> on non-null inputs", because an operator that did return NULL</code> there would break the very index it claims to support.</li> </ul> When all three hold, the NULL</code>-handling mismatch evaporates and NOT IN</code> is finally allowed to become a JOIN_ANTI</code>: result->jointype = under_not ? JOIN_ANTI : JOIN_SEMI;</code></pre> It lifts a planner limitation that has stood since anti-joins themselves arrived in 8.4, but only when the planner can prove no NULL</code> can reach the comparison. If your column is nullable, you are exactly where you have always been, in PostgreSQL 19 as in 9.x. The semantics never changed; the optimizer merely learned to recognize the cases where NULL</code> is provably absent. Decision matrix</a> </h2> You wrote</th> Internal shape</th> NULL on right →</th> NULL on left →</th> Planner (≤ PG 18)</th></tr></thead> IN (1,2,3)</code></td> = ANY</code>, an OR</code></td> absorbed, no harm</td> row dropped (no match)</td> ScalarArrayOpExpr</code></td></tr> NOT IN (1,2,3)</code></td> <> ALL</code>, an AND</code></td> all rows dropped</td> row dropped</td> ScalarArrayOpExpr</code></td></tr> IN (subquery)</code></td> = ANY</code> sublink</td> absorbed</td> row dropped</td> Semi Join</td></tr> NOT IN (subquery)</code></td> NOT (= ANY)</code> sublink</td> all rows dropped</td> row dropped</td> opaque SubPlan</code> filter ¹</td></tr> NOT EXISTS (...)</code></td> anti-join sublink</td> row kept</td> row kept</td> Anti Join</td></tr> </tbody></table> ¹ PostgreSQL 19 promotes this to an Anti Join when both the outer expression and the subquery output column are provably NOT NULL</code>. What to do instead</a> </h2> Don't wait for PostgreSQL 19, and don't lean on it once it ships: it only fires on provably NOT NULL</code> columns. Every portable fix comes down to one rule: keep NULL</code> away from NOT IN</code>. In practice that means changing the query, and there are three moves, in order of preference. Default to NOT EXISTS</code>. Make this the habit and you never hit the trap again. Same anti-join semantics and the same plan as the working NOT IN</code> would have wanted, on every version, nullable columns or not: SELECT id, category_id FROM products p WHERE NOT EXISTS ( SELECT 1 FROM archived a WHERE a.category_id = p.category_id );</code></pre> It keeps product 3, the NULL</code> category, because NOT EXISTS</code> asks whether a matching archived row exists, not whether category_id = NULL</code> is true. That is usually the answer you wanted. If you do want the NULL</code>-category rows gone, add AND p.category_id IS NOT NULL</code> and you have decided it on purpose. Filter the NULL</code>s out of the subquery when you have to keep the NOT IN</code> (a generated query you can only edit inside the parentheses, say): SELECT id FROM products WHERE category_id NOT IN ( SELECT category_id FROM archived WHERE category_id IS NOT NULL );</code></pre> This covers the right-hand NULL</code> and nothing else: product 3 still disappears, because the left-hand NULL</code> is untouched. Use it only when you cannot reach for NOT EXISTS</code>. Use EXCEPT</code> for whole-set comparisons. It matches rows with IS NOT DISTINCT FROM</code>, so two NULL</code>s count as equal and the three-valued trap never fires. But that same rule means a NULL</code> in archived</code> removes the NULL</code> rows from products</code>: SELECT category_id FROM products EXCEPT SELECT category_id FROM archived; -- returns: {10} -- product 3 (NULL category) is dropped because archived also has a NULL</code></pre> So NOT EXISTS</code> keeps product 3 and EXCEPT</code> drops it. Pick the one whose answer matches what you're after. EDIT (2026-06-15): I originally said this stood for about 25 years, which mixed up two things. The correctness trap is as old as NOT IN</code>; the planner pessimization only goes back to 8.4 (2009), when anti-joins arrived. Reworded to keep the two separate. pg_stat_statements: everything it can't 2026-06-03T08:35:00+00:00 Part one</a> made the core case: pg_stat_statements</code> counts, it doesn't record. It walked through how the queryid jumble fragments one logical query into many rows, how the first-seen text freezes your per-request tags, and how the averages bury the p99 that actually pages you. All of that was about data the extension has and distorts. This part is about the rest: the entries it silently throws away, the query text that can vanish all at once, the plans and replicas it never records, and the knobs that bite. It ends where part one started, with the question the whole investigation was really about: is this the query store Postgres is missing, or just the floor you'd build one on? The table fills up and evicts your tail</a> </h2> pg_stat_statements.max</code> defaults to 5000. It's a hard cap on entries, set when the server starts (changing it needs a restart, because the hash table is sized in shared memory up front). When the 5001st distinct shape arrives, Postgres doesn't grow the table. It evicts, throwing out the least-executed entries to make room: If more distinct statements than that are observed, information about the least-executed statements is discarded. </blockquote> On a healthy app with a few hundred steady shapes, 5000 is plenty and you never think about it. But remember the row explosion</a> from part one. An ORM that splinters one query into hundreds of shapes, or a pre-18 app building dynamic IN</code> lists, can chew through thousands of entries an hour. Once that starts, the view becomes a sliding window over recent noise. Your steady, important queries get evicted to make room for thousands of one-offs, then rebuilt with fresh counters and a fresh first-seen text when they run again. The stats you were trusting reset themselves, and the view never says a word about it. One place does say something. The companion view pg_stat_statements_info</code> has exactly two columns, and both matter: SELECT dealloc, stats_reset FROM pg_stat_statements_info;</code></pre> dealloc | stats_reset ---------+------------------------------- 18342 | 2026-05-01 03:00:11.482+00</code></pre> dealloc</code>: Total number of times pg_stat_statements</code> entries about the least-executed statements were deallocated because more distinct statements than pg_stat_statements.max</code> were observed. </blockquote> When dealloc</code> climbs steadily between resets, your table is too small for how many shapes your workload produces. You're thrashing, and your stats are lossy. The fix is either to raise pg_stat_statements.max</code> and pay the shared memory, or to cut the cardinality at the source by normalizing shapes, moving to PG 18, or switching to array binding. Few people watch dealloc</code>, so a thrashing table quietly drops part of the workload it's meant to report, and nobody notices until the numbers stop adding up. The second column, stats_reset</code>, is the time of the last global reset of the whole view, and it isn't decoration. It's the validity stamp on every diff you compute. If you snapshot the view hourly to get usable averages, stats_reset</code> tells you whether a given window is even legitimate: if it moved forward between your two snapshots, a reset happened inside the window and every delta across it is meaningless. Watch dealloc</code> and stats_reset</code>. A steadily climbing dealloc</code> means the table is too small and your tail is being evicted, so the stats are lossy. A stats_reset</code> that moved between two snapshots means a reset landed inside your window, so any diff across it is meaningless. Almost no monitoring setup checks either. </div> When the query text vanishes</a> </h3> There's a worse failure mode hiding in the external query-text file. Because representative texts pile up on disk and the file can grow large (long queries, a high max</code>, lots of churn), the extension has a last resort: pg_stat_statements</code> may choose to discard the query texts, whereupon all existing entries in the pg_stat_statements</code> view will show null query</code> fields, though the statistics associated with each queryid</code> are preserved. If this happens, consider reducing pg_stat_statements.max</code>. </blockquote> So under pressure it keeps the counters and throws away the text. Your whole view goes query = NULL</code> while the numbers keep marching on, and you're left with a list of anonymous queryids and no idea what they used to be. The docs leave this vague, but the source is specific enough to alert on. Two thresholds are at work. The first is the garbage-collection trigger. As entries get evicted and rebuilt, their old texts leave dead space in the file. The extension rewrites the file to drop that dead space once it's more than about 50% bloat, specifically when the file on disk grows past mean_query_len × pg_stat_statements.max × 2</code>. With the default max</code> of 5000 and, say, a 1 KB average query, that's a rewrite around the 10 MB mark. That part is normal housekeeping. It's not the failure. The failure is when that rewrite itself fails, from an I/O error or because it can't allocate room to build the replacement file. At that point the file's contents can't be trusted, so the extension marks every entry's text as invalid (internally, query_len = -1</code>) and the whole query</code> column reads NULL</code>. The counters survive. The texts are gone all at once. On PostgreSQL 13 and earlier, stats_temp_directory</code> could relocate that file; the GUC was removed in 14.</div> The practical version: the file lives at $PGDATA/pg_stat_tmp/pgss_query_texts.stat</code>. Alert on its size against mean_query_len × max × 2</code>. If you're routinely near that line you're rewriting the file constantly, and you're one bad write()</code> away from a blank view. Because the trigger scales with your average query length, the cluster most at risk is the one with both lots of entries and very long query texts, which is the giant generated IN</code>-list app all over again. The fix the manual suggests, lowering pg_stat_statements.max</code>, sounds backwards but is right: a smaller table means a smaller file, less bloat to collect, and less to go wrong. What it never records at all</a> </h2> Everything so far has been about data the extension records but degrades, by normalizing, freezing, averaging, or evicting it. This section is about absence: things pg_stat_statements</code> never captures at all, because they sit outside what it models. Plans. This is the big one. The extension records that a query ran and how long it took. It doesn't record how it ran. No plan tree, no per-node costs, no join order, no index choice. A query whose plan flipped from an Index Scan to a Seq Scan overnight, the single most common cause of a sudden slowdown, shows up only as a mean_exec_time</code> that went up. The reason is gone. For plans you need EXPLAIN</code>, auto_explain</code>, or a plan-capturing hook. pg_stat_statements</code> sits at the wrong layer to ever see one. Actual versus estimated rows. With no plan there's no per-node actual_rows</code> to compare against the planner's estimate. The estimation error, which is the leading sign of stale statistics</a> and the thing that predicts a future plan flip, is invisible here. The rows</code> column is total rows returned across all calls, not an estimate measured against reality. Parameter values. The constants were jumbled out. You can't recover which id</code> was slow, only that the shape WHERE id = $1</code> was slow on average. Failed queries. This one surprises people. Stats are updated at the end of a successful phase, and only then. A query that errors out mid-execution isn't counted. The manual lays out the asymmetry exactly: plans</code> and calls</code> aren't always expected to match because planning and execution statistics are updated at their respective end phase, and only for successful operations. For example, if a statement is successfully planned but fails during the execution phase, only its planning statistics will be updated. </blockquote> So the query that runs 30 seconds and then dies on a statement timeout, exactly the one you most want to find, contributes nothing to calls</code> or total_exec_time</code>. It planned, so it might bump plans</code> if track_planning</code> is on, and that's the only trace it leaves. Some of your worst queries are invisible precisely because they're your worst queries. There's no error counter either. A statement that fails leaves no row of its own, so you can't ask pg_stat_statements</code> how often a given shape errors, only how often it succeeded.</div> Spills, parallel workers, per-node buffers. Sort and hash spills to disk, workers_planned</code> versus workers_launched</code>, the shared/local/temp buffer split per plan node, all of that lives in EXPLAIN (ANALYZE, BUFFERS)</code></a> output, not in the aggregated shared_blks_*</code> totals. The view gives you a sum of blocks touched. It can't tell you where in the plan they were touched, or whether a sort spilled. It only sees one node</a> </h2> pg_stat_statements</code> is per-instance. Its hash table lives in this server's shared memory and counts the queries this server ran. It has no idea other nodes exist. In a single-primary world that was fine. In a modern deployment (a primary with read replicas, an Aurora reader fleet, a Patroni cluster, Citus) it's a serious blind spot. Read traffic is the majority of the work for most apps, and it runs on the replicas. Each replica has its own independent pg_stat_statements</code>, with its own counters, its own eviction, its own first-seen texts. Snapshot only the primary, which is what most monitoring setups do, and you see the write path plus whatever reads happen to land on the primary, while the bulk of the read workload goes unmeasured. Merging the nodes isn't clean either. You'd want to add up calls</code> for "the same query" across nodes, but the queryid is platform- and version-sensitive, so on a mixed fleet (mid-upgrade, different architectures) the same query can carry different ids on different nodes. Merging a cluster's workload under one canonical fingerprint, with per-node drill-down, is real work that lives above the extension. The extension only ever hands you one node's local tally. The knobs, and what they cost</a> </h2> A quick tour of the settings, because the defaults hide a few traps. GUC</th> Default</th> What to know</th></tr></thead> pg_stat_statements.max</code></td> 5000</code></td> Hard cap on entries; restart to change. Watch pg_stat_statements_info.dealloc</code>.</td></tr> pg_stat_statements.track</code></td> top</code></td> top</code> tracks client-issued statements only. Queries inside PL/pgSQL functions stay invisible until you set all</code>.</td></tr> pg_stat_statements.track_utility</code></td> on</code></td> Counts non-DML (CREATE</code>, VACUUM</code>, and so on). Can flood the table with one-off DDL on schema-churning systems.</td></tr> pg_stat_statements.track_planning</code></td> off</code></td> Adds planning-time stats, but the manual warns of "a noticeable performance penalty" from contention when many connections run identical shapes and fight over one entry's spinlock. Off for a reason.</td></tr> pg_stat_statements.save</code></td> on</code></td> Persists stats across clean restarts. A crash still loses everything since the last save.</td></tr> </tbody></table> Three that aren't obvious from the table: track = 'top'</code> hides function internals. If your hot logic lives in PL/pgSQL or stored procedures, the default shows you the SELECT do_the_thing()</code> wrapper and nothing about the queries it runs inside. You have to opt into all</code> and pay for the extra entries. track_utility</code> counts your transaction verbs. BEGIN</code>, COMMIT</code>, and ROLLBACK</code> are utility statements, so with the default track_utility = on</code> each one gets its own entry. They normalize to a single shape apiece, so it's a handful of rows, not a flood, but an ORM that opens and closes a transaction around every query turns COMMIT</code> into one of your highest-calls</code> rows: a content-free verb sitting at the top of any ranking by call count, ahead of the queries you actually want to see. If your top-by-calls view is mostly transaction control, that's the signal to turn track_utility</code> off. I/O timings need a second switch. The shared_blk_read_time</code> and shared_blk_write_time</code> columns are only populated when the server-wide track_io_timing</code> is on, and it's off by default. Leave it off and those columns read zero, making every query look like it did no I/O, which is a quietly misleading kind of empty. Why track_planning</code> in particular hurts</a> </h3> The warning on track_planning</code> is worth unpacking, because the obvious objection is the one the manual seems to skip. If a hot query gets executed by thousands of backends billions of times and that's fine, why would planning the same shape be the thing that tips over? The answer is in how the counters get written. Every entry has its own spinlock (entry->mutex</code>), and updating the entry means grabbing that lock. The update happens once per phase. The extension writes the entry at the end of planning and again at the end of execution, in two separate critical sections. With track_planning</code> off, a query touches its entry's spinlock once per call, at exec-end. With it on, it touches the same lock twice, once at plan-end and once at exec-end. For a shape that lots of connections run at once, that single entry's lock is already the hottest in the system, and doubling the grabs roughly doubles the contention on it. So it isn't that planning is contended and execution is free. Execution-side contention on that per-entry lock is itself a well-known bottleneck for ultra-hot identical queries. It's that track_planning</code> adds a second grab of the already-hot lock to every call, and piles that extra traffic onto exactly the entries that were most contended to begin with (one shape, run everywhere). That's why a setting that sounds harmless, "also record planning time," ships off by default, and why you should leave it off on any workload dominated by a handful of extremely hot shapes. Mind the column renames (PG 17)</a> </h3> PostgreSQL 17 renamed the I/O timing columns. blk_read_time</code> became shared_blk_read_time</code> and blk_write_time</code> became shared_blk_write_time</code>, with new local_blk_*</code> columns added alongside. A query selecting the old names errors after the upgrade; a SELECT *</code> dashboard silently shifts. See the PG 17 release notes</a>. </div> If you have dashboards or scrapers older than PostgreSQL 17, they may have broken on the upgrade without saying so. The same release also added the stats_since</code> and minmax_stats_since</code> timestamps and a minmax_only</code> argument to pg_stat_statements_reset()</code>. The view's shape isn't a stable contract across majors, so pin your column lists and re-check them on every upgrade. Who can see what</a> </h2> One security corner to know before you wire this into a shared dashboard. The statistics are visible to everyone, but the SQL text and queryid of other users' queries are not: For security reasons, only superusers and roles with privileges of the pg_read_all_stats</code> role are allowed to see the SQL text and queryid</code> of queries executed by other users. Other users can see the statistics, however, if the view has been installed in their database. </blockquote> So a low-privilege monitoring role sees rows full of <insufficient privilege></code> where the query text should be. The fix is to grant pg_read_all_stats</code>, a built-in role, to the monitoring user. Not superuser. If your "the dashboard shows no query text" mystery has a boring cause, this is usually it. So what is it good for, and where does it leave you?</a> </h2> None of this is an argument against pg_stat_statements</code>. It's still the first thing to turn on, and for its real job, ranking your query shapes by total time, calls, or I/O on one node since the last reset, nothing else is this cheap or this immediate. Reach for it first, every time. Just be clear about what you're holding: It counts, it doesn't record. Cumulative totals, no time dimension. Snapshot and diff it yourself if you want history.</li> It gives you an average, not a spread. Great for "which shape dominates total time," useless for "what's my p99."</li> It's shape-fragmented. ORMs explode one query into many entries, and pulling them back together is a job for a normalizer on top.</li> It keeps first-seen text. Static comment tags survive, dynamic per-request tags don't, so it can never be an APM join target.</li> It's lossy under pressure. A full table evicts your tail. Watch dealloc</code>.</li> It's plan-blind. It tells you a query is slow, never why. No plan, no estimate error, no spills.</li> It's single-node. The replica fleet that serves most of your reads is invisible from the primary.</li> </ul> So pg_stat_statements</code> answers one question, what is slow?, and stops. The next questions are outside its model: why it got slow (a plan flip, stats drift), who is calling it, where across the cluster, and what would happen if you changed something. That isn't a missing feature; it's what a pile of aggregated counters fundamentally can't do. The per-execution detail isn't truly gone, it's in the logs. log_min_duration_statement = 0</code> or auto_explain</code> records every call with its parameters, tags, and timing, which is where the full distribution and the dynamic tags live. The catch is cost: you can't log every query on a busy system, which is the whole reason the cheap aggregate exists.</div> That answers the question we opened on. Postgres doesn't ship the query store you came looking for, and pg_stat_statements</code> isn't it. It's the floor you'd build one on, not the thing itself. So stop reading pg_stat_statements</code> as a picture of your workload. Read it for what it is: a cheap, lossy box of counters that tells you where to point the real instruments next. pg_stat_statements: everything it tells you 2026-06-02T22:00:00+00:00 If not first, pg_stat_statements</code> is one of the most used extensions in the PostgreSQL ecosystem. It ships in contrib and costs almost nothing to use. Most of us turn to it to answer the question: what is the database actually doing? It's genuinely useful. You can use it to get a snapshot of what happened in a given timeframe, and make a faster decision about what to fix. Coming from other database engines, you might reach for it expecting something a bit more, a query store. The built-in feature that keeps normalized queries and their plan history. Except pg_stat_statements</code> is not this. This article is going to deep dive into what the extension really provides. Because once you lean on it, you might start noticing the rough edges. The gap comes from what "a query store" might come with and what it actually tells you. The very same query might show up in many separate rows. Discrepancies between what your monitoring says and what mean_exec_time</code> shows. Missing queries. Numbers that changed overnight. None of that is a bug. It all follows from what pg_stat_statements</code> really is: a fixed-size hash table of running counters, kept in shared memory, keyed by a hash of your parse tree. It counts; it does not record. Hold that one idea in your head and every surprising thing on the list above stops being surprising. Everything below is reproducible. Paste this into a scratch database: CREATE TABLE customers ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL ); CREATE TABLE orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', created_at timestamptz NOT NULL DEFAULT CURRENT_TIMESTAMP ); INSERT INTO customers (name) SELECT 'Customer ' || i FROM generate_series(1, 2000) AS i; INSERT INTO orders (customer_id, amount, status, created_at) SELECT (random() * 1999 + 1)::int, (random() * 500 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], '2024-01-01'::date + (random() * 700)::int FROM generate_series(1, 100000); ANALYZE;</code></pre> And turn the extension on: -- in postgresql.conf config -- shared_preload_libraries = 'pg_stat_statements' (requires restart) CREATE EXTENSION pg_stat_statements;</code></pre>What it actually stores</a> </h2> We've just turned it on, so start with what it puts in front of you. SELECT * FROM pg_stat_statements;</code></pre> You get one row per distinct query shape and a set of counters behind it: calls</code>, total_exec_time</code>, rows</code>, shared_blks_hit</code>, shared_blks_read</code>, and a lot more. While there is a query</code> column, you might think it keeps your SQL. But it does not. If you look carefully query text is normalized, with the literals stripped out (WHERE id = 42</code> is stored as WHERE id = $1</code>), and there is exactly one row per shape no matter how many times it ran. The counters only add up. A statement runs, the counters for its shape tick up, and that individual execution, its values, its timing, its plan, is gone. Each row is an entry: a running total per shape, not a record of executions. Most of the limitations in this article follow from that. First, where the data lives. The counters sit in shared memory allocated at startup, and as we said, it's a fixed hash map. The number of tracked statements is capped (pg_stat_statements.max</code>, default 5000). The query text lives in a file on disk, $PGDATA/pg_stat_tmp/pgss_query_texts.stat</code>, which the view joins against the in-memory counters when you read it. Each entry in that map is keyed by queryid</code>, and that's where the trouble starts. The queryid</a> </h2> The jumble moved into core in PostgreSQL 14; before that it lived inside pg_stat_statements</code> itself. It's gated by the compute_query_id</code> setting, whose default auto</code> turns it on whenever a module that needs it, such as this one, is loaded.</div> The `queryid` is computed by Postgres core, not by the extension. When Postgres parses and analyzes a statement, it walks the resulting tree and hashes the parts that define the query's structure into single 64-bit number, leaving the constant values out. The Postgres source calls this step the *query jumble*; `pg_stat_statements` just reads the result and uses it as the key for each row. Leaving the constants out is the whole trick. WHERE id = 42</code> and WHERE id = 99</code> jumble to the same number, with the literal shown as a $n</code> placeholder, so one entry can stand in for millions of executions that differ only in their values. SELECT * FROM orders WHERE id = 42; SELECT * FROM orders WHERE id = 99; SELECT * FROM orders WHERE id = 12345; SELECT queryid, query, calls FROM pg_stat_statements WHERE query LIKE 'SELECT * FROM orders WHERE id%';</code></pre> queryid | query | calls ----------------------+------------------------------------+------- -8198406948996196422 | SELECT * FROM orders WHERE id = $1 | 3</code></pre> Three executions, one row. Now look at the queryid: it's negative. It's a signed 64-bit integer, which can easily break a dashboard that stored it in an unsigned column or printed it as a friendly identifier. It's a hash, not a sequence number. Treat it as an opaque token that might be negative and move on. This is also where the surprises start, because the jumble is more sensitive than "it ignores constants" makes it sound. The queryid isn't stable across versions</a> </h3> The manual is blunt about it: it is not safe to assume that queryid</code> will be stable across major versions of PostgreSQL. </blockquote> The hashing process is also sensitive to differences in machine architecture and other facets of the platform. </blockquote> If you followed the previous section carefully, you'll have spotted that the hash comes from the shape of the parsed tree, not the query text. Upgrade from Postgres 16 to 17, or move to a different CPU architecture, and identical SQL can hash to a different queryid. Anything that adds up query costs across a fleet by joining on queryid quietly falls apart the moment the fleet isn't uniform. The docs call out replicas specifically: logical replication schemes do not promise to keep replicas identical in all relevant details, so queryid</code> will not be a useful identifier for accumulating costs across a set of logical replicas. </blockquote> Don't treat queryid</code> as a durable key. Within one server it's stable across minor versions, and that is the only guarantee. Across a major upgrade, a different CPU architecture, or the replicas in a logical-replication set, the same SQL can hash to a different value. Any dashboard or rollup that joins on queryid</code> across machines is one upgrade away from quietly double-counting or dropping rows. </div> The queryid tracks the OID, not the name</a> </h3> The jumble uses the OIDs of objects a query touches, not their names. Two behaviors fall out of that, both straight from the manual, and both catch people off guard: pg_stat_statements</code> will consider two apparently-identical queries to be distinct, if they reference for example a function that was dropped and recreated between the executions of the two queries. </blockquote> Conversely, if a table is dropped and recreated between the executions of queries, two apparently-identical queries may be considered the same. </blockquote> Drop and recreate a table, which happens all the time during migrations, dump-and-restore, or a blue-green cutover, and every query against it gets a fresh queryid, because the table now has a fresh OID. Your old stats for that table are stranded under the old id, looking like queries that simply stopped running. The dead query on your dashboard isn't dead; its OID moved. search_path</code> hides the same trap. SELECT * FROM orders</code> resolves to tenant_a.orders</code> on one connection and tenant_b.orders</code> on another, so a schema-per-tenant app scatters one logical query across every tenant's OID. Correct behaviour, but rolling up "all the reads against orders</code>" means resolving OIDs yourself.</div> Aliases and structure leak in too</a> </h3> if the alias for a table is different for otherwise-similar queries, these queries will be considered distinct. </blockquote> That last one, alias sensitivity, is what hurts ORM users most. The ORM row explosion</a> </h2> The jumble normalizes constants. It does not normalize structure. Two queries you'd read as "the same query" but that differ in the parsed tree, say a different column list, a table alias, an extra LIMIT</code>, or a reordered AND</code>, are different shapes with different queryids in different rows. ORMs generate these variations constantly. Here are four queries an ActiveRecord, Hibernate, or Ecto app might emit for what is really one operation, "load a customer by id": SELECT id, name FROM customers WHERE id = $1; -- queryid A SELECT id, name, created_at FROM customers WHERE id = $1; -- queryid B (.select added a column) SELECT c.id, c.name FROM customers c WHERE c.id = $1; -- queryid C (aliased) SELECT id, name FROM customers WHERE id = $1 LIMIT $2; -- queryid D (.first)</code></pre> Four rows. The calls split four ways, and each one looks minor even when the real operation behind them might be your single hottest path. The mean_exec_time</code> is per-row, so a slowdown that hits all four equally gets diluted across all four. ORMs don't stop at four. Every conditional .where</code>, every optional .select</code>, every eager-load is another shape; one logical query spreads across dozens or hundreds of entries. You can see real production snapshots where a single logical query lands under hundreds of distinct queryids, each looking minor on its own. This is the job of a tool like qshape</a>, part of the boringSQL stack.</div> The extension can't fix this, and it isn't supposed to. The jumble runs before anything understands that "these read different columns from the same table for the same reason." Clustering the variants back together means re-parsing each entry's text, canonicalizing the tree (strip aliases, sort AND</code> predicates, spot column-set supersets), and re-fingerprinting. pg_stat_statements</code> hands you the raw, fragmented shapes; reassembling them into logical queries is a layer it doesn't have. A downstream re-parse can work because normalization only looks like it threw the information away. The jumble strips one thing, the constant values (= 42</code> becomes = $1</code>), and leaves all the structure in the query</code> text: aliases, column list, the AND</code>/OR</code> tree, LIMIT</code>, join order. That structure is everything a canonicalizer needs to see that WHERE a = $1 AND b = $2</code> and WHERE b = $1 AND a = $2</code> are the same predicate, or that c.id</code> and id</code> are the same column. The extension over-normalizes constants and under-normalizes structure. One thing genuinely is gone, though: anything that was evicted. A canonicalizer only sees the rows currently in the view, so if your ORM has flooded the hash table and dealloc</code> has dropped the tail, there's nothing left to fingerprint. Keep cardinality under control at the source first, by raising pg_stat_statements.max</code>, getting on PG 18, or switching to array binding (more on that next), or you'll just be fingerprinting whichever rows happened to survive, not your real workload. The IN-list special case, and its PG 18 fix</a> </h3> There's one structural variation Postgres historically handled badly: lists of constants. SELECT * FROM orders WHERE id IN (1, 2, 3); SELECT * FROM orders WHERE id IN (1, 2, 3, 4); SELECT * FROM orders WHERE id IN (1, 2, 3, 4, 5);</code></pre> Before PostgreSQL 18, each of those was its own entry, because the length of the list changed the parsed tree. An app that builds IN</code> clauses on the fly, which is most apps, could burn through thousands of entries with a single query shape and evict everything else on the way (more on eviction below). PostgreSQL 18 finally collapses them. The three statements above, differing only in list length, now land in a single entry: query | SELECT * FROM orders WHERE id IN ($1 /*, ... */) calls | 3</code></pre> PostgreSQL 18: IN</code>-lists squashed. A constant list that differs only in length now collapses to one entry, shown as IN ($1 /*, ... */)</code>. It's unconditional, with no GUC to turn on. The limit: it only fires for inline literals, so a driver that binds IN ($1, $2, $3)</code> as parameters still gets a fresh entry for every list length. </div> That's a real improvement, and a good reason to be on 18. Two things keep it from being the whole answer. Most production clusters are still on 17 or earlier, where the explosion is alive and well. And the parameter-binding drivers it skips, JDBC and friends, are often the worst offenders: they send IN ($1, $2, $3)</code> as bind parameters, so the values were never constants in the parsed tree and there is nothing to squash. IN ($1,$2,$3)</code> and IN ($1,$2,$3,$4)</code> stay separate entries, one new shape per list length. The squashing briefly sat behind a GUC, query_id_squash_values</code>, during PG 18 development. It was removed before release</a>, so the behaviour is now always on.</div> If you're on one of those drivers, you're stuck regardless of your Postgres version, but there's a real fix, and it beats the PG 18 squashing because it works everywhere. Stop sending IN</code>-lists and send an array. Rewrite this: SELECT * FROM orders WHERE id IN ($1, $2, $3, ... $n); -- n params, n distinct shapes</code></pre> as this: SELECT * FROM orders WHERE id = ANY($1); -- one param, one shape, forever</code></pre> = ANY(array)</code> takes a single parameter, one array value, so the parsed tree is identical whether the array holds 3 elements or 3,000. That's one queryid, on every Postgres version and every driver, prepared or not, because the array length is never part of the structure the jumble sees. Most drivers and ORMs support array binding directly (setArray</code>, ARRAY[...]</code>, pgx's slice binding, ActiveRecord's where(id: array)</code>), and the planner uses it for index selection much like an IN</code>-list. For an ORM-heavy team it's the highest-leverage way to keep pg_stat_statements</code> readable. The first-seen text is frozen</a> </h2> Every entry shows one query text, and it's the first one that created the entry. Not the most recent, not the most common, the first. After that the counters keep climbing on every matching execution, but the text on disk never changes, at least until the entry gets evicted and rebuilt. Most of the time that's fine, because the constants are normalized to $1</code> anyway. It stops being fine the moment your queries carry comments, and modern apps carry a lot of them. The sqlcommenter</a> and marginalia conventions tack structured key-value context onto every query: SELECT id, name FROM customers WHERE id = $1 /*application='checkout',controller='orders',action='show'*/</code></pre> Comments aren't part of the parsed tree, so they don't change the queryid. Every execution still maps to one entry. But the text on that entry is whatever showed up first. That splits comment tags into two kinds, and the split is the whole story: Tag class</th> Examples</th> What pg_stat_statements</code> does with it</th></tr></thead> Static(same value every call from one place)</td> application</code>, controller</code>, action</code>, job</code>, framework</code></td> Survives. The first-seen value is representative, because every call from that spot carries the same value. You can scrape it from the query</code> text and trust it.</td></tr> Dynamic(a new value every request)</td> traceparent</code>, request_id</code>, trace_id</code>, tenant_id</code>, user_id</code></td> Lost. The first request's value is frozen into the text forever, and every later per-request value is silently dropped.</td></tr> </tbody></table> That second row is why pg_stat_statements</code> can never be a join target for an APM or tracing tool. You can't ask it "show me the queries from the trace that timed out at 14:32," because it kept exactly one traceparent</code>, the first it ever saw, and threw the rest away. A counter is a sum, and a sum has no room to remember which request contributed what. If you need dynamic tags, you need a per-execution capture path running alongside the extension (auto_explain, or a hook-based harvester), not the extension itself. A traceparent</code> in the pg_stat_statements</code> text is a warning sign, not a feature: a per-request value reached the entry first and is now frozen there as if it were static context. If you scrape tags from these texts into a catalog, drop the hex-id-shaped ones. The averages lie to you</a> </h2> This is the limitation that catches the most people, because the numbers look so trustworthy. Every counter is cumulative since the last reset. There's no time dimension inside the extension at all. The view is one snapshot of running totals, and that has two consequences that feed each other. There's no history built in. Say you want to know what got slower this week. The obvious move is to read mean_exec_time</code>, and that's exactly where it goes wrong. That number is the average over all time since the last reset, so on a cluster that's been up for months it blends a cold-cache Monday morning, the nightly batch job, and a quiet Sunday into one figure that describes no real moment. To get a "this week" number you have to do the bookkeeping yourself: snapshot the view on a schedule, keep each snapshot in a table, and subtract two of them. The average over the last hour is just (total_exec_time_now - total_exec_time_then) / (calls_now - calls_then)</code>. The view hands you a running total; turning it into a rate is on you. That subtraction is less safe than it looks. It quietly assumes the "now" counters carry on from the "then" counters, that they only ever climb. Usually true, not always. Eviction and resets, both covered in part two</a>, break that assumption, and a snapshot diff you'd actually trust has to handle three cases: The entry was evicted and rebuilt. If a query drops out of the table and later runs again, it comes back with counters at zero. Now total_exec_time_now</code> is smaller than total_exec_time_then</code>, and the subtraction gives you a negative delta. Don't read that as "the query got faster." Read it as "this window is invalid for this queryid," because the entry's history was cut.</li> A global reset happened between snapshots. Someone ran pg_stat_statements_reset()</code>, or a deploy script did. Every row's "now" is smaller than its "then." This one is easy to catch: compare pg_stat_statements_info.stats_reset</code> against the time of your earlier snapshot, and if the reset is newer, the whole window crossed a reset and the entire diff is garbage.</li> The queryid is on only one side. A query in "then" but missing from "now" was evicted, or its table was dropped and recreated (new OID, new id). A query in "now" but not "then" is new or rebuilt. Join the two snapshots with a full outer join, and treat the rows that don't match on both sides as events, not as zeros to subtract from.</li> </ul> PostgreSQL 17 adds a cleaner per-entry guard: every row carries stats_since</code>, the moment the entry was created or recreated. If a row's stats_since</code> is newer than your earlier snapshot, its counters reset under you, so discard that window no matter what the subtraction says.</div> Before PG 17 the global stats_reset</code> is your only built-in signal, and per-entry eviction is invisible except through the counters-went-backwards trick. The bare formula is right; the monitoring layer around it is mostly this bookkeeping, and skipping it is how homegrown dashboards end up reporting impossible negative latencies. The average hides the spread. Even inside one window, the extension only keeps four numbers per shape: min</code>, max</code>, mean</code>, and stddev</code> of exec time. It builds them on the fly, folding each execution into running aggregates and then dropping the timing, so there's no histogram, no percentiles, no per-call record to go back to. A query that runs in 1ms 99% of the time and 2 seconds the other 1% has a mean_exec_time</code> around 21ms, a number that matches none of its real executions and hides the p99 that's paging your on-call. Two-humped latency, a common shape for real performance problems, just disappears into an average and a standard deviation. The extension can tell you a query is slow on average, not that it's sometimes catastrophically slow, which is usually the more dangerous case. The mean is just total_exec_time / calls</code>, but the extension maintains it (and the variance behind stddev</code>) with Welford's online algorithm: each execution folds into a running mean and sum-of-squares, then its timing is dropped. An average and a standard deviation are all that survive, never a percentile, because no samples are kept to rank.</div> If you want percentiles or a latency timeline, this is the wrong tool, and querying it harder won't help. It dropped the spread when it wrote the row. That's the first half of the story: everything pg_stat_statements</code> tells you, and the ways it quietly bends what it reports. One logical query fragmented into hundreds of shapes, the text frozen at first sight, the whole distribution flattened into a single average. The other half is what it never records at all: the entries it silently evicts, the query text that can vanish in one stroke, the plans it can't see, and the replicas it doesn't know exist. That's part two</a>, which also returns to the question we opened on: is this the query store Postgres is missing, or just the floor you'd build one on? Welcome to ORDER BY jungle 2026-05-15T11:43:00+02:00 SQL is fun and not at all boring. The latest article by Markus Winand on Order by Has Come a Long Way</a> sent me on quite a journey. First, set up a table called nums</code> with one integer column and four rows: CREATE TABLE nums (a int); INSERT INTO nums VALUES (0), (1), (2), (3);</code></pre> Try to guess what these two queries return. SELECT -a AS a FROM nums ORDER BY a; SELECT -a AS a FROM nums ORDER BY -a;</code></pre> Most of us would guess the same rows in a different order. The actual answer is that they produce exactly the same rows in exactly the same order. By the same logic you might expect SELECT a AS c FROM nums ORDER BY -c;</code></pre> to do exactly the same. Except it does not. It errors with column "c" does not exist</code> despite the alias being right there in the statement. Welcome to ORDER BY jungle. Names and expressions are not the same</a> </h2> If you ask most developers how ORDER BY</code> works, they will say "you put a column name there and it sorts the rows". In 99% of queries that is exactly what happens. People sort by created_at</code> or id</code> and move on. Strictly speaking, three, if you count ORDER BY 1</code>. Positional references are their own can of worms and out of scope for this post.</div>But `ORDER BY` accepts two different kinds of things: SELECT created_at, user_id FROM events ORDER BY created_at; SELECT created_at, user_id FROM events ORDER BY date(created_at);</code></pre> Both feel natural. And the thing nobody tells you is that they go down completely different code paths in the parser. Different scope rules, different lookups, different error messages. The first looks at your SELECT</code> list. The second looks at your FROM</code> clause. They never look at the same place. Same answer, two different sorts</a> </h2> Look at the first query again. SELECT -a AS a FROM nums ORDER BY a;</code></pre> You wrote ORDER BY a</code>. A bare identifier, no decoration. Postgres goes down the name path. It scans the SELECT</code> list for something called a</code>, finds the aliased column -a AS a</code>, and sorts by its output values. The negated values are -3, -2, -1, 0</code>, ascending is -3, -2, -1, 0</code>. That is what comes out. Now the twin. SELECT -a AS a FROM nums ORDER BY -a;</code></pre> You wrote ORDER BY -a</code>. This is no longer an identifier. It's an expression: unary minus around a column reference. The parser does not even try the same logic. Instead it switches to the expression path, where the only a</code> it knows is the column in nums</code>, and sorts the input values negated. And by arithmetic luck, the two queries land on the same row order. Same output, completely different logic. If you don't believe it is just luck, drop the negation from the SELECT</code> list and keep it in ORDER BY</code>: SELECT a AS c FROM nums ORDER BY -a;</code></pre> c --- 3 2 1 0 (4 rows)</code></pre> ORDER BY -a</code> is an expression, so it sorts by -input_a</code> ascending, which is input_a</code> descending. The alias c</code> was never consulted. The result has nothing to do with whatever c</code> happens to be. And ORDER BY -c</code> is now obvious. -c</code> is an expression, so the parser looks for column c</code> in FROM</code>, doesn't find it, and errors. The alias exists, but in a scope this code path cannot see. Above the identifier, or around it</a> </h2> Once the rule is clear (bare identifier hits the SELECT</code> list, anything else hits the table) the rest of the surprises fall out. SELECT 'hello' AS x FROM nums ORDER BY x::text; -- ERROR: column "x" does not exist</code></pre> It is probably not surprising that casts count as expressions and push the lookup to the table. The surprise might come with SELECT a AS c FROM nums ORDER BY c DESC NULLS FIRST;</code></pre> Which will work as expected. Both DESC</code> and NULLS FIRST</code> are part of the sort clause itself, not of the sort expression. They sit above the identifier in the parse tree, so they never touch it. The parser still sees a bare c</code>, takes the fast path, finds the alias, sorts by it, and then applies "descending, nulls first" on top of the resolved key. The same cannot be said about collation. SELECT 'A'::text AS x FROM nums ORDER BY x COLLATE "C"; -- ERROR: column "x" does not exist</code></pre> This is a really bad one. COLLATE</code> might look the same as a sort modifier, but it is not. It wraps the expression in the parse tree. Parentheses are a special case. SELECT -a AS a FROM nums ORDER BY (a); -- works, sorts by alias</code></pre> Postgres collapses redundant parens before the bare-identifier check, so (a)</code> is still bare a</code>. The seam is asymmetric in the way that maximises confusion: COLLATE</code> is "still a name to a human, an expression to the parser", and (a)</code> is "an expression to a human, still a name to the parser". You get both flavours of wrong intuition mixed here. Unary plus. +a</code> and a</code> evaluate to the same value, but they do not parse to the same node. SELECT -a AS a FROM nums ORDER BY a; SELECT -a AS a FROM nums ORDER BY +a;</code></pre> A plus sign you would not even think about changes which rows come out in which order. The parser stores a column reference as a list of name parts: one part when it is unqualified, two or more once you add a table or schema. The fast path only fires on lists of length one.</div> Finally, schema- and table-qualified references. ORDER BY nums.a</code> looks like an identifier, but it is not. SELECT -a AS a FROM nums ORDER BY a; SELECT -a AS a FROM nums ORDER BY nums.a; </code></pre> a ---- -3 -2 -1 0 (4 rows) a ---- 0 -1 -2 -3 (4 rows)</code></pre>Aliases that aren't the names you think</a> </h2> Here is one that cost me an afternoon once. Easy to come across once an ORM or a generated view declared the alias for you. SQLAlchemy, Hibernate, jOOQ, and most code generators quote anything that isn't pure lowercase. Two queries, identical except that the alias is quoted in one. Two different result sets. SELECT -a AS A FROM nums ORDER BY a; -- sorts by alias (-3,-2,-1,0) SELECT -a AS "A" FROM nums ORDER BY a; -- sorts by input (0,-1,-2,-3)</code></pre> The bare-identifier check compares names with strcmp</code>. Unquoted A</code> folds to lowercase a</code> and matches. Quoted "A"</code> preserves case, stays A</code>, and does not match the lowercase a</code> in the ORDER BY</code>. The lookup fails, the parser falls through to the expression path, the expression path finds the column a</code> in nums</code>, and the query runs successfully while doing something different from what you meant. GROUP BY checks the opposite scope first</a> </h2> Both GROUP BY</code> and ORDER BY</code> accept a bare identifier, and both can resolve it either way: to a table column or to a SELECT</code>-list alias. The difference is the order they check: ORDER BY a</code> looks at the SELECT</code> list first, then the table.</li> GROUP BY a</code> looks at the table first, then the SELECT</code> list.</li> </ul> For most queries this never matters. The two clauses end up picking the same thing because nothing is shadowed. The surprise happens when an alias has the same name as a base column but a different value: SELECT a/2 AS a, count(*) FROM nums GROUP BY a ORDER BY a;</code></pre> Now the two clauses disagree about what a</code> means. GROUP BY a</code> picks the input column (four distinct values, four groups, one row each). ORDER BY a</code> picks the alias, which is a/2</code>. The result has four rows because the grouping was on a finer-grained key than the projection: a | count ---+------- 0 | 1 0 | 1 1 | 1 1 | 1</code></pre> Two rows where a/2 = 0</code> (from input 0</code> and 1</code>), two where a/2 = 1</code> (from input 2</code> and 3</code>). The duplicates are real. The same identifier means two different columns in two adjacent clauses of one query. Window ORDER BY does not even pretend</a> </h2> This one trips people up because it does not look like a different clause. SELECT a, -a AS neg, row_number() OVER (ORDER BY neg) FROM nums; -- ERROR: column "neg" does not exist</code></pre> OVER (ORDER BY ...)</code> is a different parse path entirely. It does not check the targetlist at all, only the FROM scope. The bare-name fast path simply does not exist here. SELECT a, -a AS neg, row_number() OVER (ORDER BY -a) FROM nums; -- this works</code></pre> Two ORDER BY</code> clauses in the same query, two different scoping rules. UNION ORDER BY is name-only</a> </h2> When ORDER BY</code> follows a UNION</code>, neither path is fully open. -- ok (SELECT a FROM nums) UNION ALL (SELECT 9) ORDER BY a; -- ERROR (SELECT a FROM nums) UNION ALL (SELECT 9) ORDER BY -a; -- ERROR (SELECT a FROM nums) UNION ALL (SELECT 9) ORDER BY a COLLATE "C";</code></pre> The error message is unusually helpful: Only result column names can be used, not expressions or functions. HINT: Add the expression/function to every SELECT, or move the UNION into a FROM clause. </blockquote> Set operations do not have a single FROM scope to fall back to, so the expression path is closed entirely. Bare names or nothing. The seam, in the source</a> </h2> Full disclosure: I got this section wrong three times before Claude Code helped me trace the actual parse tree. Lack of sleep from a whole night of geeking out over ORDER BY</code> is the other plausible explanation.</div>Open `src/backend/parser/parse_clause.c` and find `findTargetlistEntrySQL92`. It is forty lines of comment, two `if` blocks, and a final `return`. SQL92's two resolution rules are tried first; SQL99 is the fallback. Block one: the bare-name path. The gate is a ColumnRef</code> node with exactly one name part, and that part must be a string identifier (not *</code>, which is also a ColumnRef</code> but with an A_Star</code> field). If the node passes, the function walks the target list looking for a non-resjunk</code> entry whose resname</code> equals the identifier. The loop keeps going past the first match to detect ambiguity: identical expressions are fine (this is why SELECT a, a FROM nums ORDER BY a</code> works), different expressions error out. On a unique match, return. If the loop finds nothing, the block does not return. Control falls through. This is the case behind the quoted-alias surprise earlier in the post: AS "A"</code> stores resname = "A"</code>, ORDER BY a</code> looks up resname = "a"</code>, the strcmp</code> fails, and the function moves on as if no SQL92 fast path applied. GROUP BY</code> is the small exception inside this block. The name is first tested against the FROM</code> scope, and a hit there causes the targetlist loop to be skipped. That is how GROUP BY</code> ends up preferring the input column. Block two: the positional path. The gate is IsA(node, A_Const)</code>. A non-integer constant errors immediately ("non-integer constant in ..."), which catches ORDER BY NULL</code>, ORDER BY 'a'</code>, ORDER BY TRUE</code>. An integer is used as a 1-based position into the non-resjunk</code> target list; anything outside the range errors as "position %d is not in select list". Block two never falls through. Both 1</code> and -1</code> arrive here as integer A_Const</code>s. doNegate</code> in the grammar folds '-' Iconst</code> into a single integer constant before the function ever runs, so ORDER BY 1</code> and ORDER BY -1</code> go through the same code, with only the integer value (and the result of the position lookup) differing. The fallthrough. Anything not caught above reaches the last line: /* * Otherwise, we have an expression, so process it per SQL99 rules. */ return findTargetlistEntrySQL99(pstate, node, tlist, exprKind);</code></pre> That is the seam. SQL92 succeeds in two narrow shapes: a bare identifier with a matching alias, or an in-range positive integer. Everything else, including a bare identifier whose alias lookup found nothing, becomes a SQL99 expression resolved against FROM</code>. A useful workaround</a> </h2> If you want the alias inside an expression in ORDER BY</code>, the portable trick is to wrap the query in a subselect: SELECT * FROM (SELECT -a AS x FROM nums) s ORDER BY x + 0;</code></pre> Now x</code> is a real column in the FROM scope of the outer query. The expression path finds it. The seam has been moved out of the way. This is, conceptually, what you would want the engine to do for you when you write ORDER BY x + 0</code> directly. The SQL-99 standard does not actually require that, though, and Postgres (along with SQL Server) documents explicitly that an alias inside an ORDER BY</code> expression is not supported. So you do it by hand. The boring takeaway</a> </h2> Most of the time none of this matters. You sort by a column you just selected, the alias and the input column have the same name and the same value, and either parser path gives the same answer. The seam is invisible. The minute the alias and the input column disagree in expression, value, case, or anything wrapped around the identifier, the parser picks one or the other silently, by a rule older than most working programmers. There are two parsers. The bare-name path is SQL-92, the expression path is SQL-99, and they were stitched together in the late 1990s. They still disagree about which scope your identifiers live in, and knowing which one you triggered tells you which scope. If after reading this post you still have to stop and think for a minute before predicting what SELECT -a AS a FROM nums ORDER BY a COLLATE "C";</code></pre> does, that is the right reaction. It means you have the mental model. The opening puzzle queries are from Jamie Brandon's comment on the Lobsters thread</a> discussing Markus Winand's history of ORDER BY</code></a> on modern-sql.com. Everything that follows here is the explanation that comment did not give. Both pieces are worth reading on their own. Strong views on PostgreSQL VIEWs 2026-05-10T00:00:00+00:00 VIEWs should be the cleanest abstraction SQL, and therefore Postgres, has on offer. I love the concept. The promise of decoupling logical intent from physical storage is perfect on paper. In practice, few things in the database world trigger such a heated debate or carry as much historical baggage. VIEWs mix big promises with false hopes, and the promises rarely survive contact with production. The appeal is straightforward. Abstract "active customer" once and reuse it everywhere. Every query, report and dashboard uses the same definition. The "active customer" then becomes the foundation of a "customer orders" view, which in turn powers an operational "customer summary" view. -- layer 1: who counts as an active customer? CREATE VIEW active_customers AS SELECT c.* FROM customers c WHERE c.deleted_at IS NULL AND c.status = 'active' AND c.last_login_at > now() - interval '90 days'; -- layer 2: active customers with their recent orders CREATE VIEW customer_orders AS SELECT ac.*, o.id AS order_id, o.total_cents, o.created_at AS ordered_at, o.status AS order_status FROM active_customers ac LEFT JOIN orders o ON o.customer_id = ac.id WHERE o.created_at > now() - interval '12 months' OR o.created_at IS NULL; -- layer 3: one row per customer, ready for the dashboard CREATE VIEW customer_summary AS SELECT co.id, co.email, co.name, COUNT(co.order_id) AS orders_12mo, COALESCE(SUM(co.total_cents), 0) AS revenue_12mo_cents, MAX(co.ordered_at) AS last_order_at, COUNT(*) FILTER (WHERE co.order_status = 'refunded') AS refunds_12mo FROM customer_orders co GROUP BY co.id, co.email, co.name;</code></pre> Each layer has one job. "Active customer" is defined exactly once - if marketing changes the ninety-day rule tomorrow, it is one line in one place, and the dashboard query collapses to SELECT * FROM customer_summary WHERE id = $1</code>. VIEWs also have the potential to be a real security boundary. Three hand-written queries means three places where a predicate can be forgotten, and three subtly different result sets a quarter later. With one definition, you have one result set. For simple views, there is no performance penalty either. PostgreSQL inlines them directly into the calling query, so the planner sees through the view and plans as if you had written the underlying SQL by hand. That mechanism, and the cases where it stops working, has a deep-dive of its own</a>. This article assumes it as background and focuses on the other side: what views are made of, and what happens when the schema underneath them moves. VIEWs are just like tables</a> </h2> The trouble starts with the syntax. SELECT * FROM customer_summary</code> reads exactly like SELECT * FROM customers</code>. The catalog reinforces it: pg_class.relkind = 'v'</code> instead of 'r'</code>, but everything else (column lists, grants, comments, even \d</code> output) looks the same. So developers reach for them the same way: joining them, nesting them, wrapping them in other views without thinking about it. A table is a heap with indexes. A view is a stored parse tree that gets expanded into your query at planning time. Most of the time the planner is clever enough that the difference does not matter: simple views are inlined, predicates are pushed down, the resulting plan is what you would have written by hand. The nested view spiral</a> </h3> In practice, what you get is the dependency tree from the introduction (customer_summary</code> on top of customer_orders</code> on top of active_customers</code>) and a new engineer who has no way to know that SELECT * FROM customer_summary WHERE id = $1</code> expands into a three-layer rewrite, a LEFT JOIN</code> against twelve months of orders, and a GROUP BY</code> that the planner cannot push the id</code> predicate through. Nothing in the query, the schema browser, or the ORM signals this. You only learn the shape of the tree by reading every definition top to bottom, which is exactly the work the abstraction was meant to save you. You cannot reason about the performance of a query against a view without reading the view, the views it depends on, and the tables underneath. Views are rewrite rules</a> </h3> A view is a macro. When you reference one, Postgres pastes its body into your query before the planner runs. There is no stored result, no view "object" the executor consults. The pg_class</code> row you see when you query the catalog is an empty shell: it holds the name, the column list, and the grants, but no definition. The definition itself, the SELECT</code> you wrote, is stored separately in pg_rewrite</code>: SELECT ev_class::regclass, ev_type, is_instead FROM pg_rewrite WHERE ev_class = 'active_customers'::regclass;</code></pre> ev_class | ev_type | is_instead ------------------+---------+------------ active_customers | 1 | t</code></pre> That row is the view. ev_type = 1</code> is SELECT</code>, is_instead = t</code> means "replace, do not augment" (the pg_class</code> shell has no rows to read, so substitution is the only option that makes sense). Between the parser and the planner Postgres runs a stage called the rewriter, and the rewriter's job is to walk the parse tree, find references to relations that have rules, and substitute the rule body in place. This way a query like SELECT * FROM active_customers WHERE id = $1</code> arrives at the planner already rewritten: -- what the planner actually sees SELECT * FROM ( SELECT c.id, c.email, c.name, c.status, c.last_login_at, c.deleted_at FROM customers c WHERE c.deleted_at IS NULL AND c.status = 'active' AND c.last_login_at > now() - interval '90 days' ) active_customers WHERE id = $1;</code></pre> Notice that c.*</code> is already expanded into an explicit column list. That happened at CREATE VIEW</code> time and was frozen into the stored parse tree; the rewriter does not re-expand it. From here the planner can usually flatten the subquery into the outer query through a pass called subquery pull-up, leaving id = $1</code> as a predicate next to the others so an index on customers.id</code> is reachable. That flattening is what makes simple views free. When it bails out (because the view body has LIMIT</code>, DISTINCT</code>, an aggregate, a set operation, or a few other shapes) the subquery stays put and outer predicates cannot move past it. The full list of planner barriers and how to spot them in EXPLAIN</code></a> lives in the inlining post. Historically Postgres exposed this same machinery as user-facing CREATE RULE</code>, on the theory that arbitrary query rewriting was a general-purpose feature. It mostly was not. Rules are effectively deprecated outside of views; triggers do the same jobs without the surprises. The rule infrastructure survives because views need it. Once you hold that model, every awkward thing about views falls out of it. Two in particular drive the rest of this article. Columns are referenced by attribute number, not name. The stored parse tree does not remember that it reads customers.email</code>; it remembers that it reads attribute 2 of relation 16385. Rename the column and the view keeps working, because the OID and the position are unchanged. Drop a column in the middle and Postgres refuses upfront, because the stored attribute numbers would no longer line up. Every dependency error later in this article traces back to this. The body is expanded once per reference. Every mention of the view in a query produces an independent copy of its parse tree in the rewritten plan. For a stable expression that is invisible. For a volatile one like random()</code> or clock_timestamp()</code>, each copy evaluates separately: CREATE VIEW v_rand AS SELECT id, random() AS r FROM customers; SELECT a.id, a.r AS r_a, b.r AS r_b FROM v_rand a JOIN v_rand b ON a.id = b.id LIMIT 3;</code></pre> id | r_a | r_b ----+----------------------+--------------------- 1 | 0.84043639656488 | 0.0833458769902089 2 | 0.009846241116064247 | 0.6574000469586228 3 | 0.14667469313524628 | 0.21862693208148087</code></pre> Same view, same row, two different r</code> values. The rewriter expanded v_rand</code> twice, so the rewritten query contains two independent random()</code> calls, and the planner has no reason to share them. You might hit this expecting a.r = b.r</code> and end up rewriting the view to push the volatile call outside. This is also why CREATE OR REPLACE VIEW</code> can only append columns to the end of the column list. Existing attribute numbers must stay stable for every dependent, so anything that would shift them is forbidden. A useful side effect of the same mechanism: views are immune to the search-path attacks that bite SECURITY DEFINER</code> functions. Identifiers are resolved to OIDs at CREATE VIEW</code> time, so a later SET search_path</code> cannot redirect a view to a different table. Functions resolve identifiers per call against the caller's search_path</code> unless you pin it; views never have that exposure. Writable views, the half-kept promise</a> </h3> Funnily enough, I was not aware writable views existed until earlier this year. It came up during a hallway conversation with Ervin Weber</a> and Jesper St John</a> at PgDay Nordic 2026 in Helsinki. Twelve years writing PostgreSQL and the auto-updatable rules had simply never crossed my desk. The original promise was symmetry: if a view looks like a table for reads, it should look like one for writes. Postgres delivers this for the easy case (a view over a single base table, no joins, no aggregates, no DISTINCT</code>) and calls them auto-updatable. You can INSERT</code>, UPDATE</code>, and DELETE</code> through them, and the rewrite rule translates the operation against the underlying table. INSERT INTO active_customers (email, name) VALUES ('alice@example.com', 'Alice'); -- becomes: INSERT INTO customers (email, name, status, last_login_at, deleted_at) -- VALUES ('alice@example.com', 'Alice', DEFAULT, DEFAULT, DEFAULT);</code></pre> Add a join, an aggregate, a GROUP BY</code>, or anything else outside the auto-updatable rules and the write path goes silent. SELECT</code> still works, but INSERT</code> errors and you have to wire up INSTEAD OF</code> triggers by hand. So customer_orders</code> and customer_summary</code> are read-only by accident, not by design, and the rules for what counts are not visible in the view definition itself. You find out at write time, in production. WITH CHECK OPTION</code> rejects any write that would produce a row the view cannot see, with LOCAL</code> checking only this view's predicate and CASCADED</code> (the default) checking every underlying view up the chain. It is how you hand an application a writable, scoped slice of a table without trusting it to enforce the scope itself. Views and row-level security</a> </h3> RLS policies live on tables, not views. For a regular view that matters less than it sounds: the policies still fire when the view's query hits the base table. What determines whether they fire correctly is whose context the view runs under. By default, a view runs as its owner. Superusers bypass RLS unconditionally; table owners bypass it unless FORCE ROW LEVEL SECURITY</code> is set. A view owned by a privileged role is therefore a hole in any RLS scheme, silently, because nothing in the view definition or the catalog signals it. Since PostgreSQL 15 the fix is a single option: CREATE VIEW my_view WITH (security_invoker = true) AS SELECT ... FROM base_table;</code></pre> With security_invoker</code> set, the view runs as the calling user and RLS evaluates against the actual caller. Before 15, the workaround was a dedicated low-privilege view owner plus FORCE ROW LEVEL SECURITY</code> on the base table. Materialized views are a different problem. REFRESH MATERIALIZED VIEW</code> runs as the owner, so the snapshot contains whatever the owner can see. Every user then reads that same materialized data. There is no per-row filtering because the rows are physically stored; security_invoker</code> has nothing to attach to. The architect's stigma</a> </h3> When I first got into databases, views were the thing senior people warned you off. "Don't use views in production", "views are slow", "views hide the real query". You learned to write the SQL inline, even when the same predicate appeared in twenty places, because the alternative was disapproval at the next code review. In Oracle shops the warning had specific roots. The cost-based optimiser of that era struggled to merge predicates through views with non-trivial projections, joins, or set operations, so a clean three-layer view could expand into a plan that scanned everything and filtered late. The shop-wide answer was almost always materialised views or PL/SQL packages that encapsulated every access path; a simple view sat awkwardly between the two, not cached like a matview, not encapsulated like a package, and with optimiser surprises on top. Postgres carried the same baggage by association. Nested-view spirals were real, rule-system surprises were real, and the optimiser of fifteen years ago was less capable than today's. The rule of thumb hardened into doctrine and outlived its conditions. The improvements are real; the reputation has not caught up. Reach for a view in a code review today and someone will still object on principle. Worth knowing before your PR hits review. What happens when you change a table</a> </h2> Everything covered so far stays invisible while the schema underneath holds still. View bodies live in pg_rewrite</code>, columns are pinned to attribute numbers, types are frozen at CREATE VIEW</code> time, and the planner just expands and runs. The moment someone tries to change a column the view tree references, every one of those mechanics surfaces at once. Suppose someone needs to drop the name</code> column from customers</code>: ALTER TABLE customers DROP COLUMN name;</code></pre>ERROR: cannot drop column name of table customers because other objects depend on it DETAIL: view active_customers depends on column name of table customers HINT: Use DROP ... CASCADE to drop the dependent objects too.</code></pre> active_customers</code> references name</code> (the c.*</code> was expanded at view-creation time), customer_orders</code> inherits the dependency through ac.*</code>, and customer_summary</code> is another layer down. The whole tree is frozen on one column. It is not just dropping. Try widening a type: ALTER TABLE customers ALTER COLUMN email TYPE TEXT;</code></pre>ERROR: cannot alter type of a column used by a view or rule DETAIL: rule _RETURN on view active_customers depends on column "email"</code></pre> The same wall blocks any structural change to a column a view references: drop, rename, type change. Even widening VARCHAR(255)</code> to TEXT</code>, a change that loses no data and adds no constraint, is forbidden because the view's stored definition references the old type OID. The hint suggests CASCADE</code>: ALTER TABLE customers DROP COLUMN name CASCADE;</code></pre>NOTICE: drop cascades to 3 other objects DETAIL: drop cascades to view active_customers drop cascades to view customer_orders drop cascades to view customer_summary</code></pre> All three views are dropped, not modified, along with their grants and any other dependent objects. On a schema with dozens of interconnected views, CASCADE</code> is nightmare. Never use CASCADE on production views without a full recreation script ready. CASCADE doesn't modify views, it drops them. All GRANTs, row-level security policies, and downstream dependencies disappear with them. There is no undo. </div> So the manual path is: save every view definition, drop them in reverse dependency order, alter the table, recreate them in forward dependency order, reapply all grants. For three views this is tedious. For thirty views across multiple schemas, it is a full migration project. -- 1. save definitions SELECT pg_get_viewdef('customer_summary', true); SELECT pg_get_viewdef('customer_orders', true); SELECT pg_get_viewdef('active_customers', true); -- 2. drop in leaf-first order DROP VIEW customer_summary; DROP VIEW customer_orders; DROP VIEW active_customers; -- 3. now you can alter the table ALTER TABLE customers DROP COLUMN name; -- 4. recreate in correct order CREATE VIEW active_customers AS SELECT id, email, status, last_login_at, deleted_at FROM customers WHERE deleted_at IS NULL AND status = 'active' AND last_login_at > now() - interval '90 days'; CREATE VIEW customer_orders AS ...; CREATE VIEW customer_summary AS ...; -- 5. reapply grants and so on -- ...</code></pre>The SELECT * trap</a> </h2> You might think SELECT *</code> saves you from column-level dependencies. It does not. It makes things worse. CREATE VIEW all_customers AS SELECT * FROM customers;</code></pre> This looks flexible. PostgreSQL expands SELECT *</code> at view creation time and freezes the result. Check what the database actually stored: SELECT pg_get_viewdef('all_customers'::regclass, true);</code></pre> SELECT id, email, name, status, last_login_at, deleted_at FROM customers;</code></pre> The *</code> was expanded to the columns that existed when the view was created. Now add a column to the base table: ALTER TABLE customers ADD COLUMN phone TEXT; SELECT * FROM all_customers;</code></pre> The phone</code> column is not there. The view still returns the original columns. To pick up new columns, you have to CREATE OR REPLACE VIEW all_customers AS SELECT * FROM customers</code>, which re-expands the *</code> against the current table definition. Drop a column that was in the original expansion and the migration is blocked the same way it would be with an explicit column list, except now the dependency is hidden in the catalog instead of visible in the view body. SELECT *</code> gives the illusion of flexibility while creating the same rigid coupling. This is actually documented behaviour and follows the SQL standard. But it surprises nearly everyone the first time they encounter it, and it is the source of many "my view is missing columns" bug reports. Always use explicit column lists in views. At least then the dependency is visible and the breakage is predictable. The dependency wall, the type-OID coupling, the frozen SELECT *</code> expansion: all of it is the same trade-off, applied consistently. Rigidity is the feature, not a bug. Where this isn't a PostgreSQL bug</a> </h2> The rigidity is a design choice, not a defect. Other databases made different choices, and none of them are free. Oracle marks dependent views as INVALID</code> rather than blocking the DDL. The views are recompiled automatically on next access. If they still work, great; if not, you get an error at query time. This sounds better until you realize what it means in practice: you can deploy a migration, get a clean exit code, and not discover that a critical reporting view is broken until a user hits it on Monday morning. The breakage moves from migration time, where you are paying attention, to runtime, where you may not be. For teams that value deployment confidence, this is arguably worse. SQL Server has sp_refreshview</code>, which recompiles a view's metadata against the current table definitions. You can alter a table and then refresh the dependent views to pick up the changes. But sp_refreshview</code> works on one view at a time. There is no built-in way to refresh an entire dependency chain in the right order. And if a view references a dropped column, sp_refreshview</code> fails; it does not remove the reference for you. You still have to manually edit and recreate the view. It is a convenience, not a solution. PostgreSQL chose compile-time safety: no surprise INVALID</code> views in production, no silently wrong results from a lazy recompilation papering over a structural change. The cost is manual dependency management on every schema change, and it is high enough to push experienced teams away from views entirely. The saving grace: transactional DDL</a> </h2> Before the workarounds, there is one PostgreSQL feature that materially changes the risk calculus: DDL is transactional. The entire drop-alter-recreate-regrant sequence can live inside a single BEGIN/COMMIT</code>: BEGIN; DROP VIEW customer_summary; DROP VIEW customer_orders; DROP VIEW active_customers; ALTER TABLE customers DROP COLUMN name; CREATE VIEW active_customers AS ...; CREATE VIEW customer_orders AS ...; CREATE VIEW customer_summary AS ...; GRANT SELECT ON customer_summary TO reporting; COMMIT;</code></pre> If any statement raises (a typo in a recreated view, a missing grant, an unexpected dependency) the whole migration rolls back and the database is exactly as it was. Oracle and SQL Server cannot do this for most DDL; their CREATE VIEW</code> auto-commits, so a partial failure leaves you with a half-migrated schema and no ROLLBACK</code> to lean on. PostgreSQL's "you have to drop and recreate everything" pain is real, but the recovery story is much better than Oracle's or SQL Server's. This does not solve the locking problem. DROP VIEW</code> and CREATE VIEW</code> take AccessExclusiveLock</code> on the view; ALTER TABLE</code> takes one on the table. For the duration of the transaction, anything that touches those objects waits behind it. A long view-rebuild transaction on a busy system is a stop-the-world window. Keep the transaction tight: prepare the new SQL outside the transaction, hold no other locks, and do not run it during peak traffic. Transactional DDL gives you safety, not concurrency. The workarounds</a> </h2> None of these solve the fundamental problem; they make it manageable. Avoid views entirely. Replace views with application-level query builders or ORM scopes. You lose the shared abstraction: when three services need the same definition of "active customer," each one implements it independently. Definitions drift. For teams with full control over a small codebase, this is a reasonable trade. For shared data platforms, it is not. Script the recreation. Use pg_get_viewdef()</code> to extract definitions, then write migration scripts that drop and recreate views around schema changes. The problems: no impact analysis (you have to figure out the dependency order yourself), no grant preservation (view definitions do not include permissions), and it does not scale past a handful of views. Your migration framework will not help here either. They run your SQL files in order, but figuring out what SQL to write is entirely on you. Version your views. Instead of modifying views in place, create active_customers_v2</code> alongside active_customers_v1</code>. Migrate consumers one at a time. Drop the old version when nothing uses it. This works well when views are consumed by multiple independent teams, since you cannot force everyone to migrate in the same deployment window. The trade-off is naming discipline and version proliferation: if customer_orders_v1</code> depends from active_customers_v1</code>, you now need customer_orders_v2</code> too. Schema-based versioning (CREATE SCHEMA api_v2; CREATE VIEW api_v2.active_customers AS ...</code>) is cleaner than suffixes for public-facing APIs. It gives you a natural namespace and lets you SET search_path</code> to switch versions. Query pg_depend yourself. PostgreSQL tracks all object dependencies in the pg_depend</code> system catalog. The catch: views do not depend on tables directly. The rewrite rule that implements the view does, so every traversal goes pg_depend -> pg_rewrite -> pg_class</code>. Direct view dependents of customers</code>: SELECT depns.nspname AS dependent_schema, depc.relname AS dependent_view, COALESCE( (SELECT a.attname FROM pg_attribute a WHERE a.attrelid = d.refobjid AND a.attnum = d.refobjsubid), '*' ) AS source_column FROM pg_depend d JOIN pg_rewrite r ON r.oid = d.objid JOIN pg_class depc ON depc.oid = r.ev_class JOIN pg_namespace depns ON depns.oid = depc.relnamespace WHERE d.refobjid = 'customers'::regclass AND d.classid = 'pg_rewrite'::regclass AND d.refclassid = 'pg_class'::regclass AND depc.relkind IN ('v', 'm') AND depc.oid <> 'customers'::regclass ORDER BY depns.nspname, depc.relname;</code></pre> Useful, but only one hop. customer_summary</code> depends on customers</code> transitively through two intermediate views, which this query misses. A recursive CTE walks the chain: WITH RECURSIVE view_deps AS ( -- direct dependents of the target table SELECT DISTINCT depc.oid AS view_oid, depns.nspname || '.' || depc.relname AS view_name, 1 AS depth FROM pg_depend d JOIN pg_rewrite r ON r.oid = d.objid JOIN pg_class depc ON depc.oid = r.ev_class JOIN pg_namespace depns ON depns.oid = depc.relnamespace WHERE d.refobjid = 'customers'::regclass AND d.classid = 'pg_rewrite'::regclass AND depc.relkind IN ('v', 'm') UNION -- views depending on views we already found SELECT depc.oid, depns.nspname || '.' || depc.relname, vd.depth + 1 FROM view_deps vd JOIN pg_depend d ON d.refobjid = vd.view_oid JOIN pg_rewrite r ON r.oid = d.objid JOIN pg_class depc ON depc.oid = r.ev_class JOIN pg_namespace depns ON depns.oid = depc.relnamespace WHERE d.classid = 'pg_rewrite'::regclass AND depc.relkind IN ('v', 'm') AND depc.oid <> vd.view_oid -- skip the view's own _RETURN rule ) SELECT view_name, MIN(depth) AS depth FROM view_deps GROUP BY view_name ORDER BY depth, view_name;</code></pre> The depc.oid <> vd.view_oid</code> filter matters: each view's _RETURN</code> rule has a self-dependency entry that without the filter sends the walk back to the view it just visited. This works, but at this point you are writing a dependency-analysis tool. Add topological sort, grant capture, matview handling, RLS-policy preservation, and you have small product. Materialized views have it worse. Everything above applies, plus you lose the cached result set on DROP</code>. After recreating, REFRESH</code> rebuilds from scratch: minutes to hours on large datasets, with no CONCURRENTLY</code> option until a unique index exists. For matviews that back dashboards, this means downtime. RLS is also off the table: REFRESH</code> runs as the matview owner, so the snapshot reflects the owner's access, not the querying user's. </div> Prior art hiding in pg_dump</a> </h2> The mechanism the manual workaround keeps reinventing, rewriting a view's body while preserving its identity, grants, and dependents, already exists inside PostgreSQL. It happens to live in pg_dump</code>, used only when needed. Most of the time pg_dump</code> does the obvious thing: builds a dependency graph, topologically sorts it, and emits each view in base-first order with a normal CREATE VIEW</code>. Dump our four-view chain and that is exactly what you get. The interesting case is when the sort fails. Cycles are uncommon but real: a view body that calls a function whose own body references the view back, a trigger on a base table that reads through a view reading the same table, cross-referencing matviews with RLS policies pointing back at views. When pg_dump</code> hits one, it falls back to the placeholder view trick: emit one of the views early as a stub with the right column list and types but a dummy body, then come back with CREATE OR REPLACE VIEW</code> to install the real definition once the rest of the cycle exists. -- emitted early, as a placeholder CREATE VIEW customer_summary AS SELECT NULL::integer AS id, NULL::text AS email, NULL::text AS name, NULL::bigint AS orders_12mo, NULL::bigint AS revenue_12mo_cents; -- emitted later, once dependencies exist CREATE OR REPLACE VIEW customer_summary AS SELECT co.id, co.email, co.name, COUNT(co.order_id) AS orders_12mo, COALESCE(SUM(co.total_cents), 0) AS revenue_12mo_cents FROM customer_orders co GROUP BY co.id, co.email, co.name;</code></pre> Grants, comments, and policies attach to the stub and survive the rewrite because the OID never changes. CREATE OR REPLACE VIEW</code> mutates the same pg_class</code> row in place. PostgreSQL has been doing this on every pg_dump --schema-only</code> for years. The mechanism is there; it is simply not exposed as user-facing DDL. Before you reach for a VIEW</a> </h2> Views are worth using. The abstraction is real: shared definitions, clean layering, column-level security, and no runtime cost when the planner can inline them. The trouble is that none of the rigidity is visible from the outside. A view looks like a table in the catalog, in \d</code>, in the ORM, and in every query that touches it. The teardown cost only appears the first time someone tries to drop a column, widen a type, or CASCADE</code> their way out of a migration on a Friday afternoon. So before you reach for one, hold the trade-off in mind: SELECT *</code> in a view body is a trap. It freezes the column list at creation time, hides the dependency in the catalog, and still blocks the same DDL an explicit list would. Always write the columns out.</li> Every layer multiplies the teardown cost. A three-deep view chain means three drops, three recreates, three sets of grants and policies to reapply, in the right order, for any structural change to a column at the bottom. Keep dependency trees shallow, and be honest about whether the abstraction is paying for itself.</li> CASCADE</code> is not a fix. It drops dependent views along with their grants, RLS policies, and downstream dependencies, with no undo. Never run it in production without a recreation script ready.</li> security_invoker</code> is not the default. A view runs as its owner, which silently bypasses RLS on base tables when the owner is privileged. On PostgreSQL 15+, WITH (security_invoker = true)</code> is what makes the security boundary real. For materialized views there is no equivalent. Enforce access at the layer that reads the matview, never below it.</li> </ul> When the migration finally has to happen, two things make the cycle survivable. Wrap the whole drop-alter-recreate-regrant sequence in a transaction so a typo or missed grant rolls back cleanly, and keep that transaction tight — AccessExclusiveLock</code> blocks everything else for its duration. Then map the dependency graph before you cut: pg_depend</code> joined through pg_rewrite</code>, walked with a recursive CTE, tells you what actually breaks. Run that query before the migration, not after the incident. The pain traces back to one missing primitive. ALTER VIEW</code> today handles renames, owner and schema changes, column defaults, and options like security_barrier</code>, but nothing structural. CREATE OR REPLACE VIEW</code> can append columns at the end and nothing else. A real ALTER VIEW DROP COLUMN</code>, ADD COLUMN</code>, ALTER COLUMN TYPE</code> is what would make views safe to evolve. The catalog and the in-place-rewrite mechanism are already there, as pg_dump</code>'s placeholder trick demonstrates; the user-facing DDL is what is missing. Even without it, views are still worth using. Just don't pretend they're tables. EDIT (2026-05-17): The original article understated how RLS interacts with views. Added a section on security_invoker</code> and why the ownership model matters, plus a note on why materialized views are structurally incompatible with RLS. Don't let AI touch your production database 2026-04-06T22:24:00+02:00 Not so long ago, the biggest threat to production databases was the developer who claimed it worked on their machine. If you've attended my sessions, you know this is a topic I'm particularly sensitive to. These days, AI agents are writing your SQL. The models are getting incredibly good at producing plausible code. It looks right, it feels right, and often it passes a cursory glance. But "plausible" isn't a performance metric, and it doesn't care about your execution plan or locking strategy. AI-generated SQL is syntactically correct, which is the easy part. The hard part is knowing what a statement does to a running system: which locks it takes, how long it holds them, whether it rewrites the table on disk. The spectacular failures get the headlines. In July 2025, an AI coding agent wiped a production database</a> during a code freeze -- ran destructive commands, panicked, then lied about what it had done. But the biggest damage is quieter. It's the migration that passes every test, ships through CI and then locks a production table during peak traffic. The query written with random assumptions. Indexes added based on copy/paste from psql</code>. The cumulative effect that builds over time, when nobody is looking. How to give AI agent "eyes"</a> </h2> When you're running Claude Code or any other agentic coding tool, the bottleneck isn't the model's intelligence. It's the fidelity of the environment. Standard AI coding involves the agent guessing column names based on your description, or in better cases parsing a schema.sql</code> file or using a local database with seed data. None of these give the agent the one thing it actually needs: awareness of your production schema. The table sizes, the indexes, the constraints, the statistics that determine whether a query index-scans in 2ms or sequential-scans for 40 seconds. The obvious fix is to give it a database connection. Let it query pg_catalog</code>, read table sizes, check existing indexes. This is how Anthropic's reference PostgreSQL MCP server worked, and that's the one Datadog Security Labs found a SQL injection in</a>. The vulnerable npm package still gets 21,000 downloads per week. The server wrapped queries in read-only transactions, but accepted semicolons. COMMIT; DROP SCHEMA public CASCADE;</code> works because the COMMIT</code> ends the read-only transaction and everything after runs with full privileges. Read-only transactions are not a security boundary when the protocol accepts semicolons. But here's the thing: the AI doesn't need a connection. It needs the information. Schema metadata is structural data. You can extract it once, save it as a JSON file, commit it to git, and hand that file to every tool in your stack. MCP</a> is an open standard for connecting AI assistants to external tools and data sources. Instead of relying on training data, the agent calls structured tools exposed by an MCP server and gets deterministic answers back. </div> What's been missing is a standard way to wire that information into the agent's workflow. Not a one-time dump that sits in a docs folder, but a protocol that lets the agent call tools and get structured answers back. That protocol is MCP</a> (Model Context Protocol). What are PostgreSQL MCP servers actually for?</a> </h2> This has been bugging me. If you search for "PostgreSQL MCP server" today, you'll find two kinds. The first is a connection wrapper. You give it a DATABASE_URL</code>, it exposes tools like query</code> and execute</code>. This is psql</code> with extra steps. The Anthropic reference server that got the SQL injection? Connection wrapper. The dozen community forks that replaced it? Connection wrappers with slightly better input validation. The second is a managed-provider gateway. Cloud database platforms expose their platform APIs through MCP. Create a branch, provision a database, manage users. Useful if you're on that platform, but not database intelligence. Control-plane operations wearing a protocol hat. Neither kind answers the question an AI agent actually needs answered: "is this SQL going to be a problem?" A connection wrapper can run EXPLAIN</code>, if you trust it with credentials. But who tells the agent that SET NOT NULL</code> on a 4-million-row table takes an ACCESS EXCLUSIVE</code> lock? Who tells it that its query defeats partition pruning? That's not a connection problem. It's a knowledge problem. dryrun</a> </h2> I didn't set out to build an AI tool. My primary project is RegreSQL</a>, a statistics-based SQL regression testing framework. RegreSQL needs production statistics to produce realistic query plans in test environments, and dryrun's schema capture was originally just the plumbing to get those statistics into the right place. But driving adoption of a testing tool is hard. The best place to plug in is where the coding actually happens. And these days, that's increasingly AI agents. Instead of fighting it, I turned the plumbing into a proper tool. dryrun</a> is a PostgreSQL schema intelligence tool. It ships as a CLI and as an MCP server. It doesn't connect to your database at runtime. Instead, it works from a JSON snapshot of your schema, captured once, committed to git, and available to everyone. Offline first</a> </h3> One person with production access captures the schema: dryrun dump-schema --source "$PROD_URL" --pretty --name "production" -o schema.json</code></pre> The JSON contains everything structural: tables, columns, constraints, indexes, and the rest of what you'd find in pg_catalog</code>. It also captures per-table statistics from pg_statistic</code> (histograms, most common values, correlation, n_distinct</code>) and runtime counters from pg_class</code> like reltuples</code>, seq_scan</code>, and idx_scan</code>. One file, typically 200-500KB. Structural metadata, not row data. If your schema uses COMMENT ON</code> (and it should), dryrun captures those too. A column named status</code> is ambiguous. COMMENT ON COLUMN orders.status IS 'pending | completed | refunded, set by payment processor webhook'</code> tells the agent what the values mean, where they come from, and that it shouldn't invent new ones. If you don't take my word for it, check what Bruce Momjian has to say</a> about it. MCP clients can only work with schemas, object names, and constraints. Comments are the simplest way to supply the semantic context that makes AI-generated queries meaningful. </div> Import it and every subsequent command works offline: dryrun import schema.json</code></pre> No connection needed, no credentials on developer machines or in CI. As an MCP server</a> </h3> claude mcp add dryrun -- dryrun mcp-serve</code></pre> Claude now has 16 tools for exploring your schema, validating queries, checking migrations, and suggesting improvements, all from the offline snapshot. What the agent gets</a> </h3> dryrun exposes 16 tools through MCP. The four that matter most for the argument in this article: check_migration is the one I find myself using most often, even outside of AI workflows. It analyzes DDL statements for lock implications, table rewrites, and version-specific behavior, and returns the safe multi-step alternative when one exists. Every migration example in this article was caught by this tool. suggest_index reads column statistics from the snapshot and recommends indexes informed by your actual data distribution. Not "add an index on this column" but "add a partial index on the 3% of rows that match your query pattern." It also uses correlation data to recommend BRIN indexes where appropriate: 3MB instead of 400MB for a B-tree on a column with near-perfect insertion order. validate_query parses SQL against the schema snapshot without executing it. It catches references to tables and columns that don't exist, detects anti-patterns, and flags issues before the query goes anywhere near the database. lint_schema runs 18 convention rules and 14 structural audit checks: naming, types, constraints, timestamps, partition health, duplicate indexes, foreign key columns missing indexes, tables without primary keys. All disableable in dryrun.toml</code>, because not every convention fits every project. The remaining tools cover schema exploration (list_tables, describe_table, search_schema, find_related), operations (compare_nodes, detect, vacuum_health), execution plans (explain_query), and schema maintenance (refresh_schema, check_drift). The full list is in the documentation</a>. Try it yourself</a> </h2> Here's a schema and seed data you can run in any local PostgreSQL: CREATE TABLE customers ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL, email text, created_at timestamptz NOT NULL DEFAULT now() ); CREATE TABLE orders ( id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id bigint NOT NULL, status text NOT NULL DEFAULT 'pending', total_amount numeric(10,2), created_at timestamptz NOT NULL DEFAULT now() );</code></pre> Seed it with data that has the distributions that matter, like skewed status values and nullable emails: -- 100K customers, ~5% with NULL email INSERT INTO customers (name, email, created_at) SELECT 'customer_' || i, CASE WHEN random() < 0.05 THEN NULL ELSE 'user' || i || '@example.com' END, '2023-01-01'::timestamptz + (i || ' seconds')::interval FROM generate_series(1, 100000) AS i; -- 1M orders, 97% completed / 3% pending INSERT INTO orders (customer_id, status, total_amount, created_at) SELECT 1 + (random() * 99999)::int, CASE WHEN random() < 0.03 THEN 'pending' ELSE 'completed' END, (random() * 500)::numeric(10,2), '2024-01-01'::timestamptz + (i || ' seconds')::interval FROM generate_series(1, 1000000) AS i; ANALYZE;</code></pre> Now capture the schema for dryrun and install it as an MCP server: dryrun dump-schema --source "$DATABASE_URL" -o schema.json dryrun init dryrun import schema.json claude mcp add dryrun -- dryrun mcp-serve</code></pre> Open Claude Code and try these prompts. First without the dryrun MCP server, then with it. "Add an index to speed up queries that filter orders by status"</a> </h3> Without dryrun, Claude sees a text</code> column and does the obvious thing: CREATE INDEX idx_orders_status ON orders (status);</code></pre> A full B-tree index on 1 million rows. It works, but 97% of those rows are 'completed'</code> and nobody queries for completed orders. With dryrun, Claude calls suggest_index, sees most_common_freqs</code> showing 'completed'</code> at 97% and 'pending'</code> at 3%, and suggests: CREATE INDEX idx_orders_status_pending ON orders (status) WHERE status = 'pending';</code></pre> The full index maintains 1 million entries. The partial index maintains 30,000. Every INSERT that creates a 'completed'</code> order no longer pays for index maintenance it'll never use. "Make the email column on customers required"</a> </h3> Without dryrun, Claude writes a single statement and moves on: ALTER TABLE customers ALTER COLUMN email SET NOT NULL;</code></pre> This fails immediately because 5% of rows have NULL emails. Claude didn't know the data existed. Even if the column had no NULLs, the statement would still take an ACCESS EXCLUSIVE</code> lock and perform a full table scan to verify. With dryrun, Claude calls check_migration and describe_table. It sees 100,000 rows, knows the column is nullable, and knows from the statistics that NULLs exist. It generates a three-step migration: -- Step 1: backfill NULLs UPDATE customers SET email = 'unknown-' || id || '@example.com' WHERE email IS NULL; -- Step 2: add constraint without full table scan ALTER TABLE customers ADD CONSTRAINT customers_email_not_null CHECK (email IS NOT NULL) NOT VALID; -- Step 3: validate in the background with a weaker lock ALTER TABLE customers VALIDATE CONSTRAINT customers_email_not_null;</code></pre> The first step handles the data. The second adds the constraint without scanning existing rows. The third validates with a weaker lock that allows concurrent reads and writes. Three statements instead of one. Keeping the snapshot fresh</a> </h2> dryrun is only as good as your last snapshot. If the schema changed yesterday and nobody re-dumped, the agent is working from stale data. The check_drift tool can detect this, but only if someone runs it. The best approach is to treat schema snapshots like you treat production backups: automate them. If you're already capturing production schema dumps on a schedule, add dryrun to the same pipeline. A nightly dump-schema</code> from the primary gives you current structure and statistics. If you run read replicas with different workloads, add --stats-only</code> dumps from each node for higher-fidelity statistics across the cluster. Commit the resulting schema.json</code> to your repository so every developer and every AI agent starts the day with a current view. The snapshot also serves as a lightweight audit trail. If someone adds a column through psql</code> without a migration, the next scheduled dump captures it, and the diff in git shows exactly what changed and when. What dryrun is not</a> </h2> dryrun does not execute arbitrary SQL against your database. It does not manage migrations. It does not provision infrastructure. It is not an ORM, a query builder, or a database admin panel. It's read-only and narrow by design, trustworthy enough that you'd let an AI agent use it unsupervised. What's next</a> </h2> dryrun ships today at v0.5.0. The next releases are focused on: Shadow EXPLAIN (v0.6.0): inject production statistics into a local PostgreSQL instance so EXPLAIN</code> produces realistic plans from your laptop, without any production data. Currently explain_query runs against whatever local database you point it at, but if that database has 10 rows in every table, the planner gives you toy plans that tell you nothing about production behavior.</li> CI/CD Trust Gate: a single command that takes a migration file and runs safety analysis, schema diff, lint, and index suggestions in one pass. Designed to block unsafe migrations before they merge.</li> Improved migration support: deeper analysis of multi-statement migrations, version-specific advice, and integration with common migration frameworks.</li> </ul> Getting started</a> </h2> dryrun is MIT licensed and available on GitHub</a>. The tutorial</a> covers installation, offline and online workflows, multi-node setups, and MCP integration. It's part of the boringSQL</a> suite. Good CTE, bad CTE 2026-03-29T14:47:00+02:00 The Common Table Expression</a>, or CTE, is often the first feature developers reach for beyond basic SQL, and often the only one. You write a subquery after WITH</code>, give it a name, and use it in the rest of your query. It only exists for the duration of that query. But the popularity of CTEs usually has less to do with modernizing code and more to do with the promise of imperative logic. For many, CTE acts as an easy to understand remedy for 'scary queries' and way how to force execution order on the database. The way how many write queries is as if they tell optimizer "first do this, then do that". This creates a problem. CTEs handle query decomposition, recursion and multi statement DDLs. Planner treats them differently depending how you write and use them though. For long time (prior PostgreSQL 12) CTEs acted as optimization fence. The planner couldn't push predicates into them, couldn't use indexes on the underlying tables. Couldn't do anything that materialize them and scan through the result. PostgreSQL 12 changed this. CTEs now get inlined, materialized, or something in between, depending on how you write them. Sample schema</a> </h2> We will use the same schema as in the article PostgreSQL Statistics: Why queries run slow</a>. CREATE TABLE customers ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL ); CREATE TABLE orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', note text, created_at date NOT NULL DEFAULT CURRENT_DATE ); CREATE TABLE orders_archive (LIKE orders INCLUDING ALL EXCLUDING IDENTITY); INSERT INTO customers (name) SELECT 'Customer ' || i FROM generate_series(1, 2000) AS i; INSERT INTO orders (customer_id, amount, status, note, created_at) SELECT (random() * 1999 + 1)::int, (random() * 500 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], CASE WHEN random() < 0.3 THEN 'Some note text here for padding' ELSE NULL END, '2022-01-01'::date + (random() * 1095)::int FROM generate_series(1, 100000); ANALYZE customers; ANALYZE orders;</code></pre> For recursive examples later on we'll also need an employees</code> table with a self-referencing hierarchy: CREATE TABLE employees ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL, manager_id integer REFERENCES employees(id), department text NOT NULL ); INSERT INTO employees (name, manager_id, department) VALUES ('Alice', NULL, 'Engineering'), ('Bob', 1, 'Engineering'), ('Charlie', 1, 'Engineering'), ('Diana', 2, 'Engineering'), ('Eve', 2, 'Engineering'), ('Frank', 3, 'Sales'), ('Grace', 3, 'Sales'), ('Hank', 6, 'Sales'), ('Ivy', 6, 'Sales'); ANALYZE employees;</code></pre>The optimization fence era (pre-PG 12)</a> </h2> As we already covered before PostgreSQL 12, every CTE was materialized. No exceptions. The planner would compute the CTE result set in full, store it in a temporary tuplestore, and then scan that tuplestore whenever the main query referenced the CTE. This made CTEs an optimization fence because the planner could not look through them. Consider this simple query: EXPLAIN WITH filtered AS ( SELECT * FROM orders WHERE created_at > '2025-01-01' ) SELECT * FROM filtered WHERE status = 'pending';</code></pre> On PostgreSQL 11 and earlier, the EXPLAIN</code> output would look something like this: QUERY PLAN ------------------------------------------------------------------- CTE Scan on filtered (cost=1840.00..2290.00 rows=2 width=58) Filter: (status = 'pending') CTE filtered -> Seq Scan on orders (cost=0.00..1840.00 rows=10000 width=58) Filter: (created_at > '2025-01-01'::date)</code></pre> Notice what happens here. The CTE runs a sequential scan on orders</code> with the date filter. It materializes all matching rows. Then the outer query applies the status = 'pending'</code> filter after materialization. Even if a composite index on (created_at, status)</code> existed, the planner couldn't use it as it can't see through the CTE boundary to combine the predicates. Why was it designed this way? Two reasons. First, reason was snapshot isolation. Materializing the CTE guaranteed that the result set was computed once, from a single snapshot, regardless of how many times it was referenced. Second, as protection for side-effect edge cases. If a CTE contained a data-modifying statement (INSERT</code>, UPDATE</code>, DELETE</code>), materialising ensured it executed exactly once. The workaround was well-known in the community: rewrite CTEs as subqueries. Subqueries had always been subject to the planner's normal optimisation rules, including predicate pushdown and inlining. The same query written as SELECT * FROM (SELECT * FROM orders WHERE created_at > '2025-01-01') sub WHERE status = 'pending'</code> would produce a much better plan.</div> This led to an entire workaround culture. Developers would write queries with CTEs for readability during development, then rewrite them as nested subqueries for production. The community had a saying: CTEs are optimization fences. It was repeated so often. Many developers still believe it today. But it hasn't been true since PostgreSQL 12. PostgreSQL 12: CTE inlining</a> </h2> PostgreSQL 12 introduced automatic CTE inlining. Non-recursive, side-effect-free, singly-referenced CTEs are now inlined by default. The planner started treating them as subqueries and applies all its normal optimisations. Predicate pushdown, index usage, join reordering apply exactly as if the CTE syntax never existed. The same query from the previous section now produces a completely different plan: EXPLAIN WITH filtered AS ( SELECT * FROM orders WHERE created_at > '2025-01-01' ) SELECT * FROM filtered WHERE status = 'pending';</code></pre> QUERY PLAN --------------------------------------------------------------------------- Seq Scan on orders (cost=0.00..2355.00 rows=2 width=58) Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</code></pre> The CTE is gone from the plan entirely. Both predicates are merged into a single scan on orders</code>. If there's a suitable index, the planner can use it. The CTE syntax doesn't change the execution plan. PostgreSQL 12 also introduced two new keywords that let you override the planner's decision: MATERIALIZED</code> - forces the CTE to materialize, even when the planner would inline it</li> NOT MATERIALIZED</code> - forces inlining, even when the planner would materialize it</li> </ul> -- force materialization EXPLAIN WITH filtered AS MATERIALIZED ( SELECT * FROM orders WHERE created_at > '2025-01-01' ) SELECT * FROM filtered WHERE status = 'pending'; -- Force inlining EXPLAIN WITH filtered AS NOT MATERIALIZED ( SELECT * FROM orders WHERE created_at > '2025-01-01' ) SELECT * FROM filtered WHERE status = 'pending';</code></pre> This follows the same principle as VIEW inlining</a>. When does a CTE get materialized?</a> </h2> Case 1: single reference, no side effects (INLINED)</a> </h3> The simplest and most common case. If you reference the CTE exactly once and it contains no side effects, the planner inlines it. EXPLAIN WITH recent AS ( SELECT * FROM orders WHERE created_at > '2025-01-01' ) SELECT * FROM recent WHERE status = 'pending';</code></pre> QUERY PLAN ------------------------------------------------------------------------------ Seq Scan on orders (cost=0.00..2355.00 rows=2 width=59) Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text)) (2 rows)</code></pre> Both predicates are merged. The planner considers all access paths on orders</code> directly. Case 2: multiple references (MATERIALIZED)</a> </h3> When a CTE is referenced more than once, the planner materializes it. This is actually a feature, CTE is computed once and reused. Therefore avoiding redundant work. EXPLAIN WITH summary AS ( SELECT status, count(*) AS cnt FROM orders GROUP BY status ) SELECT a.status, b.status FROM summary a, summary b WHERE a.cnt > b.cnt;</code></pre> QUERY PLAN ---------------------------------------------------------------------------- Nested Loop (cost=2355.04..2355.52 rows=5 width=64) Join Filter: (a.cnt > b.cnt) CTE summary -> HashAggregate (cost=2355.00..2355.04 rows=4 width=17) Group Key: orders.status -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=9) -> CTE Scan on summary a (cost=0.00..0.08 rows=4 width=40) -> CTE Scan on summary b (cost=0.00..0.08 rows=4 width=40) (8 rows)</code></pre> The CTE Scan</code> nodes appear twice, but the HashAggregate</code> runs only once. For expensive computations referenced multiple times, this is exactly what you want. Case 3: recursive CTE (ALWAYS MATERIALIZED)</a> </h3> Recursive CTEs must maintain a working table between iterations. There's no way to inline them. We'll cover recursion in detail later in the article. EXPLAIN WITH RECURSIVE subordinates AS ( SELECT id, name, manager_id FROM employees WHERE id = 1 UNION ALL SELECT e.id, e.name, e.manager_id FROM employees e JOIN subordinates s ON e.manager_id = s.id ) SELECT * FROM subordinates;</code></pre> QUERY PLAN ------------------------------------------------------------------------------- CTE Scan on subordinates (cost=17.21..18.83 rows=81 width=40) CTE subordinates -> Recursive Union (cost=0.00..17.21 rows=81 width=13) -> Seq Scan on employees (cost=0.00..1.11 rows=1 width=13) Filter: (id = 1) -> Hash Join (cost=0.33..1.53 rows=8 width=13) Hash Cond: (e.manager_id = s.id) -> Seq Scan on employees e (cost=0.00..1.09 rows=9 width=13) -> Hash (cost=0.20..0.20 rows=10 width=4) -> WorkTable Scan on subordinates s (cost=0.00..0.20 rows=10 width=4) (10 rows)</code></pre>Case 4: data-modifying CTE (ALWAYS MATERIALIZED)</a> </h3> CTEs that contain INSERT</code>, UPDATE</code>, or DELETE</code> are always materialized. The side effects must execute exactly once, in a predictable order. EXPLAIN WITH deleted AS ( DELETE FROM orders WHERE status = 'cancelled' RETURNING * ) SELECT count(*) FROM deleted;</code></pre> QUERY PLAN --------------------------------------------------------------------------- Aggregate (cost=2670.13..2670.14 rows=1 width=8) CTE deleted -> Delete on orders (cost=0.00..2105.00 rows=25117 width=6) -> Seq Scan on orders (cost=0.00..2105.00 rows=25117 width=6) Filter: (status = 'cancelled'::text) -> CTE Scan on deleted (cost=0.00..502.34 rows=25117 width=0) (6 rows)</code></pre> The CTE Scan</code> is present because the DELETE</code> must be fully executed before the count(*)</code> can run. Case 5: VOLATILE function (MATERIALIZED)</a> </h3> If a CTE contains a VOLATILE</code> function, the planner materializes it to prevent the function from being evaluated multiple times with potentially different results. EXPLAIN WITH rand AS ( SELECT id, random() AS r FROM orders ) SELECT * FROM rand WHERE r < 0.01;</code></pre> QUERY PLAN ----------------------------------------------------------------------- CTE Scan on rand (cost=2105.00..4355.00 rows=33333 width=12) Filter: (r < '0.01'::double precision) CTE rand -> Seq Scan on orders (cost=0.00..2105.00 rows=100000 width=12) (4 rows)</code></pre> Even though rand</code> is referenced only once, the CTE Scan</code> is there. random()</code> is VOLATILE</code>, which forces materialization. Case 6: STABLE functions (INLINED)</a> </h3> STABLE</code> functions like now()</code> do not prevent inlining. The reason is that the time is frozen at transaction start. EXPLAIN WITH recent AS ( SELECT * FROM orders WHERE created_at > now() - interval '7 days' ) SELECT * FROM recent WHERE status = 'pending';</code></pre> QUERY PLAN ------------------------------------------------------------------------------- Seq Scan on orders (cost=0.00..2855.00 rows=2 width=59) Filter: ((status = 'pending'::text) AND (created_at > (now() - '7 days'::interval))) (2 rows)</code></pre> No CTE Scan</code>. The planner inlines the CTE and merges both predicates, just like Case 1. now()</code> is STABLE</code> as it returns the same value within a transaction - and the planner's inlining check only looks for VOLATILE</code> functions (via contain_volatile_functions()</code>). STABLE</code> passes that check. Why do people think STABLE</code> blocks inlining? Because before PostgreSQL 12, all CTEs were materialized regardless of volatility. When PG 12 introduced inlining, the only function-level barrier was VOLATILE</code>. But the old "CTEs are optimization fences" mental model was so deeply ingrained that many developers assumed STABLE</code> was also a problem. It isn't. The function that actually blocks inlining: clock_timestamp()</code>. Unlike now()</code>, it's VOLATILE</code> and returns a different value on every call. A CTE with clock_timestamp()</code> will be materialized. Similarly, random()</code> and nextval()</code> are VOLATILE</code> and force materialization (as shown in Case 5). </div> If you see a CTE with now()</code> being materialized, the cause is something else. Either multiple references, a data-modifying statement, or an explicit MATERIALIZED</code> hint. Don't blame STABLE</code>. Case 7: forcing behaviour with hints</a> </h3> You can always override the planner's decision. -- force materialization on something that would normally be inlined EXPLAIN WITH filtered AS MATERIALIZED ( SELECT * FROM orders WHERE status = 'pending' ) SELECT * FROM filtered WHERE amount > 400;</code></pre> QUERY PLAN ---------------------------------------------------------------------- CTE Scan on filtered (cost=2105.00..2670.58 rows=5290 width=92) Filter: (amount > '400'::numeric) CTE filtered -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59) Filter: (status = 'pending'::text) (5 rows)</code></pre>-- force inlining on something that would normally be materialized EXPLAIN WITH filtered AS NOT MATERIALIZED ( SELECT * FROM orders WHERE status = 'pending' ) SELECT * FROM filtered a JOIN filtered b ON a.customer_id = b.customer_id;</code></pre> QUERY PLAN -------------------------------------------------------------------------------- Hash Join (cost=2419.21..10686.65 rows=317742 width=118) Hash Cond: (orders.customer_id = orders_1.customer_id) -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59) Filter: (status = 'pending'::text) -> Hash (cost=2105.00..2105.00 rows=25137 width=59) -> Seq Scan on orders orders_1 (cost=0.00..2105.00 rows=25137 width=59) Filter: (status = 'pending'::text) (7 rows) </code></pre> If the CTE were materialized, you would see a single scan of the orders</code> table, followed by two CTE Scans on the result. Instead, the planner has treated your query as if you had written a standard join between two subqueries. Be careful with NOT MATERIALIZED</code> on multiply-referenced CTEs. When you force inlining, the subquery runs once for each reference. In the example above, the orders</code> table is scanned twice. Once for a</code> and once for b</code>. For small result sets this might be fine. For large ones, you're doing double the work. Measure before using. </div> Case 8: FOR UPDATE / FOR SHARE (MATERIALIZED)</a> </h3> Row-locking clauses force materialization even on a singly-referenced, side-effect-free CTE. Internally, the planner's contain_dml()</code> check treats FOR UPDATE</code> and FOR SHARE</code> the same as data-modifying statements. EXPLAIN WITH locked AS ( SELECT * FROM orders WHERE status = 'pending' FOR UPDATE ) SELECT * FROM locked WHERE amount > 400;</code></pre> QUERY PLAN ---------------------------------------------------------------------------- CTE Scan on locked (cost=2356.37..2921.95 rows=5290 width=92) Filter: (amount > '400'::numeric) CTE locked -> LockRows (cost=0.00..2356.37 rows=25137 width=65) -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=65) Filter: (status = 'pending'::text) (6 rows)</code></pre> Without FOR UPDATE</code>, this CTE would be inlined. The LockRows</code> node and CTE Scan</code> confirm materialization. Decision matrix</a> </h3> Here's the full picture across PostgreSQL versions: Condition</th> PG ≤ 11</th> PG 12–16</th> PG 17–18</th></tr></thead> Single ref, pure SELECT</td> Materialized</td> Inlined</td> Inlined</td></tr> Multiple refs, pure SELECT</td> Materialized</td> Materialized</td> Materialized (better stats)</td></tr> VOLATILE function</td> Materialized</td> Materialized</td> Materialized</td></tr> STABLE function</td> Materialized</td> Inlined</td> Inlined</td></tr> Data-modifying (DML)</td> Materialized</td> Materialized</td> Materialized</td></tr> FOR UPDATE / FOR SHARE</td> Materialized</td> Materialized</td> Materialized</td></tr> Recursive</td> Materialized</td> Materialized</td> Materialized</td></tr> Explicit MATERIALIZED</code></td> -</td> Materialized</td> Materialized</td></tr> Explicit NOT MATERIALIZED</code></td> -</td> Inlined</td> Inlined</td></tr> </tbody></table> The statistics black hole</a> </h2> As mentioned in PostgreSQL Statistics: Why queries run slow</a>, materialized CTEs are one of the places "where no statistics go." This is arguably the biggest practical problem with CTE materialization. When the planner materializes a CTE, the result set is stored in a temporary tuplestore. This tuplestore has no pg_statistic</code> entries - no histograms, no MCVs, no correlation data. The planner has to estimate row counts and value distributions using hardcoded defaults. Let's see this in action. Here's a CTE over 10,000 rows: EXPLAIN WITH all_orders AS MATERIALIZED ( SELECT * FROM orders ) SELECT * FROM all_orders WHERE status = 'pending' AND amount > 400;</code></pre> QUERY PLAN ----------------------------------------------------------------------- CTE Scan on all_orders (cost=1855.00..4355.00 rows=5290 width=92) Filter: ((amount > '400'::numeric) AND (status = 'pending'::text)) CTE all_orders -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=59) (4 rows)</code></pre> The planner estimated 5,290 rows. Where does that number come from? The planner has no MCV list for status</code> inside the CTE, no histogram for amount</code>. It falls back to default selectivities of 0.3333</code> for the range comparison on amount</code> and a rough guess for the equality on status</code>, and multiplies them against the 100,000 input rows. If this CTE were inlined, the planner would read the actual statistics from pg_statistic</code> for the orders</code> table and produce estimates based on real data distribution, not defaults.</div> In a simple query this might not matter much. But when a materialized CTE feeds into a join, default-based estimates can cascade. The planner might choose a nested loop where a hash join would be better, or vice versa. It might underestimate memory needs and spill to disk unexpectedly. PG 17: Statistics propagation</a> </h3> PostgreSQL 17 brought two significant improvements to materialized CTEs: Column statistics propagation. When the planner creates a CTE Scan</code> node, it now propagates column statistics from the underlying query into the scan node. This means n_distinct</code>, MCV lists, and histograms from the source table can inform estimates on the CTE scan. Path key propagation. Materialized CTEs now preserve sort order information. If the CTE's subquery produces sorted output, the planner knows about it and can skip redundant sorts downstream. These improvements significantly reduce the estimation gap, but they don't eliminate it. Inlined CTEs are still strictly better for planning accuracy, because the planner works directly with the base table statistics rather than propagated copies. If your CTE doesn't need to be materialized, don't force it. When materialization helps</a> </h2> Materialization isn't always bad. Multiple references. If the CTE result is used in multiple places, materialization computes it once. Without it, the subquery runs once per reference. EXPLAIN WITH monthly_totals AS ( SELECT date_trunc('month', created_at) AS month, status, sum(amount) AS total FROM orders GROUP BY 1, 2 ) SELECT cur.month, cur.status, cur.total, prev.total AS prev_month_total, cur.total - prev.total AS delta FROM monthly_totals cur LEFT JOIN monthly_totals prev ON cur.month = prev.month + interval '1 month' AND cur.status = prev.status; </code></pre> QUERY PLAN ------------------------------------------------------------------------------- Merge Left Join (cost=3887.46..3990.90 rows=4384 width=136) Merge Cond: ((cur.month = ((prev.month + '1 mon'::interval))) AND (cur.status = prev.status)) CTE monthly_totals -> HashAggregate (cost=3105.00..3181.72 rows=4384 width=49) Group Key: date_trunc('month'::text, (orders.created_at)::timestamp with time zone), orders.status -> Seq Scan on orders (cost=0.00..2355.00 rows=100000 width=23) -> Sort (cost=352.87..363.83 rows=4384 width=72) Sort Key: cur.month, cur.status -> CTE Scan on monthly_totals cur (cost=0.00..87.68 rows=4384 width=72) -> Sort (cost=352.87..363.83 rows=4384 width=72) Sort Key: ((prev.month + '1 mon'::interval)), prev.status -> CTE Scan on monthly_totals prev (cost=0.00..87.68 rows=4384 width=72) (12 rows)</code></pre> The aggregation runs once. Both cur</code> and prev</code> read from the materialized result. Without materialization, the entire aggregation would run twice. Expensive VOLATILE expressions. If a CTE contains calls to volatile functions or expensive computations, materialization ensures they execute exactly once. Data-modifying operations. The whole point of writable CTEs is that the side effects happen once and the RETURNING</code> data is available downstream. Materialization is not optional here. When inlining isn't enough</a> </h2> The imperative mindset from the introduction, "first do this, then do that", doesn't go away just because the planner inlines your CTEs. And this one is my personal favourite, and source that never stop delivering query refactoring. Developers still structure queries as sequential pipelines, and that structure itself can create performance problems that have nothing to do with materialization. A common pattern is building queries as an assembly line: one CTE filters rows, the next LEFT JOINs related tables and aggregates metadata with GROUP BY</code>, the next filters on the aggregated results. It reads like a clean pipeline, but the GROUP BY</code> in the middle creates a wall the planner can't optimize past. WITH recent_orders AS ( SELECT * FROM orders WHERE created_at > '2024-01-01' ), order_metadata AS ( SELECT o.id, bool_or(oa.id IS NOT NULL) AS was_archived, count(o2.id) AS related_count FROM recent_orders o LEFT JOIN orders_archive oa ON o.id = oa.id LEFT JOIN orders o2 ON o.customer_id = o2.customer_id AND o2.id != o.id GROUP BY o.id ) SELECT o.*, m.was_archived, m.related_count FROM recent_orders o JOIN order_metadata m ON o.id = m.id WHERE m.was_archived = false AND m.related_count > 0;</code></pre> Each CTE here is referenced once, so they all get inlined. No materialization, no optimization fence. The planner sees the full query. So what's the problem? The GROUP BY</code> in order_metadata</code>. Even after inlining, the planner cannot push the was_archived = false</code> predicate past the aggregation. It must first LEFT JOIN every filtered order against orders_archive</code> and self-join orders</code>, compute the aggregates for all of them, and only then discard the rows that don't match. If recent_orders</code> returns 50,000 rows but only 200 were ever archived, you're joining and aggregating 49,800 rows for nothing. The fix is to replace the aggregation-then-filter pattern with correlated EXISTS</code> subqueries: SELECT o.* FROM orders o WHERE o.created_at > '2024-01-01' AND NOT EXISTS ( SELECT 1 FROM orders_archive oa WHERE oa.id = o.id ) AND EXISTS ( SELECT 1 FROM orders o2 WHERE o2.customer_id = o.customer_id AND o2.id != o.id );</code></pre> EXISTS</code> short-circuits after finding the first matching row. The planner can push created_at > '2024-01-01'</code> all the way down to an index scan on orders</code>, then probe each related table per result. No aggregation, no wasted work. The rule of thumb: if your CTE contains a GROUP BY</code> or a LEFT JOIN</code> just to compute a boolean ("does this row have related data?"), you've built a wall the planner can't see past. A correlated EXISTS</code> lets the planner push filters down and stop scanning early. This applies whether the CTE is materialized or inlined. </div> Writable CTEs (the power and the traps)</a> </h2> Data-modifying CTEs let you INSERT</code>, UPDATE</code>, or DELETE</code> inside a WITH</code> clause and use the RETURNING</code> data in subsequent CTEs or the main query. EXPLAIN WITH deleted AS ( DELETE FROM orders WHERE status = 'cancelled' AND created_at < '2023-01-01' RETURNING * ), archived AS ( INSERT INTO orders_archive SELECT * FROM deleted RETURNING id ) SELECT count(*) FROM archived;</code></pre> QUERY PLAN ---------------------------------------------------------------------------------------------- Aggregate (cost=2709.19..2709.20 rows=1 width=8) CTE deleted -> Delete on orders (cost=0.00..2355.00 rows=8334 width=6) -> Seq Scan on orders (cost=0.00..2355.00 rows=8334 width=6) Filter: ((created_at < '2023-01-01'::date) AND (status = 'cancelled'::text)) CTE archived -> Insert on orders_archive (cost=0.00..166.68 rows=8334 width=92) -> CTE Scan on deleted (cost=0.00..166.68 rows=8334 width=92) -> CTE Scan on archived (cost=0.00..166.68 rows=8334 width=0) (9 rows) </code></pre> This deletes old cancelled orders, moves them to an archive table, and counts how many were archived - all in a single atomic statement, no application-level coordination needed. But there are sharp edges. You cannot read what you just wrote</a> </h3> All sub-statements in a data-modifying CTE see the same snapshot. This means the effects of one CTE are not visible to other CTEs or the main query when they read the target table. Only the RETURNING</code> clause communicates data between CTE steps. SELECT count(1) FROM orders WHERE customer_id = 1;</code></pre> count ------- 31 (1 row)</code></pre>WITH ins AS ( INSERT INTO orders (customer_id, amount, status, created_at) VALUES (1, 100.00, 'pending', CURRENT_DATE) RETURNING id ) -- this does NOT see the row we just inserted SELECT count(1) FROM orders WHERE customer_id = 1;</code></pre> count ------- 31 (1 row)</code></pre> The count(1)</code> query sees the pre-insert snapshot. If you need the inserted data, you must use the RETURNING</code> clause from the ins</code> CTE, not re-read the table. Tuple shuffling</a> </h3> A common pattern is using writable CTEs to move rows between tables atomically: WITH moved AS ( DELETE FROM orders_staging RETURNING * ) INSERT INTO orders SELECT * FROM moved;</code></pre> This deletes all rows from the staging table and inserts them into the production table in a single atomic operation. No window where data exists in both or neither table. Data-modifying CTEs disable parallel query for the entire statement. If you have a complex query that mixes reads and writes, the write CTE prevents parallelism even for the read-only parts.</div> Recursive CTEs are always materialized</a> </h2> Recursive CTEs use an iterative working-table mechanism. Despite the name, they aren't truly recursive. PostgreSQL doesn't "call itself" by creating a nested stack of unfinished queries. Instead, it operates in loops. Execute the non-recursive term (the "seed"). Put results in the working table.</li> Execute the recursive term using the working table as input. New rows become the next working table.</li> Repeat until the recursive term returns no new rows.</li> Return the union of all iterations.</li> </ol> WITH RECURSIVE org_chart AS ( -- Seed: start from the CEO SELECT id, name, manager_id, 1 AS depth FROM employees WHERE manager_id IS NULL UNION ALL -- Recursive term: find direct reports SELECT e.id, e.name, e.manager_id, oc.depth + 1 FROM employees e JOIN org_chart oc ON e.manager_id = oc.id ) SELECT * FROM org_chart ORDER BY depth, name;</code></pre> id | name | manager_id | depth ----+---------+------------+------- 1 | Alice | | 1 2 | Bob | 1 | 2 3 | Charlie | 1 | 2 4 | Diana | 2 | 3 5 | Eve | 2 | 3 6 | Frank | 3 | 3 7 | Grace | 3 | 3 8 | Hank | 6 | 4 9 | Ivy | 6 | 4 (9 rows)</code></pre>UNION vs UNION ALL</a> </h3> The choice between UNION</code> and UNION ALL</code> in recursive CTEs matters more than in regular queries. UNION ALL</code> keeps all rows, including duplicates. This is faster but dangerous: if your graph has cycles, the recursion never terminates. PostgreSQL will run until you cancel the query or memory is exhausted. UNION</code> deduplicates at each iteration. This prevents infinite loops in cyclic graphs but adds the cost of hashing and comparing rows at every step. PostgreSQL 14 added SQL-standard SEARCH</code> and CYCLE</code> clauses that replace the manual patterns for controlling traversal order (breadth-first vs. depth-first) and detecting cycles. SEARCH BREADTH FIRST BY</code> / SEARCH DEPTH FIRST BY</code> control the order, while CYCLE</code> automatically detects and marks cycles, which is far cleaner than the old pattern of accumulating an array of visited IDs. For pure hierarchical data (trees without cycles), consider the ltree</code> extension as an alternative to recursive CTEs. It stores the full path as a label tree and supports efficient ancestor/descendant queries with GiST indexes. The trade-off is denormalized storage vs. on-the-fly recursion. </div> The exotic edge cases</a> </h2> Partition pruning lost</a> </h3> When you materialize a CTE over a partitioned table, partition pruning cannot happen on the CTE scan side. The materialized result is a flat tuplestore disconnected from the partition metadata. -- assume orders is range-partitioned by created_at WITH recent AS MATERIALIZED ( SELECT * FROM orders ) SELECT * FROM recent WHERE created_at > '2025-06-01';</code></pre> The created_at > '2025-06-01'</code> predicate is applied after materialization. All partitions are scanned to build the CTE, even though only one or two would be needed. Use NOT MATERIALIZED</code> (or simply let the planner inline it) to preserve partition pruning. Prepared statements and plan caching</a> </h3> PostgreSQL generates custom plans for the first 5 executions of a prepared statement. After that, it may switch to a generic plan. CTE inlining decisions can differ between custom and generic plans, because generic plans don't know the actual parameter values. This means a CTE that gets inlined during your first 5 calls might start materializing on the 6th or vice versa. If you see sudden plan changes with prepared statements, check whether the CTE inlining behaviour has shifted. work_mem spilling</a> </h3> Materialized CTEs store their results in memory, bounded by work_mem</code>. When the result set exceeds this limit, it silently spills to disk as a temporary file. This is not an error - it just gets slower. Monitor with log_temp_files = 0</code> (logs all temp files) or check EXPLAIN (ANALYZE, BUFFERS)</code> for temp read/write counts. PostgreSQL 18: EXPLAIN now shows memory/disk usage Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> reports memory and disk usage for Material nodes, including CTE materialization. You can see exactly how much memory a materialized CTE consumed and whether it spilled to disk. </div> CTE and security barrier views</a> </h3> A `security_barrier` view is a "black box" that forces the database to fully resolve the view's internal logic before any outer filters are applied.</div> Security-barrier views already prevent subquery flattening as a security measure (to stop user-defined functions from seeing rows they shouldn't). When you combine a security-barrier view with a CTE, you compound the optimisation barriers. The planner can neither inline the view nor the CTE. If performance matters in this scenario, consider materializing the security-sensitive filtering into a temporary table first. CTE vs. subquery vs. temporary table</a> </h2> CTE, referenced once the planner inlines it on PG 12+, so it doesn't affect the execution plan. This is the default choice for breaking up complex queries. CTE, referenced multiple times (small result) represents acceptable cost. Materialization means the subquery runs once. For aggregations or filtered subsets that produce a few hundred rows, the overhead is minimal. As Henrietta Dombrovskaya</a> emphasizes, "The best temporary table is the one you didn't create". Always exhaust your indexing and query-rewriting options before reaching for a temp table, as the DDL overhead often outweighs the execution gains.</div> **CTE, referenced multiple times (large result)** ss case when you might consider a temporary table instead. A materialized CTE has no indexes and no statistics. A temporary table can have both, and as covered in [Introduction to Buffers](/posts/introduction-to-buffers/), temporary tables use local buffers with simpler locking and no WAL overhead. If you're joining against 100k+ rows from a CTE, create a temp table, add an index, and `ANALYZE` it. Data-modifying operations with writable CTE. No alternative gives you the atomic, single-statement behavior. Recursive CTEs. There's no alternative in pure SQL. Large intermediate results needing indexes/statistics. If you really need them use temporary table. As discussed in Reading Buffer statistics</a>, temporary tables offer full planner support - indexes, statistics, and buffer management - that materialized CTEs simply don't have. Scenario</th> Recommendation</th></tr></thead> Readability, single reference</td> CTE (inlined, free)</td></tr> Compute once, use many times (small)</td> CTE (materialized)</td></tr> Compute once, use many times (large)</td> Temporary table</td></tr> Atomic data modification</td> Writable CTE</td></tr> Hierarchy / graph traversal</td> Recursive CTE</td></tr> Need indexes on intermediate data</td> Temporary table</td></tr> </tbody></table> The PG 18 state of affairs</a> </h2> PostgreSQL 18 continues to refine CTE handling without any revolutionary changes: EXPLAIN shows memory/disk usage for CTE materialization nodes. You can finally see whether your CTE fit in work_mem</code> or spilled to disk.</li> Better query plans for CTEs on the same table. The planner is smarter about eliminating redundant scans when multiple CTEs reference the same underlying table.</li> Data-modifying CTE fix for updatable views with rules. An edge case where writable CTEs interacted incorrectly with views defined using rules has been resolved.</li> CTE inlining is mature. The core inlining logic hasn't changed since PG 12. What has improved is everything around it - better statistics propagation (PG 17), better materialisation diagnostics (PG 18), better cost estimation.</li> </ul> CTEs are a good tool. Just know when you're holding the sharp end. pg_regresql: truly portable PostgreSQL statistics 2026-03-21T14:32:00+00:00 The previous article</a> showed that PostgreSQL 18 makes optimizer statistics portable, but left one gap open: It's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportionally. </blockquote> The planner doesn't trust pg_class.relpages</code>. It calls smgrnblocks()</code> to read the actual number of 8KB pages from disk. Your table is 74 pages on disk but pg_class.relpages</code> says 123,513? The planner uses the ratio to scale reltuples</code> down to match the actual file size. The selectivity ratios stay correct, plan shapes mostly survive, but the absolute cost estimates are off. For debugging a single query, that's usually fine. For automated regression testing where you compare EXPLAIN</code> costs across runs, it breaks things. A cost threshold of 2× means something different when the baseline was computed from fake-scaled numbers. As part of my work on RegreSQL</a> I'm happy to announce pg_regresql</a> extension which fixes this by hooking directly into the planner. Why the planner ignores relpages</a> </h2> When PostgreSQL's planner calls get_relation_info()</code> in plancat.c</code>, it delegates to estimate_rel_size()</code> which ends up in table_block_relation_estimate_size()</code> in tableam.c</code>. There, the actual page count comes from the storage manager: curpages = RelationGetNumberOfBlocks(rel);</code></pre> The function then computes a tuple density from pg_class</code> (reltuples / relpages</code>) and multiplies it by curpages</code> to estimate tuples. So pg_class.reltuples</code> isn't ignored, it's scaled to match the real file size. The reasoning is sound for normal operation: the catalog might be stale, but the file system is always current. The same applies to indexes. The planner reads their actual sizes from disk too. What pg_regresql does</a> </h2> The extension hooks into get_relation_info_hook</code>, a planner callback that runs after PostgreSQL reads the physical file stats. The hook replaces the file-based numbers with the values stored in pg_class</code>: rel->pages</code> ← pg_class.relpages</code></li> rel->tuples</code> ← pg_class.reltuples</code></li> rel->allvisfrac</code> ← pg_class.relallvisible / pg_class.relpages</code></li> </ul> It does the same for every index in rel->indexlist</code>. Pages and tuples for each index are overridden from the index's own pg_class</code> entry. The guard conditions are simple: skip the override if relpages == 0</code> (empty or never analyzed) or reltuples == -1</code> (never analyzed). The hook only activates for tables that have been ANALYZE</code>d or had statistics injected. Installation</a> </h2> Build from source using PGXS: cd pg_ext make make install</code></pre> It requires no GUCs, background workers, or shared memory. Usage</a> </h2> Load the extension in your session: LOAD 'pg_regresql';</code></pre> That's it. Every EXPLAIN</code> in this session will now use catalog statistics instead of file sizes. There are no functions to call, no tables to configure. You can also load it per-database by adding it to session_preload_libraries</code> in postgresql.conf</code> or via ALTER DATABASE</code>: ALTER DATABASE test_db SET session_preload_libraries = 'pg_regresql';</code></pre>The difference it makes</a> </h2> Using the same test_orders</code> example from the previous article</a>: 10,000 actual rows, injected with production statistics claiming 50 million rows across 123,513 pages. Without pg_regresql</code>: EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ---------------------------------------------------------------------------------------------------- Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26) Index Cond: (created_at > '2024-06-01'::date)</code></pre> The plan shape is correct (index scan thanks to the histogram), but the row estimate is 6,340. For a 50-million-row table where the filter covers roughly 10% of the histogram range, the expected estimate should be in the millions. The planner saw 74 real pages on disk, scaled reltuples</code> down to ~30,000, then applied selectivity. The ratio is preserved but the absolute number is wrong. With pg_regresql</code>: LOAD 'pg_regresql'; EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ----------------------------------------------------------------------------------------------------------- Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153212.27 rows=10791836 width=27) Index Cond: (created_at > '2024-06-01'::date) (2 rows)</code></pre> Cost numbers now reflect the full 50 million rows. Where this matters</a> </h2> Cost-based regression testing. If you're comparing EXPLAIN</code> costs between schema versions (which is what RegreSQL</a> does), you need the absolute numbers to be stable and realistic. With the scaling behavior, your baseline costs are proportional to your test database size, not production. A migration that doubles a cost in production might show a 1.3× increase in CI because the scaled-down numbers compress the range. Reproducing production plans on a laptop. Sometimes the plan shape itself changes depending on the absolute numbers. A query with multiple joins might get a different join order when the planner sees 50 million rows vs. 30,000 rows, because the cost crossover between hash join and nested loop depends on the absolute row count, not just the ratio. Index-only scans. The allvisfrac</code> (fraction of all-visible pages) matters for index-only scan costing. Without the hook, allvisfrac</code> is computed from the real relallvisible</code> catalog value divided by the real page count. With injected stats, relallvisible</code> might be 120,000 but the real page count is 74, so the fraction clamps to 1.0 and the planner overestimates how cheap index-only scans are. The hook fixes this by using the injected relpages</code> as the denominator. What it doesn't do</a> </h2> Column-level statistics and ANALYZE</code> behavior are unchanged. The extension only affects how the planner reads table and index sizes. One thing worth noting: EXPLAIN ANALYZE</code> will still show actual row counts from the real (small) data. The extension changes the planner's cost estimates, not query execution. The full workflow</a> </h2> Combining PostgreSQL 18's portable statistics with pg_regresql</code>, the full workflow looks like this: # 1. dump schema and statistics from production pg_dump --schema-only -d production_db > schema.sql pg_dump --statistics-only -d production_db > stats.sql # 2. create test database createdb test_db psql -d test_db -f schema.sql # 3. load minimal fixture data (optional) psql -d test_db -f fixtures.sql # 4. inject production statistics psql -d test_db -f stats.sql # 5. install pg_regresql and prevent stats from being overwritten psql -d test_db <<SQL ALTER DATABASE test_db SET session_preload_libraries = 'pg_regresql'; ALTER TABLE orders SET (autovacuum_enabled = false); -- repeat for other tables SQL # 6. reconnect and verify (plans now match production) psql -d test_db -c "EXPLAIN SELECT * FROM orders WHERE status = 'pending'"</code></pre>Compatibility</a> </h2> The extension works with PostgreSQL 13 through 18. The portable statistics functions (pg_restore_relation_stats</code>, pg_restore_attribute_stats</code>) require PostgreSQL 18, but pg_regresql</code> works with any method of writing to pg_class</code>, including direct catalog updates on older versions. PostgreSQL 19 will need a small update: the get_relation_info_hook</code> used by pg_regresql has been replaced</a> with build_simple_rel_hook</code>. The new hook runs slightly later with different arguments, but the override logic stays the same. Don't use this in production. The extension makes the planner ignore reality. That's exactly what you want for testing with injected statistics. In production, you want the planner to see the actual file sizes so it can adapt to data growth, bloat, and vacuum activity. Keep pg_regresql</code> in your dev/test and CI databases. </div> Production query plans without production data 2026-03-08T22:05:00+00:00 In the previous article</a> we covered how the PostgreSQL planner reads pg_class</code> and pg_statistic</code> to estimate row counts, choose join strategies, and decide whether an index scan is worth it. The message was clear: when statistics are wrong, everything else goes with it. Streaming replication provides bit-to-bit replication, so all replicas share the same statistics with primary server.</div> But there was one thing we didn't talk about. Statistics are specific to the database cluster that generated them. The primary way to populate them is `ANALYZE` which requires the actual data. PostgreSQL 18 changed that. Two new functions: pg_restore_relation_stats</code> and pg_restore_attribute_stats</code> write numbers directly into the catalog tables. Combined with pg_dump --statistics-only</code>, you can treat optimizer statistics as a deployable artifact. Compact, portable, plain SQL. The feature was driven by the upgrade use case</a>. In the past, major version upgrades used to leave pg_statistic</code> empty, forcing you to run ANALYZE</code>. Which might take hours on large clusters. With PostgreSQL 18 upgrades now transfer statistics automatically. But that's just the beginning. The same logic lets you export statistics from production and inject them anywhere - test database, local debugging, or as part of CI pipelines. The problem</a> </h2> Your CI database has 1,000 rows. Production has 50 million. The planner makes completely different decisions for each. Running EXPLAIN</code> in CI tells you nothing about the production plan. This is the core premise behind RegreSQL</a>. Catching query plan regressions in CI is far more reliable when the planner sees production-scale statistics. Same applies to debugging. A query is slow in production and you want to reproduce the plan locally, but your database has different statistics, and planner chooses the predictable path. Porting production stats can provide you that snapshot of thinking planner has to do in production, without actually going to production. pg_restore_relation_stats</a> </h2> The first of function behind portable PostgreSQL statistics is pg_restore_relation_stats</code>. It writes table-level data directly into pg_class</code> in form of variadic name/value pairs. SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer, 'relallfrozen', 120000::integer );</code></pre> But that's just an example. Let's modify some real statistics to see the full value. We will create a small table, inject fake production-like statistics and watch the planner to change its mind. CREATE TABLE test_orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL, amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO test_orders (customer_id, amount, status, created_at) SELECT (random() * 9999 + 1)::int, (random() * 5000 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], '2024-01-01'::date + (random() * 365)::int FROM generate_series(1, 10000); CREATE INDEX ON test_orders (created_at); CREATE INDEX ON test_orders (status); ANALYZE test_orders;</code></pre> When you check the current statistics, it has predictable data. SELECT relname, relpages, reltuples FROM pg_class WHERE relname = 'test_orders';</code></pre> relname | relpages | reltuples -------------+----------+----------- test_orders | 74 | 10000 (1 row)</code></pre> With 10,000 rows across 74 pages, the planner picks a sequential scan. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ----------------------------------------------------------------- Seq Scan on test_orders (cost=0.00..199.00 rows=5891 width=26) Filter: (created_at > '2024-06-01'::date) (2 rows)</code></pre> Now inject production-scale table stats: SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'test_orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer );</code></pre> And you might be surprised by the result. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=17649 width=26) Filter: (created_at > '2024-06-01'::date)</code></pre> The planner is still using the sequential plan. Only the estimated number of rows has changed. Why? If you remember from previous article, it's where column level statistics come into play. Histogram bounds still match the original 10,000 rows we inserted. pg_restore_attribute_stats</a> </h2> This function writes column-level statistics into pg_statistic</code> the same catalog that ANALYZE populates</a> with MCVs, histograms, and correlation</a>. In previous section, we left the planner stuck on a sequential scan despite believing the table has 50 million rows. The missing piece is column-level statistics. Let's pick up where we left off and inject histogram bounds for created_at</code>. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'created_at', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 4::integer, 'n_distinct', -0.05::real, 'histogram_bounds', '{2019-01-01,2019-07-01,2020-01-01,2020-07-01,2021-01-01,2021-07-01,2022-01-01,2022-07-01,2023-01-01,2023-07-01,2024-01-01}'::text, 'correlation', 0.98::real );</code></pre> Now the planner knows the data spans 5 years. A query filtering on the last 6 months of 2024 covers a narrow slice. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ---------------------------------------------------------------------------------------------------- Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26) Index Cond: (created_at > '2024-06-01'::date)</code></pre> Histogram bounds divide the non-MCV portion of the data into equal-population buckets. If most_common_vals</code> accounts for most of the data, the histogram covers only the remaining tail. The number of buckets is controlled by default_statistics_target</code> (default 100, meaning 101 bounds). </div> And that's a plan flip! The histogram tells the planner the data spans 2019–2024, so > '2024-06-01'</code> matches a narrow tail. A small fraction of 50 million rows. The index scan that was ignored before is now the obvious choice. Table-level stats set the scale, column-level stats shaped the selectivity, and together they changed the plan. The correlation</code> statistic tells the planner how closely the physical row order matches the column's sort order. A value near 1.0 means sequential access patterns - making index scan cheaper</a> because the next row is likely on the same or adjacent page. For time-series data like created_at</code> where rows are inserted chronologically, correlation is typically very high. </div> Injecting a skewed distribution</a> </h2> The same function handles MCV lists</a>. In production, your status</code> column isn't uniform, 95% of orders are delivered, 1.5% are pending. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'status', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 9::integer, 'n_distinct', 5::real, 'most_common_vals', '{delivered,shipped,cancelled,pending,returned}'::text, 'most_common_freqs', '{0.95,0.015,0.015,0.015,0.005}'::real[] );</code></pre> You can see EXPLAIN SELECT * FROM test_orders WHERE status = 'pending';</code></pre> QUERY PLAN --------------------------------------------------------------------------------------- Bitmap Heap Scan on test_orders (cost=8.93..90.42 rows=599 width=27) Recheck Cond: (status = 'pending'::text) -> Bitmap Index Scan on test_orders_status_idx (cost=0.00..8.78 rows=599 width=0) Index Cond: (status = 'pending'::text) (4 rows)</code></pre> and compare it with EXPLAIN SELECT * FROM test_orders WHERE status = 'delivered';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=28458 width=27) Filter: (status = 'delivered'::text) (2 rows)</code></pre> Same column, same operator, different plans. The planner uses a bitmap index scan for pending</code> (1.5% rare enough to justify the index) and a sequential scan for delivered</code> (95% being most of the table). The selectivity ratios from the MCV list drive the plan choice. You might have noticed the row estimates (599 and 28,458) are lower than you'd expect for a 50-million-row table. The planner checks the actual physical file size. Our table is only 74 pages on disk, not the 123,513 we injected. Hence the planner scales reltuples</code> and relpages</code> down proportionally. The absolute numbers shrink, but the ratios between them stay correct, and it's the ratios that determine plan shape. When you use pg_dump --statistics-only</code> in practice, you're typically restoring into a database with comparable data volume, so the estimates align naturally. </div> </div> </div> </div> </div> pg_regresql extension The pg_regresql</code> extension fixes this scaling problem. It hooks into the planner to trust the injected relpages</code> value instead of reading the physical file size, so cost estimates match production even when your test database is tiny. Read more</a> </div> </div> pg_dump</a> </h2> The functions we covered are the mechanics. For operational use pg_dump</code> provides everything you need. PostgreSQL 18 added three flags. Flag</th> Effect</th></tr></thead> --statistics</code></td> dump the statistics (you have to request it explicitely)</td></tr> --statistics-only</code></td> dump only the statistics, not schema or data</td></tr> --no-statistics</code></td> do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database pg_dump --statistics-only -d production_db > stats.sql </code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above. The full workflow to turn your production data into testable plans might look like this: # 1. dump schema from production pg_dump --schema-only -d production_db > schema.sql # 2. dump statistics from production pg_dump --statistics-only -d production_db > stats.sql # 3. create test database with schema createdb test_db psql -d test_db -f schema.sql # 4. load fixture data (optional; masked, minimal) psql -d test_db -f fixtures.sql # 5. inject production statistics psql -d test_db -f stats.sql # 6. query plans now match production psql -d test_db -c "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started. To prevent this, disable autovacuum analyze on the tables you've injected. -- disable autovacuum ALTER TABLE test_orders SET (autovacuum_enabled = false); -- or set analyze threshold so high it nevers kicks-in ALTER TABLE test_orders SET (autovacuum_analyze_threshold = 2147483647);</code></pre> Be careful here. If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run. And please, never ever do this in production! </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups). It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>. Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>. The easiest way to grant it for automation: GRANT pg_maintain TO ci_service_account;</code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser. PostgreSQL Statistics: Why queries run slow 2026-02-26T23:35:00+00:00 Every query starts with a plan. Every slow query probably starts with a bad one. And more often than not, the statistics are to blame. But how does it really work? PostgreSQL doesn't run the query to find out - it estimates the cost. It reads pre-computed data from pg_class</code> and pg_statistic</code> and does the maths to figure out the cheapest path to your data. In ideal scenario, the numbers read are accurate, and you get the plan you expect. But when they are stale, the situation gets out of control. Planner estimates 500 rows, plans a nested loop, and hits 25,000. What seemed as optimal plan turns into a cascading failure. How do statistics get stale? It can be either bulk load, a schema migration, faster-than-expected growth, or simply VACUUM</code> not keeping up. Whatever the cause, the result is the same. The planner is flying blind. Choosing paths based on reality that no longer exists. In this post we will go inside the two catalogs the planner depends on, understand what ANALYZE</code> actually gets for you from a 30,000-row table, and see how those numbers determine whether your query takes milliseconds or minutes. Sample schema</a> </h2> For demonstration purposes we will use the same schema as in the article Reading Buffer statistics in EXPLAIN output</a>. CREATE TABLE customers ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name text NOT NULL ); CREATE TABLE orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL REFERENCES customers(id), amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', note text, created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO customers (name) SELECT 'Customer ' || i FROM generate_series(1, 2000) AS i; INSERT INTO orders (customer_id, amount, status, note, created_at) SELECT (random() * 1999 + 1)::int, (random() * 500 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], CASE WHEN random() < 0.3 THEN 'Some note text here for padding' ELSE NULL END, '2022-01-01'::date + (random() * 1095)::int FROM generate_series(1, 100000); ANALYZE customers; ANALYZE orders;</code></pre>What the Planner Reads</a> </h2> As mentioned above, every decision the planner makes is based on two sources: table-level metadata from pg_class</code></li> column-level metadata from pg_statistic</code>.</li> </ul> pg_class - relational-level stats</a> </h3> It actually tracks all relations. Not just tables and indexes, but also partitions, TOAST tables, sequences, composite types, and views. </div>Every table, index and materialized view has a row in `pg_class`. Before it even looks at column-level statistics, the planner reads four values from it: Column</th> Meaning</th></tr></thead> relpages</code></td> Number of 8KB pages representing the table on the disk</td></tr> reltuples</code></td> Estimated number of live rows in the table</td></tr> relallvisible</code></td> Pages where all tuples are visible to all transactions</td></tr> </tbody></table> For our sample table it looks like this. SELECT relname, relpages, reltuples, relallvisible FROM pg_class WHERE relname = 'orders';</code></pre> relname | relpages | reltuples | relallvisible ---------+----------+-----------+--------------- orders | 856 | 100000 | 856</code></pre>Compared to reading page from disk, handling single tuple represents tiny overhead. Per-row overhead is configured using cpu_tuple_cost</code>, which defaults to 0.01</code></div> The planner sees 100,000 rows spread across 856 pages. Every cost estimate starts from those two numbers. relpages</code> drives sequential scan cost - each page is one unit of I/O work as configured via seq_page_cost</code>. reltuples</code> controls estimates for joins, aggregations, and pretty much everything else. The reltuples</code> value is only an estimate, not a live count. It's updated by ANALYZE</code> (and autovacuum), not by individual INSERTs or DELETEs. Between ANALYZE runs, PostgreSQL scales reltuples</code> proportionally when relpages</code> value changes - if the table grows by 10% in pages, the planner assumes 10% more rows too. This works well enough for normal growth, but breaks down with bloat. If dead tuples are inflating the number of pages used, without adding real rows, the planner overestimates the table size. </div> The other column mentioned above matters for specific operations. relallvisible</code> tells the planner how much of the table can be read with an index-only scan. Meaning an index-only scan can return results using the index without checking the heap for visibility. pg_statistic (via pg_stats) - column-level stats</a> </h3> Knowing the size of a table is only half the picture. To estimate how many rows might match, the planner needs to understand the data inside each column. PostgreSQL maintains statistics in pg_statistic</code> catalog. Which you're most likely never going to use directly. In practice you will use the view pg_stats</code> which presents the human-friendly data behind it. The most interesting values it exposes are: Statistic</th> What it tells the planner</th></tr></thead> null_frac</code></td> Fraction of entries that are NULL values</td></tr> avg_width</code></td> Average width in bytes</td></tr> n_distinct</code></td> Number of distinct values (negative means fraction of rows)</td></tr> most_common_vals</code></td> Most frequent values</td></tr> most_common_freqs</code></td> Frequencies of those values</td></tr> histogram_bounds</code></td> Values dividing the remaining data into equal-population buckets (most_common_vals are excluded)</td></tr> correlation</code></td> Statistical correlation between physical row ordering and logical ordering of the column values</td></tr> </tbody></table> SELECT attname, null_frac, avg_width, n_distinct, most_common_vals, most_common_freqs, histogram_bounds, correlation FROM pg_stats WHERE tablename = 'orders' AND attname = 'status';</code></pre>-[ RECORD 1 ]-----+-------------------------------------- attname | status null_frac | 0 avg_width | 8 n_distinct | 4 most_common_vals | {pending,shipped,delivered,cancelled} most_common_freqs | {0.25396666,0.25,0.24973333,0.2463} histogram_bounds | correlation | 0.2524199</code></pre> From this planner knows there are exactly 4 distinct values, more or less equally distributed, without any NULL values. When you write a predicate WHERE status = 'pending'</code> it will estimate ~25% of rows are going to match. All that by not running a query, but reading the catalog row. Different situation is when checking column note</code>. SELECT attname, null_frac, avg_width, n_distinct, array_length(most_common_vals, 1) AS mcv_count, array_length(histogram_bounds, 1) AS histogram_buckets FROM pg_stats WHERE tablename = 'orders' AND attname = 'note';</code></pre>-[ RECORD 1 ]-----+------- attname | note null_frac | 0.6982 avg_width | 32 n_distinct | 1 mcv_count | 1 histogram_buckets |</code></pre> For this column there's ~70% of NULLs, and only one distinct value. Therefore WHERE note IS NOT NULL</code> will estimate 30% of rows. Selectivity in Action</a> </h2> Now that we have covered what data the planner has available, let's have a look at how it's used to estimate how many rows a certain part of a query will read. This "guess" is called Selectivity. And it's defined as a floating-point number between 0 and 1. The formula is pretty simple: Estimated Rows = Total Rows * Selectivity</code></pre> But the way Selectivity is calculated depends entirely on the operator you use (=</code>, <</code>, ></code> or LIKE</code>). The equality (Most Common Values)</a> </h3> Equality is easiest to start with. When you use WHERE status = 'shipped'</code> the planner first checks most_common_vals</code> (MCV) list. If there's a match, the selectivity is same as the one provided by most_common_freqs</code>. If the value isn't in the list, the planner assumes the value is part of the "remaining" population. It subtracts all MCV frequencies from 1.0 and divides the remainder by the number of other distinct values. (1.0 - (SELECT sum(s) FROM unnest(most_common_freqs) s)) / (n_distinct - array_length(most_common_vals, 1))</code></pre>The Range lookup (The Histogram)</a> </h3> Life would be easy if we would be looking only for exact values. Most of the time we need to utilise range lookup. For example WHERE amount > 400</code>. MCVs are useless in this case as there might be thousands or millions of unique constants. This is where histogram_bounds</code> comes in. PostgreSQL divides the column values into a number of buckets, where each bucket contains equal number of rows (not values). The Selectivity in this case is determined by how many buckets your query covers. In our example, if we have bounds (100, 200, 300, 400, 500, 600)</code> the planner will establish it covers 2 full buckets. Since there are 5 buckets in total the Selectivity is going to be 0.4 (2/5). If you are now wondering where 2 and 5 come from? The histogram_bounds</code> array defines boundaries between buckets. With bounds (100, 200, 300, 400, 500, 600)</code> there are 5 buckets in total. (100-200), (200-300), (300-400), (400-500), (500-600)</code></pre> And the same logic applies to matching. The condition amount > 400</code> matches on (400-500)</code> and (500-600)</code> buckets (2 in total). The histogram's biggest weakness is linear interpolation. If your data has massive spikes in distribution the planner will always assume a perfect distribution.</div> Slightly more complex situation happens in the cases where your value falls inside a bucket. If your query has WHERE amount > 350</code> the planner locates the bucket containing value 350 (in our example (300-400)</code>), assumes the data is linearly distributed and calculates the ratio within that bucket. Search and pattern matching</a> </h3> This is probably the most treacherous territory for the planner. For substring matching patterns like WHERE note LIKE '%middle%'</code>, there's no histogram or list of values to rely on. The planner must fall back to "magic constants" hardcoded in the PostgreSQL source code. The default for generic patterns is 0.5% of the total rows, defined as #define DEFAULT_MATCH_SEL 0.005</code></pre> Slightly better situation comes for prefixed matches like WHERE note LIKE 'boringSQL%'</code> where PostgreSQL can fall back to range conditions and use histogram bounds. While this is a subtle difference, it makes a night and day difference. Correlation and index scan cost</a> </h3> Remember correlation</code> from the pg_stats</code>? It says how closely the physical order of rows on disk matches the logical order of column values. Values close to 1.0 mean high correlation, values near 0 mean data is laid out randomly across pages. If you recall how data is organized in 8KB page</a> the row locality matters. This matters because it determines whether an index scan is worth it. The planner assumes a random page read costs 4× more than a sequential one (random_page_cost = 4.0</code> vs seq_page_cost = 1.0</code>). When correlation is high, the rows an index points to are physically adjacent. The planner expects sequential I/O and costs the scan cheaply. When correlation is low, each lookup likely hits a different page, and the planner costs each of those reads at the higher random rate. That difference alone can be enough to make a sequential scan cheaper than an index scan. n_distinct and join estimation</a> </h3> Value n_distinct</code> plays an important role in joins. This further stresses the importance of running ANALYZE</code> after bulk data changes. Let's imagine this simplified logic for the equality join: In reality, Postgres also accounts for null_frac</code> (NULLs don't join) and MCVs on both sides. If both sides have MCVs, it calculates the "inner product" of the frequencies.</div> estimated_rows = (rows_left × rows_right) / max(n_distinct_left, n_distinct_right)</code></pre> Let's say you're trying to join two tables, both having roughly 2,000 distinct values for the join key. Outdated n_distinct</code> will cause significant estimation drifts and the planner may pick a wrong join strategy altogether. This is the heap we are talking about</a> </h3> Please, keep in mind this describes how the planner estimates rows for a basic heap scan. Indexes, join selectivity, and complex types like JSONB each come with their own estimation logic, operator handling and quirks. The fundamentals of estimation are the same nevertheless. What if there are no statistics?</a> </h2> So far we have touched on why up-to-date statistics are a must. But what if there are no statistics at all? For example for a new table or new column when ANALYZE</code> has not yet run. In those cases PostgreSQL falls back to hardcoded defaults. Condition type</th> Default selectivity</th> Constant</th></tr></thead> Equality (=</code>)</td> 0.5%</td> DEFAULT_EQ_SEL = 0.005</code></td></tr> Range (></code>, <</code>)</td> 33.3%</td> DEFAULT_INEQ_SEL = 0.3333</code></td></tr> Range (BETWEEN</code>)</td> 0.5%</td> DEFAULT_RANGE_INEQ_SEL = 0.005</code></td></tr> Pattern matching (LIKE</code>)</td> 0.5%</td> DEFAULT_MATCH_SEL = 0.005</code></td></tr> IS NULL</code></td> 0.5%</td> DEFAULT_UNK_SEL = 0.005</code></td></tr> IS NOT NULL</code></td> 99.5%</td> DEFAULT_NOT_UNK_SEL = 0.995</code></td></tr> </tbody></table> Nothing that a quick ANALYZE</code> can't fix, correct? Or maybe not. Where No Statistics Go While this article focuses on statistics and getting them right, there are situations where no statistics will be available (never or not predictably). If you believe it won't affect you, please, think twice. CTEs and subqueries when not inlined/materialized have no statistics. See Good CTE, Bad CTE</a> for when exactly this happens.</li> Temporary tables are not touched by autovacuum, so no automatic ANALYZE</code>.</li> Foreign tables do not guarantee stats are propagated.</li> And, to a big surprise, computed expressions in WHERE like WHERE amount * 1.1 > 500</code> or lower(email) = 'hello@example.com'</code>; unless you create an expression index or extended statistics.</li> </ul> And btw, do you know about TRUNCATE</code> - it's fast way to get rid of the data, but it leaves statistics behind... </div> How ANALYZE</code> Works</a> </h2> As we have already mentioned several times, ANALYZE</code> is the only mechanism that populates pg_class</code> and pg_statistic</code> with fresh data. Understanding what it samples, what it computes, and what it misses, is key to understanding why statistics are sometimes wrong. The process itself consists of 6 separate steps (as reported by pg_stat_progress_analyze</code>): initializing</li> acquiring sample rows</li> acquiring inherited sample rows (child/partitioned tables)</li> computing statistics (MCVs, histograms, correlation, etc.)</li> computing extended statistics (see below)</li> finalizing and writing to pg_statistic</code></li> </ol> For our purposes we will only cover sampling, computing statistics and writing to pg_statistic</code>. The sampling mechanism</a> </h3> ANALYZE actually doesn't read the entire table. It samples what is considered to be a statistically justified minimum sample size (defined as 300 for the reservoir sampling</a> algorithm). For PostgreSQL that means 300 × default_statistics_target</code> rows. SHOW default_statistics_target;</code></pre> default_statistics_target --------------------------- 100 (1 row)</code></pre> With the default target of 100, that's 30,000 rows. For our 100,000-row orders table, ANALYZE</code> reads about 30% of the data. For a 50-million-row table, it reads 0.06%. The same target also controls the size of the MCV list and histogram. Up to 100 entries each. With a target of 100, don't be surprised that histogram_bounds</code> contains 101 values. 100 buckets need 101 boundaries to close them.</div> The sampling is two-stage. First, ANALYZE</code> selects a random set of pages. Then it reads all live rows from those pages. This gives a representative cross-section without reading every page. Computing statistics</a> </h3> Once ANALYZE has its 30,000 sample rows (considering the default values), it processes each column independently. The pipeline for a single column looks roughly like this: First, it counts NULLs and calculates null_frac</code> and avg_width</code> - the cheapest statistics to compute. Then it sorts the non-null values and builds the MCV list by counting duplicates. Values that appear frequently enough make the cut; the rest are passed to the histogram builder, which divides them into equal-population buckets. Finally, because the values are already sorted, ANALYZE compares the logical sort order against the physical tuple positions (which page each row came from) to compute correlation</code>. The key detail here is that MCVs and histograms are not built from the same pool. Values that land in most_common_vals</code> are excluded from histogram_bounds</code>. This is why you'll sometimes see a column with MCVs but no histogram, or a histogram but no MCVs. They represent different slices of the same data. Writing to pg_statistic</a> </h3> Once all columns are processed, ANALYZE writes the results into pg_statistic</code> - one row per column. If a row for that column already exists, it's updated in place. This is a regular heap update, which means the old row becomes a dead tuple. On tables with many columns or frequent ANALYZE runs, this can cause pg_statistic</code> itself to bloat. After pg_statistic</code> is updated, ANALYZE refreshes relpages</code>, reltuples</code>, and relallvisible</code> in pg_class</code>. These values are recalculated from the sampling, not from a full table scan (they are estimates too). Controlling statistics quality</a> </h2> default_statistics_target</a> </h3> The default target of 100 works well for most columns. It means up to 100 MCVs, 101 histogram bounds, and a 30,000-row sample. Increasing it helps when: A column has many distinct values and the top 100 don't cover enough of the distribution</li> Range queries on skewed data produce bad estimates because histogram buckets are too coarse</li> Join estimates are off because n_distinct</code> is inaccurate</li> </ul> The cost scales linearly. Setting it to 1000 means 300,000 sampled rows, up to 1000 MCVs, more catalog storage, and slower planning from larger arrays to search. The maximum is 10,000. You don't have to raise it globally. For a single problematic column: ALTER TABLE orders ALTER COLUMN status SET STATISTICS 500; ANALYZE orders;</code></pre> Now status</code> gets up to 500 MCVs and 501 histogram bounds, while every other column stays at 100. Extended statistics</a> </h3> Standard statistics treat each column independently. This means the planner can't know that city = 'Edinburgh'</code> and country = 'UK'</code> are correlated. It multiplies their selectivities independently, potentially underestimating by orders of magnitude. Extended statistics solve this for specific column combinations: CREATE STATISTICS orders_status_date (dependencies, ndistinct, mcv) ON status, created_at FROM orders; ANALYZE orders;</code></pre> This tells ANALYZE to compute functional dependencies, combined distinct counts, and combined MCVs between the columns. The planner can then use these to avoid the independence assumption for queries filtering on both columns. The three types of extended statistics serve different purposes: dependencies capture functional dependencies between columns. Helps when knowing one column's value determines or narrows another's (e.g. zip_code</code> largely determines city</code>).</li> ndistinct tracks the number of distinct value combinations across columns. Helps with GROUP BY</code> on multiple columns where the planner would otherwise multiply distinct counts independently.</li> mcv builds a combined most-common-values list for column tuples. The most powerful but most expensive option. Helps with multi-column WHERE conditions on correlated values.</li> </ul> You can create extended statistics with any combination of these types. Start with dependencies</code> as it's cheapest, add mcv</code> when multi-column filter estimates are consistently wrong. Extended statistics are worth creating when you see EXPLAIN</code> estimates that are consistently wrong on multi-column filters, and the columns are logically correlated. They are computed during step 5 of the ANALYZE process and stored in pg_statistic_ext_data</code>. Diagnosing bad estimates</a> </h2> When a query is slow, the first question should always be: did the planner estimate correctly? Compare the estimate to reality with EXPLAIN ANALYZE</code>. Estimate off by handful of rows means statistics are fine. But when you see estimates off by 10x or more, that's where planning goes wrong. A nested loop that looks cheap for 100 rows becomes a disaster at 10,000. The statistics tell you what the planner believed, and comparing that with reality tells you what to do next. Either run ANALYZE</code>, consider tuning the statistics target for a specific column, or creating extended statistics if multiple columns are involved. The planner is only as good as what it reads from the catalog. When estimates go wrong, don't blame the planner. Check the data it's working with. Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment? When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden. You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run CREATE DATABASE dbname;</code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use CREATE DATABASE dbname TEMPLATE template1;</code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>. In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine. CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic. Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time. The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options. FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files. When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>. With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space. Quick setup checklist: Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database. CREATE DATABASE source_db; \c source_db CREATE TABLE boring_data ( id serial PRIMARY KEY, payload text ); -- generate 50m rows INSERT INTO boring_data (payload) SELECT md5(random()::text) || md5(random()::text) FROM generate_series(1, 50000000); -- force a checkpoint CHECKPOINT;</code></pre> You can verify the database now has roughly 6GB of data. Name | source_db Owner | postgres Encoding | UTF8 Locale Provider | libc Collate | en_US.UTF-8 Ctype | en_US.UTF-8 Locale | ICU Rules | Access privileges | Size | 6289 MB Tablespace | pg_default Description |</code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get CREATE DATABASE slow_copy TEMPLATE source_db; CREATE DATABASE Time: 67000.615 ms (01:07.001)</code></pre> Now, let's verify our configuration is set for speed: show file_copy_method; file_copy_method ------------------ clone (1 row)</code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time. CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY; CREATE DATABASE Time: 212.053 ms</code></pre> That's a quite an improvement, isn't it? Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes? When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage. This can create some initial confusion. If you ask PostgreSQL for the size: SELECT pg_database_size('source_db') as source, pg_database_size('fast_clone') as clone;</code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size. -[ RECORD 1 ]------ source | 6594041535 clone | 6594041535</code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page. A single UPDATE will therefore trigger copy-on-write on multiple pages: the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage. XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</code></pre> All it takes is to update some rows using update boring_data set payload = 'new value' || id where id IN (select id from boring_data limit 20);</code></pre> and the situation will start to change. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10471550.. 10471589: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10297326.. 10297365: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 8 extents found root@clone-demo:/var/lib/postgresql#</code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared. Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections. Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy. Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it. VACUUM Is a Lie (About Your Indexes) 2025-12-11T23:34:00+00:00 There is common misconception that troubles most developers using PostgreSQL: tune VACUUM or run VACUUM, and your database will stay healthy. Dead tuples will get cleaned up. Transaction IDs recycled. Space reclaimed. Your database will live happily ever after. But there are couple of dirty "secrets" people are not aware of. First of them being VACUUM is lying to you about your indexes. The anatomy of storage</a> </h2> When you delete a row in PostgreSQL, it is just marked as a 'dead tuple'. Invisible for new transactions but still physically present. Only when all transactions referencing the row are finished, VACUUM can come along and actually remove them - reclamining the space in the heap (table) space. To understand why this matters differently for tables versus indexes, you need to picture how PostgreSQL actually stores your data. Your table data lives in the heap - a collection of 8 KB pages where rows are stored wherever they fit. There's no inherent order. When you INSERT a row, PostgreSQL finds a page with enough free space and slots the row in. Delete a row, and there's a gap. Insert another, and it might fill that gap - or not - they might fit somewhere else entirely. This is why SELECT * FROM users</code> without an ORDER BY can return rows in order initially, and after some updates in seemingly random order, and that order can change over time. The heap is like Tetris. Rows drop into whatever space is available, leaving gaps when deleted. When VACUUM runs, it removes those dead tuples and compacts the remaining rows within each page. If an entire page becomes empty, PostgreSQL can reclaim it entirely. And while indexes are on surface the same collection of 8KB pages, they are different. A B-tree index must maintain sorted order - that's the whole point of their existence and the reason why WHERE id = 12345</code> is so fast — though even indexes can be ignored</a> under certain conditions. PostgreSQL can binary-search down the tree instead of scanning every possible row. You can learn more about the fundamentals of B-Tree Indexes and what makes them fast</a>. But if the design of the indexes is what makes them fast, it's also their biggest responsibility. While PostgreSQL can fit rows into whatever space is available, it can't move the entries in index pages to fit as much as possible. VACUUM can remove dead index entries. But it doesn't restructure the B-tree. When VACUUM processes the heap, it can compact rows within a page and reclaim empty pages. The heap has no ordering constraint - rows can be anywhere. But B-tree pages? They're locked into a structure. VACUUM can remove dead index entries, yes. Many developers assume VACUUM treats all pages same. No matter whether they are heap or index pages. VACUUM is supposed to remove the dead entries, right? Yes. But here's what it doesn't do - it doesn't restructure the B-tree. What VACUUM actually does Removes dead tuple pointers from index pages</li> Marks completely empty pages as reusable</li> Updates the free space map</li> </ul> What VACUUM cannot do: Merge sparse pages together (can do it for empty pages)</li> Reduce tree depth</li> Deallocate empty-but-still-linked pages</li> Change the physical structure of the B-tree</li> </ul> Your heap is Tetris, gaps can get filled. Your B-tree is a sorted bookshelf. VACUUM can pull books out, but can't slide the remaining ones together. You're left walking past empty slots every time you scan. The experiment</a> </h2> Let's get hands-on and create a table, fill it, delete most of it and watch what happens. CREATE EXTENSION IF NOT EXISTS pgstattuple; CREATE TABLE demo (id integer PRIMARY KEY, data text); -- insert 100,000 rows INSERT INTO demo (id, data) SELECT g, 'Row number ' || g || ' with some extra data' FROM generate_series(1, 100000) g; ANALYZE demo;</code></pre> At this point, our index is healthy. Let's capture the baseline: SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 6434 kB demo_pkey | 2208 kB | 1563 kB</code></pre> Now remove some data, 80% to be precise - somewhere in the middle: DELETE FROM demo WHERE id BETWEEN 10001 AND 90000;</code></pre> The goal is to simulate a common real-world pattern: data retention policies, bulk cleanup operations, or the aftermath of a data migration gone wrong</a>. VACUUM demo; SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 1278 kB demo_pkey | 2208 kB | 313 kB</code></pre> The table shrunk significantly, while index remained unchanged You now have 20,000 rows indexed by a structure built to handle 100,000. But notice something important: even though actual_data</code> dropped to ~1.3 MB (reflecting our 20,000 remaining rows), the file_size</code> for the index is still 2208 kB. VACUUM cleaned out the dead tuple data, but it doesn't return space to the OS or compact index structures - it only marks pages as reusable within PostgreSQL. EDIT: the original table had a size without running VACCUM, which I must left out during what's sometime cumbersome process of authoring blog posts. Big thanks come to Creston Jamison of Scaling PostgreSQL</a> to notice it, and actually take what must have been a lot of time to analyze my posts. </blockquote> This experiment is really an extreme case, but demonstrates the problem. Understanding page states</a> </h2> Leaf pages have several states: Full page (>80% density), when the page contains many index entries, efficiently utilizing space. Each 8KB page</a> read returns substantial useful data. This is optimal state. Partial page (40-80% density) with some wasted space, but still reasonably efficient. Common at tree edges or after light churn. Nothing to be worried about. Sparse page (<40% density) is mostly empty. You're reading an 8KB page to find a handful of entries. The I/O cost is the same as a full page, but you get far less value. Empty page (0% density) with zero live entries, but the page still exists in the tree structure. Pure overhead. You might read this page during a range scan and find absolutely nothing useful. A note on fillfactor</a> </h3> You might be wondering how can fillfactor help with this? It's the setting you can apply both for heap and leaf pages, and controls how full PostgreSQL packs the pages during the data storage. The default value for B-tree indexes is 90%. This leaves 10% of free space on each leaf page for future insertions. CREATE INDEX demo_index ON demo(id) WITH (fillfactor = 70);</code></pre> A lower fillfactor (like 70%) leaves more room, which can reduce page splits when you're inserting into the middle of an index - useful for tables random index column inserts or those with heavily updated index columns. But if you followed carefully the anatomy of storage section, it doesn't help with the bloat problem. Quite the opposite. If you set lower fillfactor and then delete majority of your rows, you actually start with more pages, and bigger chance to end up with more sparse pages than partial pages. Leaf page fillfactor is about optimizing for updates and inserts. It's not a solution for deletion or index-column update bloat. Why the planner gets fooled</a> </h2> PostgreSQL's query planner estimates costs based on physical statistics</a>, including the number of pages in an index. EXPLAIN ANALYZE SELECT * FROM demo WHERE id BETWEEN 10001 AND 90000;</code></pre>QUERY PLAN -------------------------------------------------------------------------------------------------------------------- Index Scan using demo_pkey on demo (cost=0.29..29.29 rows=200 width=41) (actual time=0.111..0.112 rows=0 loops=1) Index Cond: ((id >= 10001) AND (id <= 90000)) Planning Time: 1.701 ms Execution Time: 0.240 ms (4 rows)</code></pre> While the execution is almost instant, you need to look behind the scenes. The planner estimated 200 rows and got zero. It traversed the B-tree structure expecting data that doesn't exist. On a single query with warm cache, this is trivial. Under production load with thousands of queries and cold pages, you're paying I/O cost for nothing. Again and again. If you dig further you discover much bigger problem. SELECT relname, reltuples::bigint as row_estimate, relpages as page_estimate FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | row_estimate | page_estimate -----------+--------------+--------------- demo | 20000 | 934 demo_pkey | 20000 | 276</code></pre> The relpages</code> value comes from the physical file size divided by the 8 KB page size</a>. PostgreSQL updates it during VACUUM and ANALYZE, but it reflects the actual file on disk - not how much useful data is inside. Our index file is still 2.2 MB (276 pages × 8 KB), even though most pages are empty. The planner sees 276 pages for 20,000 rows and calculates a very low rows-per-page ratio. This is when planner can come to conclusion - this index is very sparse - let's do a sequential scan instead. Oops. "But wait," you say, "doesn't ANALYZE fix statistics?" Yes and no. ANALYZE</code> updates the row count estimate. It will no longer think you have 100,000 rows but 20,000. But it does not shrink relpages, because that reflects the physical file size on disk. ANALYZE</code> can't change that. The planner now has accurate row estimates but wildly inaccurate page estimates. The useful data is packed into just ~57 pages worth of entries, but the planner doesn't know that. cost = random_page_cost × pages + cpu_index_tuple_cost × tuples ([visible in EXPLAIN output](/posts/explain-buffers/))</code></pre> With a bloated index: pages are oversized (276 instead of ~57)</li> The per-page cost gets multiplied by empty pages</li> Total estimated cost is artificially high</li> </ul> The hollow index</a> </h2> We can dig even more into the index problem when we look at internal stats: SELECT * FROM pgstatindex('demo_pkey');</code></pre>-[ RECORD 1 ]------+-------- version | 4 tree_level | 1 index_size | 2260992 root_block_no | 3 internal_pages | 1 leaf_pages | 57 empty_pages | 0 deleted_pages | 217 avg_leaf_density | 86.37 leaf_fragmentation | 0</code></pre> Wait, what? The avg_leaf_density is 86% and it looks perfectly healthy. That's a trap. Due to the hollow index (we removed 80% right in the middle) we have 57 well-packed leaf pages, but the index still contains 217 deleted pages. This is why avg_leaf_density</code> alone is misleading. The density of used pages looks great, but 79% of your index file is dead weight. The simplest way to spot index bloat is comparing actual size to expected size. SELECT c.relname as index_name, pg_size_pretty(pg_relation_size(c.oid)) as actual_size, pg_size_pretty((c.reltuples * 40)::bigint) as expected_size, round((pg_relation_size(c.oid) / nullif(c.reltuples * 40, 0))::numeric, 1) as bloat_ratio FROM pg_class c JOIN pg_index i ON c.oid = i.indexrelid WHERE c.relkind = 'i' AND c.reltuples > 0 AND c.relname NOT LIKE 'pg_%' AND pg_relation_size(c.oid) > 1024 * 1024 -- only indexes > 1 MB ORDER BY bloat_ratio DESC NULLS LAST;</code></pre>index_name | actual_size | expected_size | bloat_ratio ------------+-------------+---------------+------------- demo_pkey | 2208 kB | 781 kB | 2.8</code></pre> A bloat_ratio</code> of 2.8 means the index is nearly 3x larger than expected. Anything above 1.8 - 2.0 deserves investigation. We filter to indexes over 1 MB - bloat on tiny indexes doesn't matter that much. Please, adjust the threshold based on your environment; for large databases, you might only care about indexes over 100 MB. But here comes BIG WARNING: pgstatindex() we used earlier physically reads the entire index. On a 10 GB index, that's 10 GB of I/O. Don't run it against all indexes on a production server - unless you know what you are doing! REINDEX</a> </h2> How to actually fix index bloat problem? REINDEX</code> is s straightforward solution as it rebuilds the index from scratch. REINDEX INDEX CONCURRENTLY demo_pkey ;</code></pre> After which we can check the index health: SELECT * FROM pgstatindex('demo_pkey');</code></pre>-[ RECORD 1 ]------+------- version | 4 tree_level | 1 index_size | 466944 root_block_no | 3 internal_pages | 1 leaf_pages | 55 empty_pages | 0 deleted_pages | 0 avg_leaf_density | 89.5 leaf_fragmentation | 0</code></pre> And SELECT relname, pg_size_pretty(pg_relation_size(oid)) as file_size, pg_size_pretty((pgstattuple(oid)).tuple_len) as actual_data FROM pg_class WHERE relname IN ('demo', 'demo_pkey');</code></pre>relname | file_size | actual_data -----------+-----------+------------- demo | 7472 kB | 1278 kB demo_pkey | 456 kB | 313 kB</code></pre> Our index shrunk from 2.2 MB to 456 KB - 79% reduction (not a big surprise though). As you might have noticed we have used CONCURRENTLY</code> to avoid using ACCESS EXCLUSIVE lock. This is available since PostgreSQL 12+, and while there's an option to omit it - the pretty much only reason to do so is during planned maintenance to speed up the index rebuild time. pg_squeeze</a> </h2> If you look above at the file_size of our relations, we have managed to reclaim the disk space for the affected index (it was REINDEX</code> after all), but the table space was not returned back to the operating system. That's where pg_squeeze</a> shines. Unlike trigger-based alternatives, pg_squeeze uses logical decoding, resulting in lower impact on your running system. It rebuilds both the table and all its indexes online, with minimal locking: CREATE EXTENSION pg_squeeze; SELECT squeeze.squeeze_table('public', 'demo');</code></pre> The exclusive lock is only needed during the final swap phase, and its duration can be configured. Even better, pg_squeeze is designed for regular automated processing - you can register tables and let it handle maintenance whenever bloat thresholds are met. pg_squeeze makes sense when both table and indexes are bloated, or when you want automated management. REINDEX CONCURRENTLY is simpler when only indexes need work. There's also older tool pg_repack - for a deeper comparison of bloat-busting tools, see article The Bloat Busters: pg_repack vs pg_squeeze</a>. VACUUM FULL (The nuclear option)</a> </h2> VACUUM FULL</code> rewrites the entire table and all indexes. While it fixes everything it comes with a big but - it requires an ACCESS EXCLUSIVE lock - completely blocking all reads and writes for the entire duration. For a large table, this could mean hours of downtime. Generally avoid this in production. Use pg_squeeze instead for the same result without the downtime. When to act, and when to chill</a> </h2> Before you now go and REINDEX</code> everything in sight, let's talk about when index bloat actually matters. B-trees expand and contract with your data. With random insertions affecting index columns - UUIDs, hash keys, etc. the page splits happen constantly. Index efficiency might get hit at occassion and also settle around 70 - 80% over different natural cycles of your system usage. That's not bloat. That's the tree finding its natural shape for your data. The bloat we demonstrated - 57 useful pages drowning in 217 deleted ones - is extreme. It came from deleting 80% of contiguous data. You won't see this from normal day to day operations. When do you need to act immediately: after a massive DELETE (retention policy, GDPR purge, failed migration cleanup)</li> bloat_ratio</code> exceeds 2.0 and keeps climbing</li> query plans suddenly prefer sequential scans on indexed columns</li> index size is wildly disproportionate to row count</li> </ul> But in most cases you don't have to panic. Monitor weekly and when indexes bloat ratio continously grow above warning levels, schedule a REINDEX CONCURRENTLY</code> during low traffic period. Index bloat isn't an emergency until it is. Know the signs, have the tools ready, and don't let VACUUM's silence fool you into thinking everything's fine. Conclusion</a> </h2> VACUUM is essential for PostgreSQL. Run it. Let autovacuum do its job. But understand its limitations: it cleans up dead tuples, not index structure. The truth about PostgreSQL maintenance is that VACUUM handles heap bloat reasonably well, but index bloat requires explicit intervention. Know when your indexes are actually sick versus just breathing normally - and when to reach for REINDEX. VACUUM handles heap bloat. Index bloat is your problem. Know the difference. RegreSQL: Regression Testing for PostgreSQL Queries 2025-11-13T22:47:00+00:00 TL;DR - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production. As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification. For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries. I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level. Introduction</a> </h2> The RegreSQL</a> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later). But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere? This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations. The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management. Basic regression testing</a> </h2> Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like $ regresql text Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓ Running regression tests... ✓ album-by-artist_list-albums-by-artist.1.json (0.00s) ✓ album-by-artist_list-albums-by-artist.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s) ✓ artist_top-artists-by-album.1.json (0.00s) ✓ genre-topn_genre-top-n.top-1.json (0.00s) ✓ genre-topn_genre-top-n.top-3.json (0.00s) ✓ genre-tracks_tracks-by-genre.json (0.00s) Results: 8 passed, 0 failed, 8 skipped (0.00s)</code></pre> In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for *.sql</code> files and attempts to run all queries against the configured PostgreSQL connection. The individual files can contain either single or multiple sql queries. Like following example -- name: top-artists-by-album -- Get the list of the N artists with the most albums SELECT artist.name, count(*) AS albums FROM artist LEFT JOIN album USING (artist_id) GROUP BY artist.name ORDER BY albums DESC LIMIT :n;</code></pre> The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now). This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines. Performance regression testing</a> </h2> Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts. RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated $ regresql baseline Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓ Creating baselines directory: regresql/baselines Creating directory 'regresql/baselines' Creating baselines for queries: ./ Created baseline: album-by-artist_list-albums-by-artist.1.json Created baseline: album-by-artist_list-albums-by-artist.2.json Created baseline: album-tracks_list-tracks-by-albumid.1.json Created baseline: album-tracks_list-tracks-by-albumid.2.json Created baseline: artist_top-artists-by-album.1.json Created baseline: genre-topn_genre-top-n.top-1.json Created baseline: genre-topn_genre-top-n.top-3.json Created baseline: genre-tracks_tracks-by-genre.json Baselines have been created successfully! Baseline files are stored in: regresql/baselines</code></pre> the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place. To demonstrate this, let's review the test output with the baselines. Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓ Running regression tests... ✓ album-by-artist_list-albums-by-artist.1.json (0.00s) ✓ album-by-artist_list-albums-by-artist.2.json (0.00s) ✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s) ⚠️ Sequential scan detected on table 'artist' Suggestion: Consider adding an index if this table is large or this query is frequently executed ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s) ⚠️ Sequential scan detected on table 'artist' Suggestion: Consider adding an index if this table is large or this query is frequently executed ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s) ✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s) ✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s) ✓ artist_top-artists-by-album.1.json (0.00s) ✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: album, artist Suggestion: Review query and consider adding indexes on filtered/joined columns ✓ genre-topn_genre-top-n.top-1.json (0.00s) ✓ genre-topn_genre-top-n.top-3.json (0.00s) ✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: genre, artist Suggestion: Review query and consider adding indexes on filtered/joined columns ⚠️ Multiple sort operations detected (2 sorts) Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: artist, genre Suggestion: Review query and consider adding indexes on filtered/joined columns ⚠️ Multiple sort operations detected (2 sorts) Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting ⚠️ Nested loop join with sequential scan detected Suggestion: Add index on join column to avoid repeated sequential scans ✓ genre-tracks_tracks-by-genre.json (0.00s) ✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s) ⚠️ Multiple sequential scans detected on tables: genre, track Suggestion: Review query and consider adding indexes on filtered/joined columns Results: 16 passed (0.00s)</code></pre> As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready". In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below. -- name: query_name -- metadata: key1=value1, key2=value2 SELECT ...;</code></pre> At this point RegreSQL recognizes notest</code> to skip the query testing altogether (not just cost tracking)</li> nobaseline</code> to skip cost tracking</li> noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li> and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li> </ul> -- name: query_name -- regresql: notest, nobaseline -- regresql: noseqscanwarn -- regresql: difffloattolerance:0.25 -- query that can vary in cost by 20% without being considered a failure SELECT ...;</code></pre>ORM enters the room</a> </h2> ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data: orders = ( session.query(Order) .filter(Order.user_id == user_id) .options( joinedload(Order.user), joinedload(Order.shipping_address), selectinload(Order.items) # NEW: Load order items ) .all() )</code></pre> That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan. RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system: @event.listens_for(engine, "before_cursor_execute") def capture_sql(conn, cursor, statement, *args): captured_queries.append(statement)</code></pre> Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development. Test Data Management</a> </h2> Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky. Traditional approach to create test data might involve Database dumps become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li> SQL scripts might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li> Factories in application code might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li> Shared test database is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li> </ul> What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000). This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale. RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example fixture: basic_users description: a handful of test users cleanup: rollback data: - table: users rows: - id: 1 email: alice@example.com name: Alice Anderson created_at: 2024-01-15 - id: 2 email: bob@example.com name: Bob Builder created_at: 2024-02-20</code></pre> To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture fixtures: - basic_users "1": email: alice@example.com "2": email: bob@example.com</code></pre> And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators including sequences, random integer, decimal, string, uuid, email and name generators</li> date_between for generating random timestamps within a range</li> foreign key references to be able to reuse data from other table's fixtures</li> range to select value from predefined sources</li> Go template support</li> </ul> fixture: realistic_orders generate: - table: customers count: 1000 columns: id: generator: sequence start: 1 email: generator: email domain: shop.example.com name: generator: name type: full created_at: generator: date_between start: "2023-01-01" end: "2024-12-31" - table: orders count: 5000 columns: id: generator: sequence start: 1 customer_id: generator: int min: 1 max: 1000 amount: generator: decimal min: 10.00 max: 999.99 precision: 2 order_date: generator: date_between start: "2023-01-01" end: "2024-12-31"</code></pre> This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like. The fixtures are also stackable and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse. fixture: orders_with_shipping depends_on: - basic_users data: - table: orders rows: - id: 101 user_id: 1 # References Alice from basic_users total: 99.99 status: shipped</code></pre> Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation. fixture: mixed_setup description: Combine SQL with YAML and generated data cleanup: rollback # SQL executes first (either as file or inline) sql: - file: sql/setup_schema.sql - inline: "INSERT INTO config (key, value) VALUES ('version', '1.0');" # followed YAML data data: - table: users rows: - id: 1 email: admin@example.com # and finally generated data generate: - table: orders count: 100 columns: id: generator: sequence start: 1 user_id: generator: int min: 1 max: 1</code></pre> RegreSQL provides commands to inspect and validate your fixtures # List all available fixtures regresql fixtures list # Show fixture details and dependencies regresql fixtures show realistic_orders # Validate fixture definitions regresql fixtures validate # Show dependency graph regresql fixtures deps # Apply fixture manually (for debugging) regresql fixtures apply basic_users</code></pre> The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves. (EDIT 2025-11-20) The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use Fixed generated fixtures re-used between tests</li> Use RegreSQL to test fixtures before/after migration</li> </ul> I'm working on next release of RegreSQL to address DX of fixtures usage. RegreSQL future</a> </h2> Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development. At the same time RegreSQL is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable. RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>. Beyond Start and End: PostgreSQL Range Types 2025-11-02T21:40:00+00:00 One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them. Bug Not Invented Here</a> </h2> But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets. CREATE TABLE seat_holds ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL REFERENCES seats(id), user_session_id UUID NOT NULL, -- define the hold period explicitly hold_started_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, hold_expires_at TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP ); CREATE INDEX seat_holds_on_seat_id ON seat_holds(seat_id); CREATE INDEX seat_holds_on_expiration ON seat_holds(hold_expires_at);</code></pre> While the table design looks perfectly reasonable, it has one serious flaw - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee. If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain. Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.). And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level. -- needed to add GiST support to equality of integers (for seat_id) CREATE EXTENSION IF NOT EXISTS btree_gist; ALTER TABLE seat_holds ADD CONSTRAINT seat_holds_no_overlap EXCLUDE USING gist ( seat_id WITH =, tsrange(hold_started_at, hold_expires_at) WITH && );</code></pre> Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - you're making it impossible to have invalid state in your database, not just unlikely. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load. But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box. Introducing the Data Range Types</a> </h2> You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in? This brings us to the core of the matter: PostgreSQL's native Range Types. PostgreSQL provides a set of built-in range types, all following the pattern of type and range: int4range</code> for integer</li> int8range</code> for bigint</li> numrange</code> for numeric</li> tsrange</code> for timestamp without time zone</li> tstzrange</code> for timestamp with time zone (which we briefly saw above)</li> daterange</code> for date</li> </ul> And it does not stop there. You can easily define your own custom range types over any basic data type. First win: cleaner schema</a> </h2> When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code. When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct. The application code no longer needs to manage two separate boundaries. CREATE TABLE seat_holds_native ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL REFERENCES seats(id), user_session_id UUID NOT NULL, hold_period TSTZRANGE NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP );</code></pre> This is the power of a first-class data type. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly. Second win: Atomicity guaranteed</a> </h2> While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint as demonstrated previously. ALTER TABLE seat_holds_native ADD CONSTRAINT seat_holds_no_overlap EXCLUDE USING gist ( seat_id WITH =, hold_period WITH && );</code></pre> This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once: seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li> hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li> </ol> Finally EXCLUDE USING gist</code> is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint. This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.). Range boundaries: A quick math refresher</a> </h2> Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics. PostgreSQL ranges can have four different boundary types: Inclusive boundaries - [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to". -- [10, 20] includes both 10 and 20 -- represents 10 ≤ x ≤ 20 SELECT int4range(10, 20, '[]');</code></pre> Exclusive boundaries - (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality. -- (10, 20) excludes both 10 and 20 -- represents 10 < x < 20 SELECT int4range(10, 20, '()'); </code></pre> Mixed boundaries - [)</code> or (]</code> You can mix and match. The most common and default pattern in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next". As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods. -- these ranges are adjacent, not overlapping -- Week 1 SELECT '[2025-11-01, 2025-11-08)'::tstzrange; -- Week 2 SELECT '[2025-11-08, 2025-11-15)'::tstzrange;</code></pre> With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments. Canonicalization and Range Set Operations</a> </h2> Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges. For discrete range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper). PostgreSQL uses a canonicalization function to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing. -- [10, 20] includes integers 10, 11, ..., 20. SELECT int4range(10, 20, '[]') AS original_range; -- Result: [10,21) -- (9, 21) includes integers 10, 11, ..., 20. SELECT int4range(9, 21, '()') AS original_range; -- Result: [10,21)</code></pre> Exception are continuous ranges (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning. The operator toolkit</a> </h2> The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier. Overlap operator - &&</code> As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points. -- find holds active at any point between 10:00 and 11:00 SELECT * FROM seat_holds_native WHERE hold_period && '[2025-12-25 10:00, 2025-12-25 11:00)'::tstzrange;</code></pre> Contains operator - @></code> For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type). -- find holds that are active at the specific momement SELECT * FROM seat_holds_native WHERE hold_period @> '2025-11-05 15:00'::timestamptz; -- find holds that are active at the specific time range SELECT * FROM seat_holds_native WHERE hold_period @> '[2025-12-25 10:00, 2025-12-25 10:15)'::tstzrange;</code></pre> Contained By operator - <@</code> In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right. -- find holds that are within November '25 SELECT * FROM seat_holds_native WHERE hold_period <@ '[2025-11-01, 2025-12-01)'::tstzrange;</code></pre> Strictly Before/After operators - <<</code> and >></code> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries). -- find holds that finished strictly before 10 November SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</code></pre> Boundary Extension operators - &<</code> and &></code> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range). -- find holds that end before or at the same time as reference range ends SELECT * FROM seat_holds_native WHERE hold_period &< '[2025-11-08 17:00, 2025-11-08 18:00)'::tstzrange; -- find holds that start at or after reference range starts SELECT * FROM seat_holds_native WHERE hold_period &>'[2025-11-08 09:00, 2025-11-08 18:00)'::tstzrange;</code></pre> Finally the Adjecent operator - -|-</code> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict. -- find holds that are immediately adjacent (touching) to given range SELECT * FROM seat_holds_native WHERE hold_period -|- '[2025-11-08 17:00, 2025-11-08 18:00)'::tstzrange;</code></pre>To Infinity and Beyond</a> </h2> Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>. Let's start with infinite bounds, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time). -- range that never expires (upper bound is infinite) SELECT '[2025-11-01 10:00, infinity)'::tstzrange; -- range that has always been valid (lower bound is infinite) SELECT '[-infinity, 2025-11-01 10:00)'::tstzrange; -- range covering all time SELECT '[-infinity, infinity)'::tstzrange;</code></pre> This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription. -- lifetime subscription that never expires INSERT INTO subscriptions (user_id, plan, active_period) VALUES (42, 'lifetime', '[2025-11-01, infinity)'); -- all active subscriptions right now SELECT * FROM subscriptions WHERE active_period @> NOW();</code></pre> Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent. At any point you can validate whatever range has infinite bounds: SELECT lower_inf('[2025-11-01, infinity)'::tstzrange) AS lower_is_infinite, upper_inf('[2025-11-01, infinity)'::tstzrange) AS upper_is_infinite;</code></pre>Understanding NULL vs empty: Schrödinger's Range</a> </h2> Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty. Let's see this in practice: -- NULL range: we don't know what the period is INSERT INTO seat_holds_native (seat_id, user_session_id, hold_period) VALUES (42, 'abc-123', NULL); -- empty range: we know the period is explicitly "nothing" INSERT INTO seat_holds_native (seat_id, user_session_id, hold_period) VALUES (43, 'def-456', 'empty');</code></pre> The main difference between them is when it comes to handling. SELECT NULL::tstzrange && '[2025-11-01, 2025-11-08)'::tstzrange; -- Result: NULL (not true, not false—we don't know) SELECT 'empty'::tstzrange && '[2025-11-01, 2025-11-08)'::tstzrange; -- Result: false (we know it doesn't overlap)</code></pre> And you can check for them in your queries using built-in function isempty</code>. -- check for NULL (like any column) SELECT * FROM seat_holds_native WHERE hold_period IS NULL; -- check for empty (special function) SELECT * FROM seat_holds_native WHERE isempty(hold_period);</code></pre> In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail. Practical Integer ranges for Tiered pricing</a> </h2> While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing. CREATE TABLE quantity_discounts ( discount_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, product_id INTEGER NOT NULL, quantity_range INT4RANGE NOT NULL, discount_percentage NUMERIC(5,2) NOT NULL, -- no overlapping tiers EXCLUDE USING GIST ( product_id WITH =, quantity_range WITH && ) ); INSERT INTO quantity_discounts (product_id, quantity_range, discount_percentage) VALUES -- 1-9 units: no discount (1, '[1,10)', 0.00), -- 10-49 units: 5% off (1, '[10,50)', 5.00), -- 50-99 units: 10% off (1, '[50,100)', 10.00), -- 100+ units: 15% off (1, '[100,1000)', 15.00); -- verify what discount we offer for ordering 75 units? SELECT discount_percentage FROM quantity_discounts WHERE product_id = 1 AND quantity_range @> 75; -- Result: 10.00</code></pre>Making Bad Data Impossible</a> </h2> If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database. CREATE TABLE promotional_campaigns ( campaign_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, active_period TSTZRANGE NOT NULL, budget_range NUMRANGE NOT NULL, discount_percentage NUMERIC(5,2) NOT NULL, -- campaigns must be at least 1 days long CONSTRAINT campaigns_minimum_duration CHECK (upper(active_period) - lower(active_period) >= INTERVAL '1 days'), -- budget must be between $1000 and $100000 CONSTRAINT campaigns_valid_budget CHECK (budget_range <@ numrange(1000, 100000, '[]')), -- active period must not be empty CONSTRAINT campaigns_valid_period CHECK (NOT isempty(active_period)) );</code></pre> While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions). Multiranges: When one range is not enough</a> </h2> Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column. PostgreSQL 14 introduced multirange types for all the built-in range types: int4multirange</code>, int8multirange</code>, nummultirange</code></li> tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li> </ul> The real power of multiranges lies in schema density and query efficiency. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas. Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data. CREATE TABLE user_periods_single_range ( period_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, user_id INTEGER NOT NULL, active_period TSTZRANGE NOT NULL ); CREATE INDEX user_periods_single_range_gist_idx ON user_periods_single_range USING gist (active_period); -- 20 fragmented periods for user_id 42 (20 rows total) INSERT INTO user_periods_single_range (user_id, active_period) SELECT 42, tstzrange( '2025-01-01'::timestamptz + (i * 2 || ' days')::interval, '2025-01-01'::timestamptz + (i * 2 + 1 || ' days')::interval ) FROM generate_series(1, 20) AS s(i);</code></pre> compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data. CREATE TABLE user_periods_multirange ( user_id INTEGER PRIMARY KEY, active_periods TSTZMULTIRANGE NOT NULL ); CREATE INDEX user_periods_multirange_gist_idx ON user_periods_multirange USING gist (active_periods); -- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row INSERT INTO user_periods_multirange (user_id, active_periods) SELECT user_id, -- range_agg function automatically handles merging adjacent ranges range_agg(active_period)::TSTZMULTIRANGE FROM user_periods_single_range WHERE user_id = 42 GROUP BY user_id;</code></pre> consider now the difference between EXPLAIN SELECT period_id, user_id FROM user_periods_single_range WHERE user_id = 42 AND active_period @> '2025-01-20 12:00:00+00'::timestamptz;</code></pre> QUERY PLAN ------------------------------------------------------------------------------------------------- Bitmap Heap Scan on user_periods_single_range (cost=4.19..13.67 rows=1 width=12) Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone) Filter: (user_id = 42) -> Bitmap Index Scan on user_periods_single_range_gist_idx (cost=0.00..4.19 rows=6 width=0) Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</code></pre> and same version of the consolidated data EXPLAIN ANALYZE SELECT user_id, active_periods FROM user_periods_multirange WHERE user_id = 42 AND active_periods @> '2025-01-20 12:00:00+00'::timestamptz;</code></pre> QUERY PLAN ------------------------------------------------------------------------------------------------------------- Index Scan using user_periods_multirange_pkey on user_periods_multirange (cost=0.15..8.17 rows=1 width=36) Index Cond: (user_id = 42) Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</code></pre> Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements. One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps. Creating custom range types</a> </h2> While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges. To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values: -- function to calculate difference between two IP addresses using bigint CREATE OR REPLACE FUNCTION inet_diff(x INET, y INET) RETURNS FLOAT8 AS $$ SELECT ( (host(x)::inet - '0.0.0.0'::inet) - (host(y)::inet - '0.0.0.0'::inet) )::float8; $$ LANGUAGE SQL IMMUTABLE STRICT; -- custom inetrange type CREATE TYPE inetrange AS RANGE ( subtype = inet, subtype_diff = inet_diff ); -- convert CIDR ranges to inetranges CREATE OR REPLACE FUNCTION cidr_to_inetrange(cidr CIDR) RETURNS inetrange AS $$ SELECT inetrange( host(network($1))::inet, host(broadcast($1))::inet, '[]' ); $$ LANGUAGE SQL IMMUTABLE STRICT;</code></pre> Now you can use these custom types just like the built-in ones: CREATE TABLE ip_blocklists ( blocklist_name TEXT PRIMARY KEY, blocked_range inetrange NOT NULL ); INSERT INTO ip_blocklists (blocklist_name, blocked_range) VALUES ('attack #434401', cidr_to_inetrange('192.168.1.0/24')), ('attack #434401 (1)', '[203.0.113.50,203.0.113.99]'), ('attack #434401 (2)', '[203.0.113.143,203.0.113.159]');</code></pre> and locate which attack has been assigned to particular malicious IP address. SELECT blocklist_name, blocked_range FROM ip_blocklists WHERE blocked_range @> '192.168.1.25'::INET;</code></pre> blocklist_name | blocked_range ----------------+----------------------------- attack #434401 | [192.168.1.0,192.168.1.255]</code></pre> But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block. Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges. SELECT typname FROM pg_type WHERE typname LIKE 'inet%range';</code></pre> typname ---------------- inetmultirange inetrange</code></pre> Danger zone! When creating a custom range type the subtype_diff function is more than just simple helper. It plays important role in indexing and query performance. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges. In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans. Performance deep dive: GiST vs GIN</a> </h2> Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt. Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree) is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index) would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space. The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with EXCLUDE constraints. GiST is mandatory there. Therefore while GiST index is going to be preferred for cases like CREATE TABLE seat_holds ( hold_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, seat_id INTEGER NOT NULL, hold_period TSTZRANGE NOT NULL );</code></pre> the GIN index is actually preferred for CREATE TABLE venue_blackouts ( venue_id INTEGER, blocked_dates DATERANGE NOT NULL );</code></pre> The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day. The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available. Conclusion</a> </h2> From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design. But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article. For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes. Your future self - the one debugging a problems in least convenient time - will thank you. PostgreSQL maintenance without superuser 2025-09-13T20:55:00+00:00 How many people/services have superuser access to your PostgreSQL cluster(s)? Did you ever ask why your software engineers might need it? Or your BI team? Why those use cases require same privileges as someone who can drop your databases? The answer isn't because these operations are inherently dangerous - it's because PostgreSQL historically offered limited options for operational access or simply because not enough people are aware of the options. So the common practice is to either got basic permissions or handover the keys to the kingdom. PostgreSQL's built-in predefined roles solve this problem by providing purpose-built privileges for common maintenance tasks. Instead of granting superuser access for routine operations, you can delegate specific capabilities - monitoring teams get comprehensive observability access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need, nothing more. What are Predefined Roles?</a> </h2> PostgreSQL's built-in administrative roles are purpose-built permission sets that solve the superuser dilemma for common maintenance tasks. Out of the box, there are 15 predefined roles that provide granular access to specific operational capabilities without requiring full superuser privileges. While you can view their list and description in official documentation</a>, in this article we will explore them bit more thoroughly and at the same time look into system catalogs to understand them better. The individual roles can be grouped by their functionality and most of them are quite easy to grasp, ranging from simple monitoring access to powerful filesystem operations that require careful consideration. Data Access Role pg_database_owner</code> - Database-specific ownership (special case)</li> pg_read_all_data</code> - Read access to all tables, views, sequences</li> pg_write_all_data</code> - Write access to all tables, views, sequences</li> </ul> Monitoring & Observability pg_monitor</code> - Which is actually monitoring meta role that contains 3 roles listed below</li> pg_read_all_settings</code> - Configuration access</li> pg_read_all_stats</code> - Statistics views</li> pg_stat_scan_tables</code> - Table scanning for stats</li> </ul> System Operations pg_signal_backend</code> - Cancel queries, terminate sessions</li> pg_checkpoint</code> - Run CHECKPOINT command</li> pg_maintain</code> - VACUUM, ANALYZE, REINDEX operations (PostgreSQL 17+)</li> pg_signal_autovacuum_worker</code> - Signal autovacuum worker (PostgreSQL 18+)</li> </ul> File System Access pg_read_server_files</code> - Read files from server filesystem</li> pg_write_server_files</code> - Write files to server filesystem</li> pg_execute_server_program</code> - Execute programs on server</li> </ul> And specialised use cases pg_create_subscription</code> - Logical replication management</li> pg_use_reserved_connections</code> - Connection reservation (PostgreSQL 16+)</li> </ul> Why Use Predefined Roles?</a> </h2> The primary benefit of predefined roles is expanding the pool of users who can safely manage PostgreSQL databases. Traditional PostgreSQL administration created an artificial binary choice: either users had basic access with limited capabilities, or they required full superuser privileges for operational tasks. This forced many organizations to grant excessive permissions simply to perform routine operations like monitoring, backups, or maintenance. Predefined roles break this limitation by allowing more granular control over operational tasks without distributing superuser privileges. Instead of having a small number of highly privileged superusers handling all administrative work, organizations can now delegate specific capabilities to appropriate operational roles - monitoring teams get monitoring access, backup services get data reading capabilities, and maintenance scripts get precisely the permissions they need. The deeper benefit is abstraction. Not only do you switch from manually managing permissions on individual system objects, you switch to managing logical capability sets. This creates a clear separation between what you want to allow and how it's actually implemented. The advantage is clearly demonstrated in the following example -- granting select on each schema separately GRANT USAGE ON SCHEMA finance, hr, app, audit TO analytics_team; GRANT SELECT ON ALL TABLES IN SCHEMA finance TO analytics_team; GRANT SELECT ON ALL SEQUENCES IN SCHEMA finance TO analytics_team; -- versus one time setup GRANT pg_read_all_data TO analytics_team;</code></pre> The predefined roles approach automatically covers: All current tables, views and sequences</li> All schemas</li> Future objects created after the GRANT</code></li> </ul> And the good news is the benefits don't end here. The real thing on operational level comes with the scope of those predefined roles - they apply at cluster level (instead on database level). Only notable exception is role pg_database_owner</code> which we will cover next. Before going there, let's briefly mention second benefit, and that's the fact any new operational features in PostgreSQL will be covered by those predefined roles automatically. The Evolution of Predefined Roles</a> </h2> Understanding when and why predefined roles were introduced provides important insight into PostgreSQL's operational priorities and helps explain some of the design decisions. The journey from a single role in PostgreSQL 9.6 to today's comprehensive set reflects real-world pain points that PostgreSQL developers encountered in production environments. PostgreSQL 9.6 (2016) introduced first predefined role pg_signal_backend</code> to address common frustration - inability to cancel running query without giving away superuser rights. -- This became possible in 9.6 without superuser privileges SELECT pg_cancel_backend(12345); -- Cancel a query SELECT pg_terminate_backend(12345); -- Terminate a session</code></pre> PostgreSQL 10 (2017) brought biggest expansion of predefined roles with, adding four monitoring specific roles to improve database observability, and support growing importance of database monitoring in production environments. -- Create the monitoring user CREATE USER postgres_exporter WITH PASSWORD 'monitoring_password'; -- Grant comprehensive monitoring access GRANT pg_monitor TO postgres_exporter;</code></pre> Giving access to variety of metrics and settings. With single GRANT</code> you can access data: From pg_read_all_settings (inherited via pg_monitor) to SELECT * FROM pg_settings</code> - Configuration parameters for alerting on misconfigurations and configuration metrics like shared_buffers, work_mem, max_connections</li> </ul> From pg_read_all_stats (inherited via pg_monitor): -- Database-wide statistics (connections, transactions, blocks read/written) SELECT * FROM pg_stat_database -- Current query activity and connection states SELECT * FROM pg_stat_activity -- Replication lag and status SELECT * FROM pg_stat_replication -- Background writer performance SELECT * FROM pg_stat_bgwriter -- Lock contention monitoring SELECT * FROM pg_locks</code></pre> From pg_stat_scan_tables (inherited via pg_monitor): Enhanced table statistics collection with sequential scan information</li> More detailed I/O statistics for table access patterns</li> </ul> The pg_monitor</code> role is particularly clever in its design. Rather than being a single monolithic permission set, it's actually a composite of the other three roles. This allows for granular access when needed - you can grant pg_read_all_stats for basic monitoring without including configuration access, or grant the full pg_monitor bundle for comprehensive monitoring capabilities. PostgreSQL 11 (2018) laid the foundation for secure file system operations with the introduction of three powerful roles that fundamentally changed how PostgreSQL handles server-side file access: pg_read_server_files</code></li> pg_write_server_files</code></li> pg_execute_server_program</code></li> </ul> Addressing many long-standing issues with providing file system access, like ETL processes that needs to read CSV files and data processing pipelines often involving external programs. The file system roles represented a significant security advancement. Before PostgreSQL 11, operations like server-side COPY FROM file were superuser-only, forcing administrators to either grant excessive privileges or find workarounds. PostgreSQL 14 (2021) introduced three significant roles that addressed different data access patterns: pg_read_all_data</code></li> pg_write_all_data</code></li> pg_database_owner</code></li> </ul> The first two solved a common backup and analytics challenge. The special case is pg_database_owner</code> which we will cover later. -- create ETL extraction user CREATE USER etl_extractor WITH PASSWORD 'extract_password'; GRANT pg_read_all_data TO etl_extractor; -- create ETL loader with write access CREATE USER etl_loader WITH PASSWORD 'loader_password'; GRANT pg_write_all_data TO etl_loader;</code></pre> One caveat that comes with those ALL data roles is that they still (despite what the name might suggest) respect Row Level Security policies (RLS). This is a design choice of PostgreSQL which prioritises security by default, even for users with broad data access. The only way to avoid RLS is to explicitly grant the BYPASSRLS</code> role attribute. PostgreSQL 15 (2022) expanded set with pg_checkpoint</code>, reflecting the reality that checkpoint operations are sometimes needed for maintenance but don't require full superuser privileges. PostgreSQL 16 (2023) expanded the list by introduction of pg_use_reserved_connections</code> - resolving a critical problem with high-traffic databases, where previously only superusers could access reserved connections. Introduction of this roles expanded the pool of potential users who might be able to resolve incidents or high-load situations. For example with pg_signal_backend</code> grant.</li> pg_create_subscription</code> - with ability to create Logical Replication</a> subscriptions and confirming the place of PostgreSQL in the distributed database scenarios.</li> </ul> PostgreSQL 17 (2024) finally saw pg_maintain</code>, role that had a turbulent development history (initially committed for PostgreSQL 16), allowing non-superusers to perform crucial maintenance tasks like VACUUM, ANALYZE, REINDEX, REFRESH MATERIALIZED VIEW, CLUSTER, and LOCK TABLE. The pg_maintain</code> role is particularly valuable for: Automated maintenance scripts that can now run with minimal privileges, reducing the security footprint of scheduled maintenance operations.</li> Operational teams managing multiple databases who need consistent maintenance capabilities without requiring superuser access to each database.</li> Cloud and managed environments where maintenance operations need to be delegated to operations staff without granting broader system access.</li> </ul> PostgreSQL 18 (2025) continues the trend with pg_signal_autovacuum_worker</code> expanding the ability for non-superusers to signal autovacuum workers specifically, enabling them to cancel vacuum operations on specific tables or terminate autovacuum sessions that may be causing performance issues during critical periods. What "all data" actually means</a> </h2> Per Stephen Frost's introducing commit</a>, the design intent of pg_read_all_data</code> is "running pg_dump as a non-superuser without explicitly granting access to all tables, schemas, et al." — and the role's reach follows from that, going beyond the "tables, views, sequences" the official documentation</a> calls out. A complete pg_dumpall</code> includes more than application data: stored password material for every role, foreign-server credentials, subscription connection strings, and the contents of large objects. The role permits reading all of it, regardless of the ordinary REVOKEs and view-layer masks PostgreSQL applies elsewhere. This is intentional, since each is something a complete dump genuinely needs to ship. pg_read_all_data</code> is closer to a "complete cluster dump" role than a "general read-only user" role. For analytics, BI, monitoring, or audit identities, narrower mechanisms (pg_monitor</code>, explicit per-schema GRANT SELECT</code>, application-specific roles) are usually what you actually want. </div> The Magic of pg_database_owner</a> </h2> If you paid attention, there's one role that stands apart from all others - pg_database_owner</code>. While it does not provide any specific privileges that wouldn't be possible without superuser access, it serves as a marker for the owner of the database. This role is crucial for maintaining database ownership and ensuring that the database is managed by the correct user. Before PostgreSQL 15 you would have public</code> schema owned by postgres</code> user. demo=> \dn List of schemas Name | Owner --------+---------- public | postgres</code></pre> It came with the baggage of relaxed default permission, allowing every user (PUBLIC) to create objects. Making this a maintenance nightmare. Starting with PostgreSQL 15 and above, the public</code> schema is owned by pg_database_owner</code> role. demo=# \dn List of schemas Name | Owner --------+------------------- public | pg_database_owner</code></pre> Unlike any other roles, pg_database_owner</code> has a unique behaviour - it membership changes with the current database. As the code snippet above shows the "shape-shifting nature" of this role fixed the public</code> schema and complexity of the permissions associated with it. Another special behaviour is that the role itself does not come with any permissions. Let that sink in - zero permissions. Its power only comes from what you grant it or inherit. It opens up system for elaborate template inheritance. -- define functionality in template1 \c template1 postgres CREATE OR REPLACE FUNCTION db_owner_stats() RETURN ... SECURITY DEFINER; GRANT EXECUTE ON FUNCTION db_owner_stats() TO pg_database_owner; -- and let it automatically apply to all new databases CREATE DATABASE app_prod OWNER first_app; CREATE DATABASE demo_prod OWNER second_app; -- both first_app and second_app automatically inherit db_owner_stats function on their databases</code></pre> And when we are talking about magic, the special status of the role prevents this role to be granted to any other role. demo=# GRANT pg_database_owner TO labs_app; ERROR: role "pg_database_owner" cannot have explicit members</code></pre>Conclusion</a> </h2> Predefined roles transform PostgreSQL administration from ad-hoc permission management into systematic capability delegation. They solve the fundamental problem of needing operational access without operational trust. The evolution continues - each PostgreSQL release adds new predefined roles addressing real-world operational challenges. The pattern is established: identify common pain points, create focused roles, eliminate the need for superuser privileges in routine operations. The next time you reach for superuser access to solve an operational problem, check if PostgreSQL has already provided a predefined role for exactly that purpose. You'll likely find it already has one. Update, 27 May 2026: Added What "all data" actually means</a> on pg_read_all_data</code>'s reach beyond tables, views, and sequences. Beyond the Basics of Logical Replication 2025-06-24T07:10:00+00:00 With First Steps with Logical Replication</a> we set up a basic working replication between a publisher and a subscriber and were introduced to the fundamental concepts. This part covers the operational side: initial data copies, monitoring, evolving publications, and how logical decoding actually works. Initial Data Copy</a> </h2> As we demonstrated in the first part, when setting up the subscriber, you can opt out of the initial data copy with WITH (copy_data = false)</code>. The default copy is convenient, but it has characteristics you should understand before relying on it in a production environment. The mechanism effectively asks the publisher to copy the table data by taking a snapshot (courtesy of MVCC), sending it to the subscriber, and thanks to the replication slot "bookmark," seamlessly continues streaming the changes from the point the snapshot was taken. Simplicity is the key feature here, as a single command handles the snapshot, transfer, and transition to ongoing streaming. The trade-off is performance, because each table is copied by a single worker using COPY</code>. Small tables finish quickly, but tables with gigabytes of data will take noticeably long. Although parallelism can be controlled by the max_sync_workers_per_subscription</code> configuration parameter, it still might leave you waiting for hours (and days) for any real-life database to get replicated. You can monitor whether the tables have already been synchronized or are still waiting/in progress using the pg_subscription_rel</code> catalog. SELECT srrelid::regclass AS table_name, srsubstate FROM pg_subscription_rel;</code></pre> Each table will have one of the following states: i</code> not yet started</li> d</code> copy is in progress</li> f</code> finished the table copy</li> s</code> synchronized with the main apply stream</li> r</code> ready, replicating normally</li> </ul> Tables reach r</code> independently, so changes stream to already-synchronized tables even while others are still copying. Nevertheless, the replication slot retains the LSN position, which means that the publisher will retain WAL files until the subscriber has caught up. Manual Synchronization for Production Workloads</a> </h2> As mentioned above, the implicit copy might bring performance trade-offs that are simply too much for production use, unless you consider it while designing your logical replication topology. In all other scenarios, going with manual synchronization is the way forward. The entire process has one leading principle: the data you load manually must be consistent with the point in time from which the logical replication stream begins. This is achieved by creating a logical replication slot before (or synchronized with) the point at which you restore the data (PITR) and enabling it once the data is transferred. Only this will allow you to correctly apply all subsequent changes on the subscriber. There are several ways to achieve this, and you will have to evaluate the mechanism based on the available constraints in your particular use case: Use the backup & restore mechanism that supports Point-in-Time Recovery (PITR) if the amount of data to be synchronized approaches the total size of the backed-up data (i.e., most of the tables).</li> Orchestrate the data change ingestion on the publisher with the backup, by stopping the incoming changes (for example, during a planned maintenance window) and only restoring them when a consistent snapshot is available to a known LSN. This is for cases where you can control table ingestion and the amount of data being transferred fits the maintenance window.</li> If none of this is available, you might be able to manually advance replication slots to a predefined LSN. This is a last resort; advancing slots by hand is easy to get wrong.</li> </ol> The recommended way for the most reliable backup and restore handling is the use of pgBackRest, as it allows you to restore a particular backup in advance and apply only changes needed later to advance to the selected Point-in-Time, hence significantly reducing the time required for the initial sync. On the other hand, if you are only using a small portion of the data within the backup on your subscriber, it might create resource constraints not acceptable for the setup. While pg_dump can't be considered a reliable backup tool, it can handle a consistent copy of a small subset of tables. The catch is where the snapshot comes from: a snapshot exported with pg_export_snapshot()</code> is not synchronized with a replication slot's start point. To get a slot-consistent snapshot, create the slot over a replication connection with CREATE_REPLICATION_SLOT ... EXPORT_SNAPSHOT</code>, keep that session open, and pass the returned snapshot name to pg_dump</code>: pg_dump -d publisher_db --snapshot="00000004-000004A2-1" --data-only -Fc -f initial_data.dump</code></pre> The topic of consistent backups and Point-in-Time Recovery is beyond the scope of this guide. Monitoring Logical Replication</a> </h2> Once you set up your publishers and subscribers and have the initial data in place, it's time to start thinking about how to keep that process running. For day-to-day operations, the basic building block for monitoring is the pg_replication_slots</code> catalog. Sample query: SELECT slot_name, plugin, slot_type, active, pg_size_pretty( pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) ) AS wal_retained_size FROM pg_replication_slots WHERE slot_type = 'logical';</code></pre> giving you a result like: -[ RECORD 1 ]-----+----------------------------- slot_name | my_sample_subscription plugin | pgoutput slot_type | logical active | t wal_retained_size | 49 MB</code></pre> You should look for two things: active</code> indicates whether the slot is being used by any connection. An inactive slot is acceptable for short durations, but a slot that stays inactive is a sign that the subscriber is down or not configured properly.</li> The computed wal_retained_size</code> is a critical metric. An increasing value indicates problems consuming changes by the subscriber, and your cluster might be at risk of running out of disk space.</li> </ul> The active</code> flag also catches dangling slots, where the subscriber has been destroyed (or simply wiped) without properly dropping the subscription, leaving the replication slot behind. In such cases, the only direct response is dropping the slot by its name: SELECT pg_drop_replication_slot('my_dangling_slot');</code></pre> If you want to dig deeper, you can also use pg_stat_replication</code> that provides more (albeit volatile) data about the replication status. It can give you other metrics, like replay_lag</code>. To wrap up the monitoring part, you can either design your custom checks or use a modified version of the query above: SELECT slot_name, CASE WHEN active THEN 1 ELSE 0 END as active_status, pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) as wal_bytes_retained, extract(epoch from now()) as time FROM pg_replication_slots WHERE slot_type = 'logical';</code></pre> as part of a Grafana alerting rule: Alert Rule: "Inactive Replication Slot" Condition: - Query: active_status - Reducer: last() - Evaluator: below 1 - For: 30m # if inactive for 30+ minutes Additional conditions (AND): - Query: wal_bytes_retained - Reducer: last() - Evaluator: above 1073741824 # 1GB limit </code></pre> Nevertheless, you need to evaluate the monitoring details applicable for your particular use case(s). There might also be other options, for example, if your architecture allows for it, to use max_slot_wal_keep_size</a> and mark the replication slot invalid and release the WAL files when a particular slot falls behind the configured value (for example, 100GB). While this might sound reasonable, you must consider whether your application or use case can recover from such data loss. Evolving Publications</a> </h2> Going back to the first part of our journey into logical replication, there's an important distinction to make. While we have described the initial use case, in real life your publisher schema will change as time goes by. Tables will be created and included implicitly (via FOR ALL TABLES</code>) or explicitly added/removed. ALTER PUBLICATION my_publication ADD TABLE public.new_table; ALTER PUBLICATION my_publication DROP TABLE public.old_table;</code></pre> Adding a new table to the publication changes nothing for an existing subscription on its own. The only way for the subscriber to reflect the changes in the underlying configuration is to refresh the subscription. It's a process where PostgreSQL compares the current subscription table list with the publication table list, (if configured to do so) starts the initial synchronization process and once finished goes to apply streamed changes. As mentioned, you can refresh a subscription with or without copying the initial data. The copy_data</code> option controls this behavior and is currently the only supported setting. ALTER SUBSCRIPTION my_subscription REFRESH PUBLICATION; ALTER SUBSCRIPTION my_subscription REFRESH PUBLICATION WITH (copy_data = false);</code></pre> When you remove a table from the publisher, the situation is much simpler as the changes for that particular table are no longer considered for logical decoding. But what happens if you add a new table to the publication without refreshing the subscription? Even though the WAL includes changes for all tables and logical decoding processes all published table changes, there is no additional WAL retention to consider. The subscriber advances confirmed_flush_lsn</code> as before: the initial state for the newly added table has not yet been recorded, so its streamed changes are (for the moment) ignored. Logical Decoding</a> </h2> If you paid attention throughout our journey or already worked a bit with logical replication, you might have come across something called the pgoutput</code> plugin. This is the built-in default mechanism responsible for logical decoding, the process that goes through the WAL stream and transforms it into a higher-level format. Where physical replication ships "byte X at offset Y, in page Z", logical decoding translates the WAL stream into row-level changes. It uses the catalog of the source database to map object identifiers to schema, table, and column names, along with the respective values (both old and new), and reassembles the changes into the correct transaction flow. The plugin's job is to format that output in a particular way and by itself is not aware of: any transport layer details</li> replication slots</li> any retry logic</li> </ul> The built-in plugins are just a start, and developers can create custom output plugins. We can also look inside the decoded stream manually. Consider this example: -- create new publication CREATE PUBLICATION all_data FOR ALL TABLES; -- create new replication slot with pgoutput plugin SELECT pg_create_logical_replication_slot('plugin_demo', 'pgoutput'); -- sample table CREATE TABLE IF NOT EXISTS demo_table ( id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT, email TEXT ); -- generate some changes with insert/update/delete INSERT INTO demo_table (name, email) VALUES ('John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 1; DELETE FROM demo_table WHERE id = 1; -- get binary changes SELECT * FROM pg_logical_slot_get_binary_changes('plugin_demo', NULL, NULL, 'proto_version', '1', 'publication_names', 'all_data'); -- drop the replication slot and the publication SELECT pg_drop_replication_slot('plugin_demo'); DROP TABLE demo_table; DROP PUBLICATION all_data;</code></pre> Giving us a peek into the recently performed changes (output is shortened): lsn | xid | data ------------+------+--------------------------------- 1/D6997380 | 1038 | \x4200000001d69974180 1/D6997380 | 1038 | \x52000042057075626c696300646... 1/D6997380 | 1038 | \x49000042054e000374000... 1/D6997448 | 1038 | \x430000000001d6997... 1/D6997448 | 1039 | \x4200000001d6997... 1/D6997448 | 1039 | \x49000042054e0.... 1/D6997510 | 1039 | \x430000000001d6...</code></pre> This is correct, but won't help us demonstrate the flow: pgoutput</code> is the plugin used for actual replication, and its output is binary. PostgreSQL also ships the test_decoding</code> plugin, which decodes the changes into a human-readable text representation. -- create replication slot with test_decoding plugin -- (unlike pgoutput, test_decoding doesn't use publications, so none is needed) SELECT pg_create_logical_replication_slot('plugin_demo', 'test_decoding'); CREATE TABLE IF NOT EXISTS demo_table ( id INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT, email TEXT ); -- generate some changes with insert/update/delete INSERT INTO demo_table (name, email) VALUES ('John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 1; DELETE FROM demo_table WHERE id = 1; -- get human readable changes SELECT * FROM pg_logical_slot_get_changes('plugin_demo', NULL, NULL); -- drop the replication slot and the table SELECT pg_drop_replication_slot('plugin_demo'); DROP TABLE demo_table;</code></pre> This time the output is actually going to be helpful. lsn | xid | data ------------+------+----------------------------------------------------------------------------------------------------- 1/D69CCC30 | 1048 | BEGIN 1048 1/D69CCC98 | 1048 | table public.demo_table: INSERT: id[integer]:1 name[text]:'John Doe' email[text]:'john@example.com' 1/D69CCDC0 | 1048 | COMMIT 1048 1/D69CCDC0 | 1049 | BEGIN 1049 1/D69CCDC0 | 1049 | table public.demo_table: UPDATE: id[integer]:1 name[text]:'John Doe' email[text]:'doe@example.com' 1/D69CCE50 | 1049 | COMMIT 1049 1/D69CCE50 | 1050 | BEGIN 1050 1/D69CCE50 | 1050 | table public.demo_table: DELETE: id[integer]:1 1/D69CCEC0 | 1050 | COMMIT 1050</code></pre> Just reading it, you can easily follow the sequence of the transaction (auto-commit in psql) and track individual changes. We can also reiterate the importance of REPLICA IDENTITY</code> with this example. The id</code> column of demo_table</code> is declared PRIMARY KEY</code>, and the default replica identity uses exactly that to identify rows. Switching the table to REPLICA IDENTITY FULL</code> shows, via test_decoding</code>, how much more verbose the decoded stream becomes. (This continues the previous example; setup and teardown omitted.) ALTER TABLE demo_table REPLICA IDENTITY FULL; INSERT INTO demo_table (id, name, email) OVERRIDING SYSTEM VALUE VALUES (999, 'John Doe', 'john@example.com'); UPDATE demo_table SET email = 'doe@example.com' WHERE id = 999; DELETE FROM demo_table WHERE id = 999; SELECT * FROM pg_logical_slot_get_changes('plugin_demo', NULL, NULL);</code></pre> And you can observe the changes. While by its nature INSERT</code> gives you the same output (all values are being sent), UPDATE</code> and DELETE</code> are no longer relying on the primary key. Instead, they have to provide all values, for the UPDATE</code> case both old and new data: lsn | 1/D69F8770 xid | 1062 data | table public.demo_table: UPDATE: old-key: id[integer]:999 name[text]:'John Doe' email[text]:'john@example.com' new-tuple: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</code></pre> and for DELETE</code> the old data. lsn | 1/D69F8828 xid | 1063 data | table public.demo_table: DELETE: id[integer]:999 name[text]:'John Doe' email[text]:'doe@example.com'</code></pre> It's obvious how a misconfigured replica identity increases the amount of data decoded and replicated. Fine-grained Publication Control</a> </h2> Before we introduce the advanced replication topologies in later parts of this series, it's worth knowing how precisely you can control what data reaches the subscribers. Filtering what you publish cuts replication overhead and network traffic, and keeps data the subscriber shouldn't see out of the stream. The first option to filter the data is by explicit column_list</code>, allowing you to exclude sensitive (PII or similar) or unnecessary data. The column list must include the replica identity columns (typically the primary key) if the publication replicates UPDATE</code> or DELETE</code>; otherwise those operations fail on the publisher. CREATE PUBLICATION hr_analytics FOR TABLE hr.employees (employee_id, first_name, last_name, department, start_date);</code></pre> Similar to column selection, you can control what operations a publication replicates. The available options are 'insert', 'update', 'delete', and 'truncate'. CREATE PUBLICATION hr_analytics FOR TABLE hr.employees (employee_id, first_name, last_name, department, start_date) WITH (publish = 'insert,delete');</code></pre>Securing Logical Replication</a> </h2> Limiting what a publication carries only helps if access is limited too. A restricted publication means little when the replication role can read (or another publication can expose) the same data without restrictions. So far in this series, we have relied for simplicity on SUPERUSER</code> access. To design logical replication for real-life production use, you need a least-privilege model. The permissions split across two roles. Creating a publication requires the CREATE</code> privilege on the database, adding a table to one requires owning that table, and FOR ALL TABLES</code> publications require a superuser. The role a subscription connects as is a separate concern: it needs the REPLICATION</code> attribute, CONNECT</code> on the database, USAGE</code> on the schema, and SELECT</code> on the published tables (or columns) for the initial data copy. Row-level security deserves explicit attention: if the replication role lacks SUPERUSER</code> and BYPASSRLS</code>, publisher row security policies can execute during the initial copy. If the role does not trust all table owners, add options=-crow_security=off</code> to the connection string; replication then halts on tables with a row security policy instead of executing it. Using the fine-grained example used above, we can demonstrate the setup. First, creating a role with LOGIN</code> and REPLICATION</code> clauses and granting database and schema access: CREATE ROLE hr_analytics_role WITH LOGIN REPLICATION PASSWORD 'a_strong_password'; GRANT CONNECT ON DATABASE my_publisher_db TO hr_analytics_role; GRANT USAGE ON SCHEMA hr TO hr_analytics_role;</code></pre> The crucial step is to correctly define the GRANT</code> for selecting the data. GRANT SELECT (employee_id, first_name, last_name, department, start_date) ON hr.employees TO hr_analytics_role;</code></pre> This way, even if someone gains access to the replication role, they cannot read data beyond what was explicitly granted. First steps with Logical Replication in PostgreSQL 2025-06-11T00:00:00+00:00 PostgreSQL's logical replication streams row-level changes from one PostgreSQL instance to another using a publish-subscribe model. Unlike physical replication, which copies the whole cluster byte for byte, it lets you pick what to replicate and where to send it, which makes it a building block for scaling out, distributing load, and integrating PostgreSQL with the rest of your architecture. Physical vs. Logical Replication</a> </h2> Both kinds of replication in PostgreSQL are built on top of the Write-Ahead Log (WAL). The WAL is a sequential, append-only log that records every change made to the cluster data. For durability purposes, all modifications are first written to the WAL before the modified data pages are flushed to disk. This allows PostgreSQL to recover from crashes by replaying logged changes. Versioned changes, necessitated by concurrent transactions, are managed through Multi-Version Concurrency Control (MVCC). Instead of overwriting data directly, MVCC creates multiple versions of rows, allowing each transaction to see a consistent snapshot of the database. It is the WAL that captures these versioned changes along with the transactional metadata to ensure data consistency at any given point in time. Physical replication is built directly on the Write-Ahead Log. It enables streaming of the binary WAL data from the primary server to one or more standby servers (replicas), effectively creating a byte-for-byte copy of the entire cluster. This requirement makes the replicas read-only, making them ideal candidates for failover or scaling purposes. Logical replication, while also built on top of the WAL data, takes a fundamentally different approach. Instead of streaming raw change data, it decodes the WAL into logical, row-level changes (INSERT</code>, UPDATE</code>, and DELETE</code>) and only then sends them to the subscribers using a publish-subscribe model. This allows selective replication and writable subscribers that are not strictly tied to a single publisher, at a cost: logical replication does not replicate DDL changes. </th> Physical</th> Logical</th></tr></thead> Data Streamed</td> Binary WAL segments</td> Row-level SQL changes</td></tr> Scope</td> Byte-for-byte stream</td> Selected tables</td></tr> Node Type</td> Read-only standby</td> Fully writable instance</td></tr> PostgreSQL Version</td> All servers must match major version</td> Supports across versions</td></tr> Database Schema</td> Changes automatically replicated</td> Changes must be applied on subscriber(s) separately</td></tr> Use Case</td> Failover, high-availability, and read scaling</td> Integration, zero-downtime upgrades and schema migrations, complex topologies</td></tr> </tbody></table> Physical replication is your go-to for high availability and disaster recovery, where you want a fast, exact copy of the entire database cluster that can take over in case of failure. It’s simple to set up and very efficient but limited in flexibility. Logical replication fits when you need fine-grained control over what data is replicated, require writable replicas, or want to integrate PostgreSQL with other systems or versions. Typical uses are zero-downtime major-version upgrades, multi-region deployments, and feeding downstream systems such as a search index or a warehouse. Setting up Logical Replication</a> </h2> Enough of boring theory for now. To get started, you will need two instances of PostgreSQL, whether you provision two virtual machines or two clusters running on the same computer. We will call one publisher and the other subscriber. Preparing the Publisher</a> </h3> First, you need to prepare the publisher to emit the logical changes. You will need to modify postgresql.conf</code> (or add particular conf.d</code> configuration) with the following parameters: wal_level = logical max_replication_slots = 10 max_wal_senders = 10</code></pre> Your publisher also needs to be reachable by the subscriber, in most cases via a TCP socket. listen_addresses = '*'</code></pre> Here, the configuration is: wal_level</code></a> set to logical</code> is a crucial piece of the configuration. It tells PostgreSQL how much information to write to the WAL, in this case, to support logical decoding.</li> max_replication_slots</code></a> defines the maximum number of replication slots that can be created on the server. Each logical replication subscription needs its own slot.</li> max_wal_senders</code></a> should be set high enough to accommodate all expected concurrent replication connections from subscribers, and should be higher than or equal to max_replication_slots</code>. Each active replication subscription consumes a wal_sender slot.</li> </ul> You will also need to configure client authentication in pg_hba.conf</code> to allow the subscriber to connect (for simplification, we allow all users). Note that the replication</code> keyword in pg_hba.conf</code> matches only physical replication connections; a logical replication subscriber connects to a regular database, so the entry must name that database (or all</code>): # TYPE DATABASE USER ADDRESS METHOD host logical_demo_publisher all subscriber_ip/32 scram-sha-256</code></pre> While for the purposes of the article we will assume the use of a superuser, making it convenient: CREATE USER my_user_name SUPERUSER PASSWORD 'my_secure_password';</code></pre> It is recommended to use a dedicated replication user for real-life deployments: CREATE USER replication_user WITH REPLICATION ENCRYPTED PASSWORD 'my_secure_password';</code></pre> Once configured, restart PostgreSQL for the configuration changes to take effect. After that, connect to the server and prepare the environment. -- example for psql CREATE DATABASE logical_demo_publisher; \c logical_demo_publisher;</code></pre> Create a sample table and seed initial data: CREATE TABLE products ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, category TEXT, price DECIMAL(10, 2) NOT NULL, stock_quantity INT DEFAULT 0, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, description TEXT, is_active BOOLEAN DEFAULT TRUE ); -- seed some data (10 records) INSERT INTO products ( name, category, price, stock_quantity, description, is_active ) SELECT 'Product Batch ' || s.id AS name, CASE (s.id % 5) WHEN 0 THEN 'Electronics' WHEN 1 THEN 'Books' WHEN 2 THEN 'Home Goods' WHEN 3 THEN 'Apparel' ELSE 'Miscellaneous' END AS category, ROUND((RANDOM() * 500 + 10)::numeric, 2) AS price, FLOOR(RANDOM() * 200)::int AS stock_quantity, 'Auto-generated description for product ID ' || s.id || '. Lorem ipsum dolor sit amet, consectetur adipiscing elit.' AS description, (s.id % 10 <> 0) AS is_active FROM generate_series(1, 10) AS s(id);</code></pre> The final step for the publisher is to create a publication to define what data we want to publish. CREATE PUBLICATION my_publication FOR TABLE products;</code></pre>Preparing the Subscriber</a> </h3> Next, we will use our subscriber instance to receive the logical changes. There's no need to make any configuration changes just for subscribing, as it's not emitting changes itself. Create the target database and schema. This step matters: logical replication does not create tables on the subscriber, so the target table must already exist and match the published one. -- example for psql CREATE DATABASE my_subscriber_db; \c my_subscriber_db; CREATE TABLE products ( id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, name TEXT NOT NULL, category TEXT, price DECIMAL(10, 2) NOT NULL, stock_quantity INT DEFAULT 0, created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP, description TEXT, is_active BOOLEAN DEFAULT TRUE );</code></pre> Now we have the foundation for testing logical replication. The simplest way is to create a subscription using connection details and the name of the publication to subscribe to. CREATE SUBSCRIPTION my_subscription CONNECTION 'host=publisher_ip_address port=5432 user=your_replication_user password=my_secure_password dbname=logical_demo_publisher' PUBLICATION my_publication;</code></pre> my_subscription</code> should be a descriptive name for your subscription.</li> CONNECTION</code> defines how to connect to the publisher (which is a regular connection string and could also be a service</a>).</li> PUBLICATION</code> specifies the name of the publication you created earlier on the publisher.</li> </ul> If your connection was correct, the subscription will by default start with the initial data sync and listen for incoming changes. If you have used the queries above, you can validate that the data is now on the subscriber. # SELECT count(1) FROM products; count --------- 10 (1 row)</code></pre> The number of records should match the number of records created using generate_series</code> above (10 in our example). You can go ahead and insert a single row again on your publisher instance and validate the data being replicated to the subscriber. That's your first logical replication setup working. Core Concepts of Logical Replication</a> </h2> With a working setup in place, here are the moving parts that made it work. Publication</a> </h3> As the name implies, a publication is where it all starts. It's essentially a catalogue of data you offer from the publisher. You can publish</a> a number of objects: --- specific tables CREATE PUBLICATION my_publication FOR TABLE products, orders; --- everything in your database (make sure you really want to do this) CREATE PUBLICATION all_data FOR ALL TABLES; --- replicate specific columns only (PostgreSQL 15 and higher) CREATE PUBLICATION generic_data FOR TABLE products (id, name, price); --- filter published data (PostgreSQL 15 and higher) CREATE PUBLICATION active_products FOR TABLE products WHERE (is_active = true); --- filter different operations CREATE PUBLICATION my_publication FOR TABLE products, orders WITH (publish = 'insert'); --- or you can mix & match it to get exactly what you need CREATE PUBLICATION eu_customers FOR TABLE customers (id, email, country, created_at) WHERE (country IN ('DE', 'FR', 'IT')), TABLE orders (id, customer_id, total_amount) WHERE (total_amount > 100) WITH (publish = 'insert, update');</code></pre> As you can see, logical replication really gives you quite a lot of options and is far from the rigid, byte-for-byte copying of physical replication. This is exactly the characteristic that allows you to build complex topologies. Once you set up your publication(s), you can review it: --- in psql \dRp \dRp+ my_publication --- using pg_catalog SELECT * FROM pg_publication_tables WHERE pubname = 'my_publication';</code></pre> From these options, it's easy to think of a publication as a customisable data feed. Subscription</a> </h3> The other side of logical replication is a subscription. It defines what and how to consume events from a publisher. You subscribe to a publication using connection details and a number of options. --- basic subscription with full connection detail CREATE SUBSCRIPTION my_subscription CONNECTION 'host=publisher_host port=5432 user=repl_user password=secret dbname=source_db' PUBLICATION my_publication; -- subscription using service definition CREATE SUBSCRIPTION realtime_only CONNECTION 'service=my_source_db' PUBLICATION my_publication;</code></pre> The default behaviour of the subscription is to copy existing data, start immediately, and continue with streaming data changes. Once you start experimenting with logical replication, you can control that behaviour. --- subscribe without initial copy of the data CREATE SUBSCRIPTION streaming_only CONNECTION 'service=my_source_db' PUBLICATION my_publication WITH (copy_data = false); --- or defer the start for later CREATE SUBSCRIPTION manual_start CONNECTION 'service=my_source_db' PUBLICATION my_publication WITH (enabled = false);</code></pre> You can monitor your subscriptions: --- in psql \dRs \dRs+ my_subscription --- or their status using pg_catalog SELECT subname, received_lsn, latest_end_lsn, latest_end_time FROM pg_stat_subscription;</code></pre>Replication Slot</a> </h3> Publications and subscriptions establish the data flow, but there's a critical piece missing: how does the publisher keep track of multiple subscribers reading the WAL at different speeds? Replication slots are the answer. They act as a persistent bookmark in the WAL stream that tracks exactly where each subscriber is (or was last time), ensuring no changes are lost even if the connection stops. A replication slot can tell us quite a lot about itself. # SELECT * FROM pg_replication_slots; -[ RECORD 1 ]-------+--------------- slot_name | my_subscription plugin | pgoutput slot_type | logical datoid | 16390 database | logical_demo_publisher temporary | f active | t active_pid | 3162 xmin | catalog_xmin | 777 restart_lsn | 0/1DEFBF0 confirmed_flush_lsn | 0/1DEFC28 wal_status | reserved safe_wal_size | two_phase | f inactive_since | conflicting | f invalidation_reason | failover | f synced | f</code></pre> The most interesting attributes are: slot_name</code></li> plugin</code> used for logical replication</li> slot_type</code> confirming it's for logical replication</li> active</code> indicates whether there's a subscriber reading from this slot</li> </ul> And most important of those being: restart_lsn</code> identifying the LSN where this slot "holds" WAL files from being released</li> confirmed_flush_lsn</code> being the last LSN position the subscriber confirmed it has successfully processed (i.e., applied).</li> </ul> While we won't go into details about WAL, it's enough to say LSN (Log Sequence Number) is a unique address or position in the WAL that identifies the position in the stream of database changes. In most cases, you don't have to create replication slots manually, as subscriptions create and manage them. The persistent nature of the slots guarantees they survive the restart of both publisher and subscriber. If the subscription is not actively consuming the changes, the publisher won't release WAL files that contain changes a slot has not consumed. This prevents data loss, but can easily fill up disk if any of the subscribers fall too far behind. You can monitor the WAL retention per each replication slot. # SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained FROM pg_replication_slots; -[ RECORD 1 ]+-------------------- slot_name | my_subscription active | t wal_retained | 265 MB</code></pre>Replication Identity</a> </h3> As we have already covered, logical replication works with row-level changes, such as INSERT</code>, UPDATE</code>, and DELETE</code>. And not all operations are equal. While INSERT</code> is relatively straightforward (a new row is sent to the subscriber), for both UPDATE</code> and DELETE</code> operations, PostgreSQL needs to be able to uniquely identify the target row to modify on the subscriber. This is managed by the REPLICA IDENTITY</code> property of each published table. The default uses the table's primary key (it has one, right?). You can also point it at another unique index with USING INDEX</code>, or specify FULL</code>, which sends all the old column values of the row to find the target row (and should generally be avoided). PostgreSQL never falls back to FULL</code> on its own: without a primary key, the identity stays DEFAULT</code> but records nothing, and any UPDATE</code> or DELETE</code> on the published table fails with an error until you set an identity explicitly. In practice: Prefer REPLICA IDENTITY DEFAULT</code> (the primary key), the most efficient choice. If the table doesn't have a primary key, consider adding one (a surrogate key is always an option).</li> Use REPLICA IDENTITY USING INDEX index_name</code> if there is a better (or smaller) unique index matching the business logic or architecture of your database model.</li> </ul> There are special cases of what not to do when it comes to replication identity, but we will touch on those cases in the advanced section of this guide. Schema Changes</a> </h3> Although it was already mentioned, it's vital to reiterate that logical replication, as it streams only row-level changes rather than full WAL segments, does not automatically replicate Data Definition Language (DDL) changes. Any schema changes made on the publisher must be manually applied to all subscribers before data reflecting the change is replicated. We will cover schema specifically in later parts of this guide, but to sum it up, this is the recommended order of operations when it comes to schema changes: (Optional) but recommended for breaking changes, pause replication where applicable.</li> Apply and verify DDL changes to the subscribers.</li> Apply and verify DDL changes on the publisher.</li> Resume the replication if applicable.</li> </ol> While this might seem like a major drawback compared to physical replication, it's the characteristic that gives us the most flexibility. Other Considerations</a> </h3> As logical replication reliably handles row-level changes, it's crucial to understand other specific limitations and behaviours. Specifically: Sequences are not replicated. While they are used to generate the value on the publisher, and their value advances there, the corresponding sequence (if it exists) on the subscriber won't be updated.</li> Other database objects won't be replicated. While it might be understandable for views, stored procedures, triggers, and rules, you also can't rely on it for materialized views.</li> Special consideration (similar to schema changes) must be paid to user-defined data types. If a replicated table uses a user-defined type (e.g., a column using an ENUM</code>), the type must already be available on the subscriber(s) and have exactly the same name and structure. As ENUM</code>s are represented as ordered sets, even the order of the values matters.</li> </ul> Living with Logical Replication</a> </h2> In our previous example, we set up the publisher and subscriber, relied on the initial copy of the data, and let things run. Both of them will survive a restart, and thanks to the replication slot, they will keep going as soon as they come online. But what if you need to perform maintenance or do regular things like a schema update on the subscriber or the publisher? Sometimes you might need to temporarily stop the replication. PostgreSQL makes this straightforward with subscription management, allowing you to stop consuming the changes. ALTER SUBSCRIPTION my_subscription DISABLE;</code></pre> As we hinted earlier, when disabled, the subscription stops consuming changes from the publisher, while keeping the replication slot. This means: No new changes are applied on the subscriber.</li> WAL files will start to accumulate on the publisher (since the replication slot holds its position).</li> </ul> While disabled, you can't really check the status from the subscriber, as only the replication slot has the data. You can use the query we mentioned above on the publisher to check the status. SELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS wal_retained FROM pg_replication_slots;</code></pre> When ready, you can resume the subscription. ALTER SUBSCRIPTION my_subscription ENABLE;</code></pre>Coordinating Schema Changes</a> </h2> As we mentioned earlier, since logical replication does not automatically replicate DDL changes, you need to coordinate schema updates manually. Here's a basic overview of how to perform such a change, step by step. First, disable the replication on the subscriber and apply your schema changes. --- disable replication ALTER SUBSCRIPTION my_subscription DISABLE; --- apply changes on the subscriber ALTER TABLE products ADD COLUMN category_id INTEGER; CREATE INDEX products_by_category ON products(category_id);</code></pre> Only then can you apply the changes to the publisher. ALTER TABLE products ADD COLUMN category_id INTEGER; CREATE INDEX products_by_category ON products(category_id);</code></pre> And resume the replication. ALTER SUBSCRIPTION my_subscription ENABLE;</code></pre> If you applied the schema change to the publisher first, any new data using the new column would fail to replicate to the subscriber that doesn't have the column yet. Handling Incompatible Changes</a> </h2> PostgreSQL's default conflict resolution is very simple. When a conflict occurs, logical replication stops, and the subscription enters an error state. For example, if you consider a product row already present on the subscriber, you will see something like this in the log files: ERROR: duplicate key value violates unique constraint "products_pkey" DETAIL: Key (id)=(452) already exists. CONTEXT: processing remote data for replication origin "pg_16444" during message type "INSERT" for replication target relation "public.products" in transaction 783, finished at 0/1E020E8 LOG: background worker "logical replication apply worker" (PID 457) exited with exit code 1</code></pre> As you can see, logical replication stopped with an error code and won't continue until you manually resolve the conflict. To do so, you need to identify the conflicting data and decide what to do with it. -- option 1: remove the conflicting data DELETE FROM products WHERE id = 452; -- option 2: update the data to match the expected state UPDATE products SET name = 'New Product', price = 29.99 WHERE id = 452;</code></pre> Only after the conflict is resolved can the logical replication continue. This has been only a very simple example of a conflict that might arise during logical replication. Other examples might involve: Data problems, constraint violations, or type mismatch</li> Schema conflicts (missing or renamed table/column)</li> Permissions issues or row-level security</li> </ul> In all those cases, you still need to go and fix the problem before logical replication can resume.