Query</th>	Plan</th>	Time</th></tr></thead>
`count(DISTINCT user_id)</code></td>`	serial sort, 115MB to disk</td>	1211 ms</td></tr>
`count(*) FROM (… GROUP BY user_id)</code></td>`	parallel hash, in memory</td>	360 ms</td></tr> </tbody></table> If an approximate answer is acceptable, this is the problem HyperLogLog is meant to solve: its sketch is a small state that, in principle, has the combine function exact distinct counts lack, so partial sketches from workers should merge. The postgresql-hll</code> extension builds on that idea, and it is often suggested for dashboard-style distinct counts at the price of a bounded error rate.</div> Both return `50001`. The rewrite is about 3.4x faster here, and the gap should widen with the table: the serial sort's cost grows with row count, while the parallel hash keeps adding throughput with each worker. ORDER BY aggregates hit the same wall</a> </h2> The block is not specific to DISTINCT</code>. Any aggregate that needs its input in a particular order, the ordered-set and ordered aggregates, fails to parallelize for the same reason, because a worker's locally-ordered partial result cannot be merged without re-ordering across workers:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT</span> string_agg</span>(country, </span>','</span> ORDER BY</span> country) </span>FROM</span> events;</span></span></code></pre> Aggregate</span></span> -> Sort</span></span> Sort Key: country</span></span> -> Seq Scan on events</span></span></code></pre> string_agg</code>, array_agg</code>, json_agg</code> with an inner ORDER BY</code>, and percentile_cont</code>/percentile_disc</code> all land here. If you have an aggregate that insists on global order or global distinctness, assume it runs on one core until EXPLAIN</code> tells you otherwise.</p> The harder case: per-group distinct counts</a> </h2> The clean rewrite above is for a single distinct count over the whole table. The per-group version of the same query,</p> SELECT</span> country, </span>count</span>(</span>DISTINCT</span> user_id) </span>FROM</span> events </span>GROUP BY</span> country;</span></span></code></pre> is also serial (a GroupAggregate</code> over a sort on country, user_id</code>). The same idea applies, deduplicate first with a grouping the workers can split, then aggregate:</p> SELECT</span> country, </span>count</span>(</span></span>)</span></span> FROM</span> (</span>SELECT</span> country, user_id </span>FROM</span> events </span>GROUP BY</span> country, user_id) s</span></span> GROUP BY</span> country;</span></span></code></pre> This makes</em> the work parallelizable, but whether the planner actually picks the parallel path depends on cardinalities and cost. In my testing the overall-count rewrite parallelized reliably, while this stacked-grouping form sometimes stayed serial because the planner judged the two hash-aggregate layers cheap enough already. The rule is the same, push the distinctness into a GROUP BY</code> first, but look at the EXPLAIN</code> before you trust it.</p> When to actually care</a> </h2> None of this actually matters on a small table. If the scan is a few thousand rows, serial is instant and the rewrite only adds noise. The distinct-aggregate penalty is a function of how many rows the single core has to sort, so it shows up exactly where it hurts Large fact tables or nightly rollups that runs on single core while rest sit idle.</p> The tell in EXPLAIN (ANALYZE)</code> is unmistakable once you know it: a top-level Aggregate</code> with no Gather</code> beneath it, a big Sort</code> with an external merge ... Disk:</code> line, and a single loops=1</code> scan of the whole table. If you see that shape above a distinct or ordered aggregate on a table that matters, it's worth the rewrite.</p> The tests passed. The plan didn't. 2026-07-12T20:15:00+00:00 TL;DR</strong> - RegreSQL 1.0 tested that your queries return the right rows. 2.0 tests that they return them the right way, and it does the checking against production's real statistics instead of your empty dev database, which lies.</em></p> A migration cleanup dropped an index nobody thought was that important. Every test passed: same rows, same order, green. Three days later the API started timing out on a query that hadn't changed a character, because the planner had quietly switched it from an index scan to a sequential scan over a table that had kept growing.</p> The first version of RegreSQL</a> would have passed that change too. It tests what your queries return</em>: run them, diff the rows against a committed expected file, go red when the output changes. That catches the query that now returns the wrong rows. It says nothing about the query that returns the right rows the wrong way, which is most of what takes a database down.</p> Version 2.0 tests that.</p> Test the plan, not just the rows</a> </h2> Here is that failure on a laptop. An orders</code> table, an index on customer_id</code>, and a query that reads one customer's orders:</p> -- orders-by-customer.sql</span></span> select</span> id, total</span></span> from</span> orders</span></span> where</span> customer_id </span>=</span> :cid</span></span> order by</span> id;</span></span></code></pre> Baseline it and run the tests. Green:</p> $ regresql test</span></span> ✓ 2 passing</span></span></code></pre> Now drop the index the way that migration did, and run the same tests again:</p> $ regresql test</span></span> ✓ 1 passing</span></span> ✗ 1 failing</span></span> </span> FAILING:</span></span> orders-by-customer.1.buffers (3898 > 109 102%, +3476.1%)</span></span> Expected buffers: 109</span></span> Actual buffers: 3898 (+3476.1%)</span></span> </span> Cost (info): 7962.41 (baseline: 421.03)</span></span> </span> ⚠️ Table 'orders': Bitmap Heap Scan → Seq Scan</span></span></code></pre> Here is the whole loop, start to failure, as it actually runs:</p> </a></p> The output check still passes; the rows didn't change. The plan check catches what the output check can't: the same query, returning the same result, now runs a sequential scan instead of the bitmap index scan it used before, reading thirty-five times the buffers. That failure blocks the merge. And the diff names the table and the exact change, in the same shape as the row diffs you already read in code review. Plan drift stops being something you find in production and becomes something you review in a pull request.</p> An empty database is a liar</a> </h2> That demo caught the regression because the table was big enough for the index to matter. On your dev database, it usually isn't. Run EXPLAIN</code> there and it says every query is fine, and it isn't lying on purpose: at a few hundred rows the cheapest plan really is a sequential scan, and it picks it. Production's planner reads different statistics and picks a different plan. A green suite on small data proves your queries are correct on small data, and nothing about the plan production will run. That gap, and why code review can't see it, is a story of its own.</p> You can't copy production data to your laptop. You don't need to. You need what the planner reads: the row counts, the histograms, the most-common-values. PostgreSQL 18 dumps and loads those on their own (pg_dump --statistics-only</code>), and RegreSQL 2.0 injects them so EXPLAIN</code>, against ten rows on your laptop, sees the real table's distribution:</p> regresql</span> test</span> --stats</span> production-stats.sql</span></span></code></pre> Now the plan you baseline is the plan production would pick, and a plan regression fails on your laptop before it reaches a server. That is the difference between testing your query and testing it against reality.</p> Gate on what happened, not what was guessed</a> </h2> Cost estimates are the planner's opinion. Opinions drift, and they drift most exactly when a plan is about to go wrong, because a bad plan is usually a bad estimate</em> first. Gating a test on estimated cost means trusting the number that breaks first to tell you when it broke.</p> So 2.0 gates on measured behavior instead. regresql baseline --analyze</code> runs the query for real and records what happened: buffers actually read, whether a sort or hash spilled to disk, how many rows actually flowed through each node, and the cardinality error, the ratio between what the planner predicted and what it got (the "q-error"). A query whose estimates were off by 2x and are now off by 200x has a plan that is one data change away from collapse, and the row counts say so long before the cost does. The baseline stores actuals; the test compares actuals; the estimate is never on the witness stand.</p> Wall-clock time is the actual everyone wants and the one that lies most, because it moves with cache state, whatever else is on the box, and luck. 2.0 measures it and treats it as evidence, not a verdict: interleaved runs, a per-query median, and a noise threshold derived by permuting the samples, so a query counts as slower only when it beats its own measurement noise. Anything that can't clear the threshold goes in an unstable bucket instead of failing the build. Timing informs the report, but only the measured actuals (buffers, spills, cardinality error) can fail the build.</p> Knowing when you can't tell</a> </h2> Here is the part that took the longest to get right, and the part I am most sure of.</p> Point RegreSQL at the same corpus on two builds of PostgreSQL and it will show you every query that plans differently between them. Early on, one query looked like a clean regression: the newer build read nearly three times the buffers, deterministically, run after run. It had every mark of a real find.</p> It was not one. Both builds had ANALYZE</code>'d their own copy of the data, and the query sat on a knife-edge join-order decision that a few unlucky sampled rows tip one way or the other. Inject one build's exact statistics into the other and they plan identically. Re-ANALYZE</code> and the "regression" moves to the other build. It was sampling noise, and a tool with less doubt would have reported it as a regression.</p> So 2.0 will not make a cross-version claim on statistics it does not trust. It injects identical statistics into both builds, so any remaining difference is code and not luck, and it re-ANALYZE</code>s the baseline several times to throw out queries whose plan is a coin-flip to begin with. A difference is reported only when it survives both. On that investigation the honest output was zero regressions, and zero was the correct answer.</p> A tool that only ever reports "found something" loses credibility over time, so RegreSQL will also report "I can't tell you that from this data" when the evidence doesn't support a verdict - that's a valid result here, not an apology.</p> Testing the planner itself</a> </h2> Chase these problems far enough and you end up at PostgreSQL's own test suite. pg_regress</code>, the harness that has guarded the database for decades, checks one thing: does the query return the expected text. It says nothing about the plan, the cost, or the cardinality error, because it folds all of them into a single .out</code> text file that either matches or does not. It is correctness-only by construction, and nothing in the standard toolchain measures whether a planner change made real queries better or worse.</p> Everything above, separating results from plan-shape from measured actuals from cardinality error into layers that can each be judged on their own, is what you would need to answer that question. Which is why 2.0 also has the pieces that only make sense pointed at PostgreSQL rather than at your application:</p> Cross-version scoreboards</strong> with the trust filter above, so a planner patch can be measured against a corpus without drowning in ANALYZE</code> noise.</li> A metamorphic check</strong>: flip an optimization that is supposed to preserve results (eager aggregation, memoize, incremental sort) and confirm the rows do not move. When they move, you have found a wrong-results bug in the optimizer, with no second database to compare against.</li> Determinism admission</strong>, which keeps only the queries whose result is stable across different plans, so an ambiguous LIMIT</code> over ties never masquerades as a regression.</li> </ul> This is a longer road, and an open one. But the same machinery that tells you your application query got slower can tell a PostgreSQL developer whether their patch made a thousand real queries slower, and that is a thing the project cannot currently see per-change. RegreSQL started as a way to test the queries you write. It turns out the honest way to do that is most of the way to a harness for the planner underneath them.</p> Leave evidence</a> </h2> None of this is useful if it only runs on your laptop. regresql test</code> exits non-zero on a failure, emits JUnit, GitHub Actions, or JSON, and reads DATABASE_URL</code> so the same project runs against a CI database without editing committed config. A plan flip fails the build the way a wrong-rows change does, and the run leaves a machine-readable record of what was checked against which snapshot. The check that advised you in the editor is the same check that blocks the merge.</p> You can go further and name the tables where a sequential scan is never acceptable, and have that fail the build outright. But which tables those are, and which of these signals is a release blocker versus a shrug, is a judgment the planner doesn't have and never will. That part is a longer story, and a separate one.</p> What "verified" means</a> </h2> RegreSQL 2.0 is careful about the word. It does not promise your database is safe</em>. It verifies what it can measure and says so plainly when it can't, which on the day it mattered meant reporting nothing at all. A green RegreSQL run is not "nothing will go wrong." It is "these queries return what they returned, plan the way they planned, against the statistics that actually apply." That is a smaller promise than most tools make, and one you can keep.</p> It runs on your laptop, against a database you already have, no production access:</p> regresql</span> init postgres://localhost/yourdb</span></span></code></pre> From there the SQL passes through the harness the way your code already passes through the type checker. The getting-started guide</a> walks the whole loop, from zero to a plan check in CI, in about ten minutes.</p> Same rows, different SUM 2026-06-28T21:25:00+00:00 Everyone knows not to store money as a double precision</code>. One can hope. The rule is so well drilled that it has stopped being interesting, and it is also not where the trouble usually starts. The float is already in the schema before anyone weighs in on it: a measurement column someone later sums for a report, telemetry that drifts into a finance dashboard, a third-party feed ingested as double precision</code> because that is how it arrived.</p> Here is the part the rule does not warn you about. Take a table of five million floating-point readings, sum the column, and run it three times in a row. Nothing else touches the table. Same connection, same data, same statement.</p> SELECT</span> sum</span>(reading) </span>FROM</span> measurements;</span></span></code></pre> sum</span></span> --------------------</span></span> 2500519211.7874823</span></span> </span> sum</span></span> -------------------</span></span> 2500519211.787477</span></span> </span> sum</span></span> --------------------</span></span> 2500519211.7874575</span></span></code></pre> Three runs, three different totals. No UPDATE</code>, no concurrent writer, no random seed. The rows did not change between runs, and the query is the same character for character. Yet the answer is not.</p> This is not a Postgres bug, and it is not specific to Postgres. It is what happens when floating-point arithmetic meets parallel aggregation, and it has been generating "my dashboard total changed and I have no idea why" tickets for as long as databases have parallelized. The non-determinism does not wait for you to opt into bad practice; it shows up the moment a parallel plan runs over whatever floats you happen to have.</p> The schema</a> </h2> One column of double precision</code>, five million rows.</p> CREATE TABLE</span> measurements</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> reading </span>double precision NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> measurements (reading)</span></span> SELECT</span> random()</span> </span> 1000</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>5000000</span>);</span></span> </span> ANALYZE measurements;</span></span></code></pre> Five million rows is enough that Postgres parallelizes the sum on its own. No settings forced, defaults all the way:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT</span> sum</span>(reading) </span>FROM</span> measurements;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------</span></span> Finalize Aggregate</span></span> -> Gather</span></span> Workers Planned: 2</span></span> -> Partial Aggregate</span></span> -> Parallel Seq Scan on measurements</span></span></code></pre> Two workers each sum part of the table in parallel, and a final step folds the partial sums together. Everything that follows is in that split.</p> Floating point does not associate</a> </h2> On paper, (a + b) + c</code> and a + (b + c)</code> are the same number. Postgres is not doing arithmetic on paper. Floating-point addition is not associative: change the grouping and you can change the answer, because every intermediate result is rounded to fit 64 bits. Move the parentheses and you move which roundings happen.</p> The textbook demonstration takes three values and one subtraction:</p> SELECT</span> 1</span>.0e20::float8 </span>+</span> 1</span>.</span>0</span>::float8 </span>-</span> 1</span>.0e20::float8;</span></span></code></pre> ?column?</span></span> ----------</span></span> 0</span></span></code></pre> The answer should be 1</code>. Read the expression left to right and watch where it goes wrong:</p> 1.0e20 + 1.0</code> should be 100000000000000000001</code>. But a double only carries about 16 significant digits, and that number needs 21. The trailing 1</code> falls off the end, below the smallest digit the number can represent. So the addition rounds back to exactly 1.0e20</code>.</li> 1.0e20 - 1.0e20</code> is then exactly 0</code>.</li> </ul> The 1</code> never made it past the first step. It was swallowed by the addition, before the subtraction even ran. This is absorption</em>: a value too small relative to its neighbour to leave a mark. It is not a database phenomenon. The same expression evaluates to 0</code> in C, Python, JavaScript, and every other language that uses IEEE 754 doubles.</p> A double precision</code> number keeps 52 bits for the mantissa, the digits that carry precision. This is around 15 to 17 decimal digits, no more. When the running total becomes big and the next number is small, the small one does not fit into those bits, so the rounding throws its tail away. Sum five million numbers and the total is, for most of its life, much bigger than any single number you add to it.</div> So the sum of a column depends on the order</em> in which the rows are added. Not on which rows, on the order. That is the seam parallel query pries open.</p> Why parallelism surfaces it</a> </h2> The plan spells out exactly how. Parallel Seq Scan</code> does not split the table into fixed slices upfront. It hands every worker a shared pointer into the same block iterator, and each worker grabs the next block, processes it, and comes back for another. Each worker runs its own Partial Aggregate</code>, summing only the rows it happened to grab, and by default the leader process pitches in as a third one (parallel_leader_participation</code>). Gather</code> collects those partial sums in whatever order they arrive, and Finalize Aggregate</code> folds them into a single total.</p> Two things vary from run to run, and neither is under your control:</p> Which rows each process sees.</strong> Because the blocks are pulled from a shared iterator on demand, the split is a race. A worker that gets a few microseconds ahead, for any reason the OS scheduler decides, grabs the next block, so the same worker ends up with a different set of rows on every run. The division is never guaranteed to be the same twice.</li> The order the partial sums come back.</strong> Finalize Aggregate</code> folds the partials into its total in arrival order. With only two values this would not matter, since a + b</code> equals b + a</code>. But there are three of them, the two workers and the leader, and combining three floats is once again an associativity question: which two get added first decides the rounding.</li> </ol> Both feed directly into the one thing float addition is sensitive to: grouping. Different row split, different partial sums; different arrival order, different final grouping of the additions. The total lands a few bits away from where it landed last time. Across five million readings that shows up in the sixth decimal place, exactly where the three runs above disagreed.</p> sum(float8)</code> is marked parallel-safe, which is why the planner is free to do this. It is safe in the sense that it will not crash or corrupt anything. It is not safe in the sense people assume, that the answer is a function of the input.</p> This is not unique to SUM</code>.</strong> AVG</code>, STDDEV</code>, VARIANCE</code>, and any other floating-point aggregate that accumulates across rows inherit the same order sensitivity. SUM</code> is just the easiest to demonstrate and the one most likely to land in a number a human reads.</p> </div> It was never really parallelism</a> </h2> It is tempting to file this under "parallel query is flaky" and turn it off. That misses the point. Parallelism does not cause the non-determinism, it only exposes</em> an order-dependence that was there the whole time.</p> Turn parallelism off completely and the sum becomes stable:</p> SET</span> max_parallel_workers_per_gather </span>=</span> 0</span>;</span></span> </span> SELECT</span> sum</span>(reading) </span>FROM</span> measurements; </span>-- 2500519211.787516</span></span> SELECT</span> sum</span>(reading) </span>FROM</span> measurements; </span>-- 2500519211.787516</span></span> SELECT</span> sum</span>(reading) </span>FROM</span> measurements; </span>-- 2500519211.787516</span></span></code></pre> Stable, but only because, with nothing else scanning the table, a serial sequential scan visits the heap in the same physical order every time, so the additions group the same way every time. The order is fixed, so the rounding is fixed.</p> "Same order every time" holds because the scan starts at block zero and walks forward. Postgres has one wrinkle that can break even that: synchronize_seqscan</code> (on by default). When a scan of a table larger than shared_buffers / 4</code> is already in progress, a second scan starts at its</em> current block to share cached pages, then wraps around at the end. So a concurrent reader can hand a serial scan a different starting point, and the float sum moves with it, no writes and no parallelism required. It is the same order-dependence, entering through a different door.</div> Change the order and the serial sum moves too. We can force different orders without any parallelism at all, using an ordered-set aggregate: SET</span> max_parallel_workers_per_gather </span>=</span> 0</span>; </span>-- serial, so ONLY order varies</span></span> </span> SELECT</span> sum</span>(reading </span>ORDER BY</span> id) </span>FROM</span> measurements; </span>-- 2500519211.787516</span></span> SELECT</span> sum</span>(reading </span>ORDER BY</span> reading) </span>FROM</span> measurements; </span>-- 2500519211.7876067</span></span> SELECT</span> sum</span>(reading </span>ORDER BY</span> reading </span>DESC</span>) </span>FROM</span> measurements; </span>-- 2500519211.7875423</span></span></code></pre> Same rows, same single thread, three summation orders, three answers. Parallelism is one way to shuffle the order, but a plain ORDER BY</code> inside the aggregate does it just as well. The dependence is on order, full stop.</p> Deterministic is not the same as correct</a> </h2> There is a subtler trap hiding in the "turn it off and it is stable" result. A stable answer is not necessarily the right answer. To see the true total, sum the same values as numeric</code>, which adds in exact decimal with no rounding until the very end:</p> SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements;</span></span></code></pre> sum</span></span> -------------------------------</span></span> 2500519211.787503050078417764</span></span></code></pre> That is near enough the exact sum of the stored values. Now line everything up:</p> How the sum was computed</th> Result</th> Off by</th></tr></thead> Exact (numeric</code>)</td> 2500519211.787503050…</code></td> reference</td></tr> Serial float8</code>, physical order</td> 2500519211.787516</code></td> ~1.3e-5</td></tr> Parallel float8</code>, run A</td> 2500519211.7874823</code></td> ~2.1e-5</td></tr> Parallel float8</code>, run B</td> 2500519211.787477</code></td> ~2.6e-5</td></tr> Ordered by value ascending</td> 2500519211.7876067</code></td> ~1.0e-4</td></tr> </tbody></table> Every floating-point total is wrong, including the "stable" serial one. They differ from the exact value in the fifth and sixth decimal places. Parallelism did not introduce the error - the serial sum was already wrong. What parallelism removed was the serial sum's one comforting property: getting the same wrong answer every time</em>, which people often mistake for correctness.</p> Summing smallest-to-largest is the classic textbook advice for reducing floating-point error, and it usually does help, but here ascending order happens to land further from the exact value than physical order does. Order changes the error; it does not reliably shrink it. If you need a smaller error there are compensated-summation algorithms (Kahan and friends), but inside a SQL aggregate you do not get to choose them.</div> Where it actually bites</a> </h2> In the sixth decimal of a five-million-row random sum, nobody cares. The problem is the places where someone does.</p> The obvious one is reconciliation. A finance team sums a revenue column stored as double precision</code>, exports it, and the number disagrees with the same query run an hour later, or with the same logic in a different tool running a different parallel degree. Nothing in the data changed, so the hunt goes to caching, to replicas, to "a dirty read somewhere" long before anyone suspects the aggregate itself.</p> The same thing wrecks regression tests, just less visibly. Assert that a float total equals an exact literal, or compare two pipelines for byte-identical sums, and the test will flap. It passes on a small fixture that runs serially, then fails in CI on a larger one that crosses the parallel threshold, with no code change in between. The test is asserting a property the type does not have, and it will keep flapping until someone notices that.</p> And anywhere a float sum feeds an equality check, a join, or a GROUP BY</code> key, you are building on sand. Two totals that "should" be equal can differ in their last bits, and now your join silently drops rows.</p> Postgres already knows this is dangerous</a> </h2> The most convincing evidence that this is a real, known hazard and not a corner case is that Postgres deliberately gives up an optimization because of it.</p> Window functions can compute a moving sum efficiently using an inverse transition function</em>: as the frame slides forward, subtract the value leaving the window instead of re-summing the whole frame. It works for numeric</code> and integers. For float4</code> and float8</code>, Postgres refuses to do it. From the documentation on user-defined aggregates</a>:</p> An example of an aggregate for which adding an inverse transition function seems easy at first, yet where this requirement cannot be met is sum</code> over float4</code> or float8</code> inputs.</p> </blockquote> The docs then show why, with the same absorption we saw earlier. If a window holds 1e20</code> and 1</code>, subtracting 1e20</code> as it leaves the frame yields 0</code>, not the 1</code> that should remain. So the built-in float sum simply does not use the inverse optimization, and a moving sum(float8)</code> recomputes the frame to stay correct:</p> SELECT</span> n, </span>sum</span>(x) </span>OVER</span> (</span>ORDER BY</span> n </span>ROWS BETWEEN</span> CURRENT </span>ROW AND</span> 1</span> FOLLOWING</span>)</span></span> FROM</span> (</span>VALUES</span> (</span>1</span>, </span>1</span>.0e20::float8), (</span>2</span>, </span>1</span>.</span>0</span>::float8)) </span>AS</span> v(n, x);</span></span></code></pre> n \| sum</span></span> ---+-------</span></span> 1 \| 1e+20</span></span> 2 \| 1</span></span></code></pre> The second row is 1</code>, correct, because Postgres declined the shortcut that would have made it 0</code>. The engine is quietly working around the imprecision of float sums on your behalf. The non-determinism in the parallel case is the same hazard showing up somewhere the engine cannot paper over it.</p> What to do about it</a> </h2> The fix is almost always the type, not the query.</p> Use numeric</code> for anything anyone reconciles, compares, or reads as a figure.</strong> Money, totals on reports, anything that has to match another system or itself across runs. numeric</code> sums in exact decimal and is order-independent: same rows, same answer, parallel or not, in any order.</p> SET</span> max_parallel_workers_per_gather </span>=</span> 4</span>;</span></span> SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements; </span>-- 2500519211.787503050078417764</span></span> SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements; </span>-- 2500519211.787503050078417764</span></span></code></pre> The cast above stabilizes a single query, but the durable fix is storing the column as numeric</code> in the first place, so the exact values are what live on disk.</p> numeric</code> is not free. It is arbitrary-precision arithmetic done in software, so it is slower to compute and takes more space on disk than a hardware float the CPU adds in a single instruction. For anything that has to reconcile, that is the price of correctness, and it is almost always worth paying. For a column nobody ever sums into a figure a person reads, it usually is not.</p> If the data is genuinely floating point</strong> (sensor readings, scientific measurements, anything where the input is already an approximation), then accept that aggregates over it are approximate and do not assert exact equality on them. Round for display, but compare against a tolerance rather than by rounding, since two totals a hair apart round to different neighbours as readily as to the same one, and never use a raw float sum as a join or grouping key.</p> Pinning max_parallel_workers_per_gather = 0</code> makes a float sum repeatable, but does not make it correct,</strong> and it costs you parallelism on every other query in the session. The result is still wrong, just consistently wrong. Reach for the type instead.</p> None of this is Postgres-specific. Any database that parallelizes aggregation, and they all do, has the same behavior over the same IEEE 754 floats. The trigger that makes it visible is simply crossing whatever threshold turns parallelism on, which is why it so often shows up when a table grows past a certain size and "nothing else changed."</p> PS: Thanks to Peter J. Holzer for spotting that the cast to numeric</code> rounds on the way in, and that rounding is the wrong way to compare floats.</p> The NULL in your NOT IN 2026-06-14T23:30:00+00:00 A NOT IN</code> query can return the wrong answer without telling you. It is valid SQL, it runs without an error, and it hands back a perfectly well-formed result set that happens to be empty when it should not be. No warning, no hint, nothing in the logs: just zero rows where you expected hundreds, and a database that considers it correct.</p> Almost always the cause is a single NULL</code> sitting somewhere you forgot to look, combined with two keywords you have typed a thousand times: NOT IN</code>. None of it is a Postgres bug. This is exactly what the SQL standard mandates, implemented faithfully. That is precisely what makes it so easy to walk into, and why the planner could not safely optimize around it for the better part of Postgres's history. It comes down to one if</code> statement in the parser.</p> Sample schema</a> </h2> Nothing elaborate. A table of products, one of which has no category assigned yet, and a table of archived categories that happens to contain a NULL</code>:</p> CREATE TABLE</span> products</span> (id </span>int</span>, category_id </span>int</span>);</span></span> INSERT INTO</span> products </span>VALUES</span> (</span>1</span>, </span>10</span>), (</span>2</span>, </span>20</span>), (</span>3</span>, </span>NULL</span>), (</span>4</span>, </span>10</span>);</span></span> </span> CREATE TABLE</span> archived</span> (category_id </span>int</span>);</span></span> INSERT INTO</span> archived </span>VALUES</span> (</span>20</span>), (</span>NULL</span>);</span></span></code></pre> The NULL</code> in archived</code> is not contrived. The moment a column is nullable (and most are, by default), a NULL</code> can find its way into any subquery you point a NOT IN</code> at. That is the whole point: this is not an exotic data condition, it is the ordinary one.</p> The query that returns nothing</a> </h2> Here is the request you have written a hundred times: give me the products whose category is not</em> archived.</p> SELECT</span> id, category_id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived);</span></span></code></pre> You expect products 1 and 4 (category 10, which is not in the archived set). What comes back is:</p> id \| category_id</span></span> ----+-------------</span></span> (0 rows)</span></span></code></pre> Every row gone. Not a subset, not an off-by-one: all of them. Drop the NULL</code> from archived</code> and the same query behaves:</p> SELECT</span> id, category_id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived</span></span> WHERE</span> category_id </span>IS NOT NULL</span>);</span></span></code></pre> id \| category_id</span></span> ----+-------------</span></span> 1 \| 10</span></span> 4 \| 10</span></span> (2 rows)</span></span></code></pre> To understand why a single NULL</code> empties the entire result, we have to stop thinking of NOT IN</code> as a single thing and watch the parser take it apart.</p> IN is an OR, NOT IN is an AND</a> </h2> IN</code> is not a primitive operator. It is shorthand that the parser rewrites into a chain of equality comparisons joined by OR</code>:</p> x </span>IN</span> (a, b, c)</span></span> -- becomes</span></span> x </span>=</span> a </span>OR</span> x </span>=</span> b </span>OR</span> x </span>=</span> c</span></span></code></pre> NOT IN</code> is the logical negation of that, and by De Morgan's law negating an OR</code> of equalities gives you an AND</code> of inequalities:</p> x </span>NOT IN</span> (a, b, c)</span></span> -- becomes</span></span> x </span><></span> a </span>AND</span> x </span><></span> b </span>AND</span> x </span><></span> c</span></span></code></pre> This is not an analogy. It is literally the expression Postgres builds, and you can read it straight off an EXPLAIN</code>. The literal-list forms collapse into array operators whose names give the whole game away:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT FROM</span> products </span>WHERE</span> category_id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span> -- Filter: (category_id = ANY ('{1,2,3}'::integer[]))</span></span> </span> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> products </span>WHERE</span> category_id </span>NOT IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span> -- Filter: (category_id <> ALL ('{1,2,3}'::integer[]))</span></span></code></pre> IN</code> is = ANY</code>: equal to any</em> element, an OR</code>. NOT IN</code> is <> ALL</code>: different from all</em> elements, an AND</code>.</p> The actual node types matter here, because they are what you end up staring at when you dump a parse tree or read a normalized pg_stat_statements</code></a> entry. A literal list compiles to a single ScalarArrayOpExpr</code>: the scalar on the left, the array on the right, and a useOr</code> flag that is the entire difference between = ANY</code> and <> ALL</code>. The subquery forms are a different node altogether, a SubLink</code>. Recognising those two names on sight tells you immediately which path the planner is on.</p> If "IN</code> and = ANY</code> are the same operator" is news: they compile to the same parse node and the same plan, with the spellings diverging only in plan-cache churn and selectivity estimates. The NOT IN</code> case in front of you here is the one corner where the choice is not cosmetic but a matter of correctness.</div> Three-valued logic does the rest</a> </h2> SQL does not have two truth values, it has three: true</strong>, false</strong>, and unknown</strong>. Any comparison against NULL</code> yields unknown</code>, because NULL</code> means "no value here" and you cannot ask whether an absent value is different from 20:</p> -- not false. unknown (displayed as a blank)</span></span> SELECT</span> 10</span> <> NULL</span>;</span></span></code></pre> Now walk the NOT IN</code> expansion for product 1 (category 10) against the archived set of 20 and NULL</code>:</p> 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a> </div> true AND unknown</code> is unknown</code>, not true</code>. A WHERE</code> clause keeps a row only when its predicate evaluates to true</strong>. Both false</code> and unknown</code> cause the row to be discarded. So product 1 is dropped. Run the same arithmetic for product 4 and you land on unknown</code> again.</p> The mechanism in one sentence:</strong> the instant a single NULL</code> enters the right-hand side, the trailing AND unknown</code> term can never be true</code>, so the whole NOT IN</code> can never be true</code>, so every</strong> row is discarded, regardless of how many million rows you have or what they contain.</p> </div> NULLs on the left side too</a> </h2> Keeping NULL</code>s out of the subquery is not enough. The same unknown</code> arises from NULL</code>s on the left</em>: product 3 (whose category_id</code> is NULL</code>) evaluates to unknown AND unknown</code>, so it is dropped even against a spotless right-hand set. IN</code> and NOT IN</code> are not complements: a row can fail both tests simultaneously. There is a NULL</code>-shaped gap between them that belongs to neither.</p> The seam, in the source</a> </h2> All of this reduces to one branch in one function. Open src/backend/parser/parse_expr.c</code> and find transformAExprIn</code>, the routine that turns both IN</code> and NOT IN</code> list expressions into something the planner can chew on. The very first thing it decides is whether it is building an OR</code> or an AND</code>:</p> /</span></span> If the operator is <>, combine with AND not OR.</span></span> /</span></span> if</span> (</span>strcmp</span>(</span>strVal</span>(</span>linitial</span>(a</span>-></span>name</span>)),</span> "<>"</span>)</span> ==</span> 0</span>)</span></span> useOr </span>=</span> false</span>;</span></span> else</span></span> useOr </span>=</span> true</span>;</span></span></code></pre> That is the entire fork. IN</code> arrives carrying the operator =</code> and gets useOr = true</code>; NOT IN</code> arrives carrying <></code> and gets useOr = false</code>. The flag rides all the way down to where the boolean tree is finally assembled, several hundred lines later:</p> result </span>=</span> (Node </span></span>)</span> makeBoolExpr</span>(useOr </span>?</span> OR_EXPR </span>:</span> AND_EXPR,</span></span> list_make2</span>(result, cmp),</span></span> a</span>-></span>location</span>);</span></span></code></pre> OR_EXPR</code> for IN</code>, AND_EXPR</code> for NOT IN</code>. There is no special-casing of NULL</code> anywhere in this function, and there does not need to be: the three-valued behavior is an emergent property of having chosen AND</code>. The parser does the obvious, correct thing, and the NULL</code> semantics fall straight out of standard boolean logic. The "bug", if you insist on the word, belongs to the SQL standard, which Postgres implements faithfully.</p> A grammar asymmetry: list vs. subquery</a> </h3> A list and a subquery are built differently. The list form is the <> ALL</code> chain of inequalities you just saw. The subquery form is not a <> ALL</code> at all: it becomes NOT (foo = ANY (subquery))</code>. Different shapes, same truth table, a NULL</code> in the comparison makes the result unknown</code>, and unknown</code> loses. That NOT (... = ANY ...)</code> shape is the one the planner sees.</p> Why the planner won't save you</a> </h2> The correctness problem has a plan-shape twin. Watch what the planner does with three sibling queries on a larger schema of 200,000 orders</code> and 1,000 vip</code> rows.</p> First, the positive case, IN</code> against a subquery:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span> WHERE</span> customer_id </span>IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre> Hash Semi Join</span></span> Hash Cond: (orders.customer_id = vip.customer_id)</span></span> -> Seq Scan on orders</span></span> -> Hash</span></span> -> Seq Scan on vip</span></span></code></pre> A clean semi-join</strong>. The planner promotes the subquery to a first-class relation, picks a hash join, and is free to reorder it against the rest of the query. Now the natural mirror, orders whose customer is not</em> a VIP, written first with NOT EXISTS</code>:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders o</span></span> WHERE NOT EXISTS</span> (</span>SELECT</span> 1</span> FROM</span> vip v </span>WHERE</span> v</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span>);</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------</span></span> Hash Anti Join</span></span> Hash Cond: (o.customer_id = v.customer_id)</span></span> -> Seq Scan on orders o</span></span> -> Hash</span></span> -> Seq Scan on vip v</span></span></code></pre> A hash anti-join</strong>, the efficient and symmetric counterpart to the semi-join. Now the same intent expressed with NOT IN</code>:</p> EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span> WHERE</span> customer_id </span>NOT IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------</span></span> Seq Scan on orders</span></span> Filter: (NOT (ANY (customer_id = (hashed SubPlan 1).col1)))</span></span> SubPlan 1</span></span> -> Seq Scan on vip</span></span> (4 rows)</span></span></code></pre> No join at all. The subquery collapses into an opaque SubPlan</code> filter bolted onto a sequential scan, the NOT (... = ANY ...)</code> shape from the grammar is right there in the filter.</p> The wording of that filter line is version-specific. PostgreSQL 16 prints the terser Filter: (NOT (hashed SubPlan 1))</code>; a later version changed EXPLAIN</code> to expose the inner comparison, which is the NOT (ANY (... = (hashed SubPlan 1).col1))</code> form shown here on 18.4. The plan underneath is the same opaque subplan in every released version.</div> The difference between those last two plans is structural, not a tuning detail. A Hash Anti Join</code> builds one hash table from the inner relation and streams the outer relation through it exactly once, and when the inner side outgrows work_mem</code> it partitions into batches and spills to disk without changing the result. A SubPlan</code> gives the planner none of that. Even the hashed</code> variant you see here is evaluated in place as a filter on the scan: it cannot be reordered during the global join search, it cannot have the outer query's join clauses pushed into it to drive an index scan, and it has no multi-batch spill logic to fall back on if the hashed set turns out larger than the estimate. The subquery stops being a relation the optimizer can plan and becomes an opaque function the optimizer has to call, once per outer row. This is the same optimization-fence effect as an uninlined CTE</a>: the planner simply cannot see inside. On small inputs nobody notices; on large ones it is the difference between milliseconds and minutes.</p> The reason the planner is stuck with that shape is precisely the NULL</code> semantics from earlier. An anti-join keeps a row when it finds no match</em>. But NOT IN</code> must discard</em> a row when the comparison goes unknown</code>, and "unknown" is not the same as "no match". Those two behaviors diverge exactly when a NULL</code> is in play, so the planner historically could not prove the rewrite safe and never attempted it.</p> Declaring the columns NOT NULL</code> does not rescue you on released Postgres.</strong> I constrained both the outer and inner columns NOT NULL</code> on 18.4 and re-ran the query, and the plan was still the opaque SubPlan</code> filter. The released planner does not even look. So NOT IN (subquery)</code> has been a correctness trap for as long as Postgres has existed, and a planner pessimization ever since anti-joins arrived in 8.4 (2009) and every other negated subquery learned to use them — neither of which the released optimizer rescues you from.</p> </div> The fix is landing in PostgreSQL 19</a> </h2> That last paragraph is finally going out of date. This is a performance fix, not a semantic one: it rescues the case that was already correct</em> but badly planned</em> (NOT NULL</code> columns stuck behind the opaque SubPlan</code> from the previous section) and leaves nullable columns exactly where the SQL standard puts them. Tracing this through a checkout of the development branch, I found a function that exists in no released Postgres: sublink_testexpr_is_not_nullable</code>, in src/backend/optimizer/plan/subselect.c</code>. It guards a brand-new branch inside convert_ANY_sublink_to_join</code>:</p> /</span></span> Per SQL spec, NOT IN is not ordinarily equivalent to an anti-join, so</span></span> * that by default we have to fail when under_not. However, if we can</span></span> * prove that neither the outer query's expressions nor the sub-select's</span></span> * output columns can be NULL, and further that the operator itself cannot</span></span> * return NULL for non-null inputs, then the logic is identical and it's</span></span> * safe to convert NOT IN to an anti-join.</span></span> /</span></span> if</span> (under_not </span>&&</span></span> (</span>!</span>sublink_testexpr_is_not_nullable</span>(root, sublink)</span> \|\|</span></span> !</span>query_outputs_are_not_nullable</span>(subselect)))</span></span> return</span> NULL</span>;</span></span></code></pre> git blame</code> dates it to commit 383eb21ebff</code>, "Convert NOT IN sublinks to anti-joins when safe"</em>, merged in March 2026 by Richard Guo. It lives on master</code>, bound for PostgreSQL 19</strong>. It appears in no REL_18</code> tag, which is exactly why my 18.4 box still produced the SubPlan</code> even with NOT NULL</code> columns. The commit message states the bargain plainly:</p> if we can prove that neither side of the comparison can yield NULL values, and further that the operator itself cannot return NULL for non-null inputs, the behavior of NOT IN and anti-join becomes identical.</p> </blockquote> The proof has to establish three things:</p> Both operands provably non-NULL</code>.</strong> Established from schema NOT NULL</code> constraints (via a NOT-NULL-attnums hash table), from outer-join nullability tracking (so a Var</code> from the nullable side of an outer join does not qualify), and from qual clauses that force a Var</code> non-null.</li> The subquery's output columns provably non-NULL</code></strong>, the query_outputs_are_not_nullable</code> half of the guard.</li> A NULL</code>-safe operator.</strong> The operator must belong to a B-tree or Hash operator family. That is a proxy for "behaves like a normal boolean comparison and won't return NULL</code> on non-null inputs", because an operator that did</em> return NULL</code> there would break the very index it claims to support.</li> </ul> When all three hold, the NULL</code>-handling mismatch evaporates and NOT IN</code> is finally allowed to become a JOIN_ANTI</code>:</p> result</span>-></span>jointype </span>=</span> under_not </span>?</span> JOIN_ANTI </span>:</span> JOIN_SEMI;</span></span></code></pre> It lifts a planner limitation that has stood since anti-joins themselves arrived in 8.4, but only when the planner can prove</em> no NULL</code> can reach the comparison. If your column is nullable, you are exactly where you have always been, in PostgreSQL 19 as in 9.x. The semantics never changed; the optimizer merely learned to recognize the cases where NULL</code> is provably absent.</p> Decision matrix</a> </h2> You wrote</th> Internal shape</th> NULL on right →</th> NULL on left →</th> Planner (≤ PG 18)</th></tr></thead> IN (1,2,3)</code></td> = ANY</code>, an OR</code></td> absorbed, no harm</td> row dropped (no match)</td> ScalarArrayOpExpr</code></td></tr> NOT IN (1,2,3)</code></td> <> ALL</code>, an AND</code></td> all rows dropped</strong></td> row dropped</strong></td> ScalarArrayOpExpr</code></td></tr> IN (subquery)</code></td> = ANY</code> sublink</td> absorbed</td> row dropped</td> Semi Join</strong></td></tr> NOT IN (subquery)</code></td> NOT (= ANY)</code> sublink</td> all rows dropped</strong></td> row dropped</strong></td> opaque SubPlan</code> filter ¹</td></tr> NOT EXISTS (...)</code></td> anti-join sublink</td> row kept</td> row kept</td> Anti Join</strong></td></tr> </tbody></table> ¹ PostgreSQL 19 promotes this to an Anti Join when both the outer expression and the subquery output column are provably NOT NULL</code>.</p> What to do instead</a> </h2> Don't wait for PostgreSQL 19, and don't lean on it once it ships: it only fires on provably NOT NULL</code> columns. Every portable fix comes down to one rule: keep NULL</code> away from NOT IN</code>. In practice that means changing the query, and there are three moves, in order of preference.</p> Default to NOT EXISTS</code>.</strong> Make this the habit and you never hit the trap again. Same anti-join semantics and the same plan as the working NOT IN</code> would have wanted, on every version, nullable columns or not:</p> SELECT</span> id, category_id </span>FROM</span> products p</span></span> WHERE NOT EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> archived a </span>WHERE</span> a</span>.</span>category_id</span> =</span> p</span>.</span>category_id</span></span> );</span></span></code></pre> It keeps product 3, the NULL</code> category, because NOT EXISTS</code> asks whether a matching archived row exists, not whether category_id = NULL</code> is true. That is usually the answer you wanted. If you do want the NULL</code>-category rows gone, add AND p.category_id IS NOT NULL</code> and you have decided it on purpose.</p> Filter the NULL</code>s out of the subquery</strong> when you have to keep the NOT IN</code> (a generated query you can only edit inside the parentheses, say):</p> SELECT</span> id </span>FROM</span> products</span></span> WHERE</span> category_id </span>NOT IN</span> (</span></span> SELECT</span> category_id </span>FROM</span> archived </span>WHERE</span> category_id </span>IS NOT NULL</span></span> );</span></span></code></pre> This covers the right-hand NULL</code> and nothing else: product 3 still disappears, because the left-hand NULL</code> is untouched. Use it only when you cannot reach for NOT EXISTS</code>.</p> Use EXCEPT</code> for whole-set comparisons.</strong> It matches rows with IS NOT DISTINCT FROM</code>, so two NULL</code>s count as equal and the three-valued trap never fires. But that same rule means a NULL</code> in archived</code> removes the NULL</code> rows from products</code>:</p> SELECT</span> category_id </span>FROM</span> products</span></span> EXCEPT</span></span> SELECT</span> category_id </span>FROM</span> archived;</span></span> -- returns: {10}</span></span> -- product 3 (NULL category) is dropped because archived also has a NULL</span></span></code></pre> So NOT EXISTS</code> keeps product 3 and EXCEPT</code> drops it. Pick the one whose answer matches what you're after.</p> EDIT (2026-06-15): I originally said this stood for about 25 years, which mixed up two things. The correctness trap is as old as NOT IN</code>; the planner pessimization only goes back to 8.4 (2009), when anti-joins arrived. Reworded to keep the two separate.</em></p> Welcome to ORDER BY jungle 2026-05-15T11:43:00+02:00 SQL is fun and not at all boring. The latest article by Markus Winand on Order by Has Come a Long Way</a> sent me on quite a journey.</p> First, set up a table called nums</code> with one integer column and four rows:</p> CREATE TABLE</span> nums</span> (a </span>int</span>);</span></span> INSERT INTO</span> nums </span>VALUES</span> (</span>0</span>), (</span>1</span>), (</span>2</span>), (</span>3</span>);</span></span></code></pre> Try to guess what these two queries return.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> Most of us would guess the same rows in a different order. The actual answer is that they produce exactly the same rows in exactly the same order</strong>. By the same logic you might expect</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>c;</span></span></code></pre> to do exactly the same. Except it does not. It errors with column "c" does not exist</code> despite the alias being right there in the statement. Welcome to ORDER BY jungle.</p> Names and expressions are not the same</a> </h2> If you ask most developers how ORDER BY</code> works, they will say "you put a column name there and it sorts the rows". In 99% of queries that is exactly what happens. People sort by created_at</code> or id</code> and move on.</p> Strictly speaking, three, if you count ORDER BY 1</code>. Positional references are their own can of worms and out of scope for this post.</div>But `ORDER BY` accepts two different kinds of things: SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY</span> created_at;</span></span> SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY date</span>(created_at);</span></span></code></pre> Both feel natural. And the thing nobody tells you is that they go down completely different code paths in the parser. Different scope rules, different lookups, different error messages. The first looks at your SELECT</code> list. The second looks at your FROM</code> clause. They never look at the same place.</p> Same answer, two different sorts</a> </h2> Look at the first query again.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span></code></pre> You wrote ORDER BY a</code>. A bare identifier, no decoration. Postgres goes down the name</em> path. It scans the SELECT</code> list for something called a</code>, finds the aliased column -a AS a</code>, and sorts by its output values. The negated values are -3, -2, -1, 0</code>, ascending is -3, -2, -1, 0</code>. That is what comes out.</p> Now the twin.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> You wrote ORDER BY -a</code>. This is no longer an identifier. It's an expression: unary minus around a column reference. The parser does not even try the same logic.</p> Instead it switches to the expression path</strong>, where the only a</code> it knows is the column in nums</code>, and sorts the input values negated. And by arithmetic luck, the two queries land on the same row order. Same output, completely different logic. If you don't believe it is just luck, drop the negation from the SELECT</code> list and keep it in ORDER BY</code>:</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> c</span></span> ---</span></span> 3</span></span> 2</span></span> 1</span></span> 0</span></span> (4 rows)</span></span></code></pre> ORDER BY -a</code> is an expression, so it sorts by -input_a</code> ascending, which is input_a</code> descending. The alias c</code> was never consulted. The result has nothing to do with whatever c</code> happens to be.</p> And ORDER BY -c</code> is now obvious. -c</code> is an expression, so the parser looks for column c</code> in FROM</code>, doesn't find it, and errors. The alias exists, but in a scope this code path cannot see.</p> Above the identifier, or around it</a> </h2> Once the rule is clear (bare identifier hits the SELECT</code> list, anything else hits the table) the rest of the surprises fall out.</p> SELECT</span> 'hello'</span> AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x::</span>text</span>;</span></span> -- ERROR: column "x" does not exist</span></span></code></pre> It is probably not surprising that casts</strong> count as expressions and push the lookup to the table.</p> The surprise might come with</p> SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY</span> c </span>DESC NULLS FIRST</span>;</span></span></code></pre> Which will work as expected. Both DESC</code> and NULLS FIRST</code> are part of the sort clause itself, not of the sort expression. They sit above the identifier in the parse tree, so they never touch it. The parser still sees a bare c</code>, takes the fast path, finds the alias, sorts by it, and then applies "descending, nulls first" on top of the resolved key.</p> The same cannot be said about collation</strong>.</p> SELECT</span> 'A'</span>::</span>text AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x </span>COLLATE</span> "C"</span>;</span></span> -- ERROR: column "x" does not exist</span></span></code></pre> This is a really bad one. COLLATE</code> might look the same as a sort modifier, but it is not. It wraps the expression in the parse tree.</p> Parentheses are a special case.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> (a);</span></span> -- works, sorts by alias</span></span></code></pre> Postgres collapses redundant parens before the bare-identifier check, so (a)</code> is still bare a</code>. The seam is asymmetric in the way that maximises confusion: COLLATE</code> is "still a name to a human, an expression to the parser", and (a)</code> is "an expression to a human, still a name to the parser". You get both flavours of wrong intuition mixed here.</p> Unary plus.</strong> +a</code> and a</code> evaluate to the same value, but they do not parse to the same node.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY +</span>a;</span></span></code></pre> A plus sign you would not even think about changes which rows come out in which order.</p> The parser stores a column reference as a list of name parts: one part when it is unqualified, two or more once you add a table or schema. The fast path only fires on lists of length one.</div> Finally, schema- and table-qualified references</strong>. ORDER BY nums.a</code> looks like an identifier, but it is not.</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> nums</span>.</span>a</span>; </span></span></code></pre> a</span></span> ----</span></span> -3</span></span> -2</span></span> -1</span></span> 0</span></span> (4 rows)</span></span> </span> a</span></span> ----</span></span> 0</span></span> -1</span></span> -2</span></span> -3</span></span> (4 rows)</span></span></code></pre>Aliases that aren't the names you think</a> </h2> Here is one that cost me an afternoon once. Easy to come across once an ORM or a generated view declared the alias for you. SQLAlchemy, Hibernate, jOOQ, and most code generators quote anything that isn't pure lowercase. Two queries, identical except that the alias is quoted in one. Two different result sets.</p> SELECT -</span>a </span>AS</span> A </span>FROM</span> nums </span>ORDER BY</span> a; </span>-- sorts by alias (-3,-2,-1,0)</span></span> SELECT -</span>a </span>AS</span> "A"</span> FROM</span> nums </span>ORDER BY</span> a; </span>-- sorts by input (0,-1,-2,-3)</span></span></code></pre> The bare-identifier check compares names with strcmp</code>. Unquoted A</code> folds to lowercase a</code> and matches. Quoted "A"</code> preserves case, stays A</code>, and does not match the lowercase a</code> in the ORDER BY</code>. The lookup fails, the parser falls through to the expression path, the expression path finds the column a</code> in nums</code>, and the query runs successfully while doing something different from what you meant.</p> GROUP BY checks the opposite scope first</a> </h2> Both GROUP BY</code> and ORDER BY</code> accept a bare identifier, and both can resolve it either way: to a table column or to a SELECT</code>-list alias. The difference is the order they check:</p> ORDER BY a</code> looks at the SELECT</code> list first, then the table.</li> GROUP BY a</code> looks at the table first, then the SELECT</code> list.</li> </ul> For most queries this never matters. The two clauses end up picking the same thing because nothing is shadowed. The surprise happens when an alias has the same name as a base column but a different value:</p> SELECT</span> a</span>/</span>2</span> AS</span> a, </span>count</span>(</span></span>)</span></span> FROM</span> nums</span></span> GROUP BY</span> a</span></span> ORDER BY</span> a;</span></span></code></pre> Now the two clauses disagree about what a</code> means. GROUP BY a</code> picks the input column (four distinct values, four groups, one row each). ORDER BY a</code> picks the alias, which is a/2</code>. The result has four rows because the grouping was on a finer-grained key than the projection:</p> a \| count</span></span> ---+-------</span></span> 0 \| 1</span></span> 0 \| 1</span></span> 1 \| 1</span></span> 1 \| 1</span></span></code></pre> Two rows where a/2 = 0</code> (from input 0</code> and 1</code>), two where a/2 = 1</code> (from input 2</code> and 3</code>). The duplicates are real. The same identifier means two different columns in two adjacent clauses of one query.</p> Window ORDER BY does not even pretend</a> </h2> This one trips people up because it does not look like a different clause.</p> SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY</span> neg) </span>FROM</span> nums;</span></span> -- ERROR: column "neg" does not exist</span></span></code></pre> OVER (ORDER BY ...)</code> is a different parse path entirely. It does not check the targetlist at all, only the FROM scope. The bare-name fast path simply does not exist here.</p> SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY -</span>a) </span>FROM</span> nums;</span></span> -- this works</span></span></code></pre> Two ORDER BY</code> clauses in the same query, two different scoping rules.</p> UNION ORDER BY is name-only</a> </h2> When ORDER BY</code> follows a UNION</code>, neither path is fully open.</p> -- ok</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a;</span></span> -- ERROR</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY -</span>a;</span></span> -- ERROR</span></span> (</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre> The error message is unusually helpful:</p> Only result column names can be used, not expressions or functions. HINT: Add the expression/function to every SELECT, or move the UNION into a FROM clause.</p> </blockquote> Set operations do not have a single FROM scope to fall back to, so the expression path is closed entirely. Bare names or nothing.</p> The seam, in the source</a> </h2> Full disclosure: I got this section wrong three times before Claude Code helped me trace the actual parse tree. Lack of sleep from a whole night of geeking out over ORDER BY</code> is the other plausible explanation.</div>Open `src/backend/parser/parse_clause.c` and find `findTargetlistEntrySQL92`. It is forty lines of comment, two `if` blocks, and a final `return`. SQL92's two resolution rules are tried first; SQL99 is the fallback. Block one: the bare-name path.</strong> The gate is a ColumnRef</code> node with exactly one name part, and that part must be a string identifier (not </code>, which is also a ColumnRef</code> but with an A_Star</code> field). If the node passes, the function walks the target list looking for a non-resjunk</code> entry whose resname</code> equals the identifier. The loop keeps going past the first match to detect ambiguity: identical expressions are fine (this is why SELECT a, a FROM nums ORDER BY a</code> works), different expressions error out. On a unique match, return.</p> If the loop finds nothing, the block does not</strong> return. Control falls through. This is the case behind the quoted-alias surprise earlier in the post: AS "A"</code> stores resname = "A"</code>, ORDER BY a</code> looks up resname = "a"</code>, the strcmp</code> fails, and the function moves on as if no SQL92 fast path applied.</p> GROUP BY</code> is the small exception inside this block. The name is first tested against the FROM</code> scope, and a hit there causes the targetlist loop to be skipped. That is how GROUP BY</code> ends up preferring the input column.</p> Block two: the positional path.</strong> The gate is IsA(node, A_Const)</code>. A non-integer constant errors immediately ("non-integer constant in ..."), which catches ORDER BY NULL</code>, ORDER BY 'a'</code>, ORDER BY TRUE</code>. An integer is used as a 1-based position into the non-resjunk</code> target list; anything outside the range errors as "position %d is not in select list". Block two never falls through.</p> Both 1</code> and -1</code> arrive here as integer A_Const</code>s. doNegate</code> in the grammar folds '-' Iconst</code> into a single integer constant before the function ever runs, so ORDER BY 1</code> and ORDER BY -1</code> go through the same code, with only the integer value (and the result of the position lookup) differing.</p> The fallthrough.</strong> Anything not caught above reaches the last line:</p> /</span></span> * Otherwise, we have an expression, so process it per SQL99 rules.</span></span> /</span></span> return</span> findTargetlistEntrySQL99</span>(pstate, node, tlist, exprKind);</span></span></code></pre> That is the seam. SQL92 succeeds in two narrow shapes: a bare identifier with a matching alias, or an in-range positive integer. Everything else, including a bare identifier whose alias lookup found nothing, becomes a SQL99 expression resolved against FROM</code>.</p> A useful workaround</a> </h2> If you want the alias inside an expression in ORDER BY</code>, the portable trick is to wrap the query in a subselect:</p> SELECT </span></span> FROM</span> (</span>SELECT -</span>a </span>AS</span> x </span>FROM</span> nums) s</span></span> ORDER BY</span> x </span>+</span> 0</span>;</span></span></code></pre> Now x</code> is a real column in the FROM scope of the outer query. The expression path finds it. The seam has been moved out of the way.</p> This is, conceptually, what you would want the engine to do for you when you write ORDER BY x + 0</code> directly. The SQL-99 standard does not actually require that, though, and Postgres (along with SQL Server) documents explicitly that an alias inside an ORDER BY</code> expression is not supported. So you do it by hand.</p> The boring takeaway</a> </h2> Most of the time none of this matters. You sort by a column you just selected, the alias and the input column have the same name and the same value, and either parser path gives the same answer. The seam is invisible.</p> The minute the alias and the input column disagree in expression, value, case, or anything wrapped around the identifier, the parser picks one or the other silently, by a rule older than most working programmers.</p> There are two parsers. The bare-name path is SQL-92, the expression path is SQL-99, and they were stitched together in the late 1990s. They still disagree about which scope your identifiers live in, and knowing which one you triggered tells you which scope.</p> If after reading this post you still have to stop and think for a minute before predicting what</p> SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre> does, that is the right reaction. It means you have the mental model.</p> The opening puzzle queries are from Jamie Brandon's comment on the Lobsters thread</a> discussing Markus Winand's history of ORDER BY</code></a> on modern-sql.com. Everything that follows here is the explanation that comment did not give. Both pieces are worth reading on their own.</em></p> Strong views on PostgreSQL VIEWs 2026-05-10T00:00:00+00:00 VIEWs should be the cleanest abstraction SQL, and therefore Postgres, has on offer. I love the concept. The promise of decoupling logical intent from physical storage is perfect on paper. In practice, few things in the database world trigger such a heated debate or carry as much historical baggage. VIEWs mix big promises with false hopes, and the promises rarely survive contact with production.</p> The appeal is straightforward. Abstract</strong> "active customer" once and reuse it everywhere. Every query, report and dashboard uses the same definition. The "active customer" then becomes the foundation of a "customer orders" view, which in turn powers an operational "customer summary" view.</p> -- layer 1: who counts as an active customer?</span></span> CREATE VIEW</span> active_customers</span> AS</span></span> SELECT</span> c.</span></span></span> FROM</span> customers c</span></span> WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span> AND</span> c</span>.</span>status</span> =</span> 'active'</span></span> AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span> </span> -- layer 2: active customers with their recent orders</span></span> CREATE VIEW</span> customer_orders</span> AS</span></span> SELECT</span></span> ac.</span></span>,</span></span> o</span>.</span>id</span> AS</span> order_id,</span></span> o</span>.</span>total_cents</span>,</span></span> o</span>.</span>created_at</span> AS</span> ordered_at,</span></span> o</span>.</span>status</span> AS</span> order_status</span></span> FROM</span> active_customers ac</span></span> LEFT JOIN</span> orders o </span>ON</span> o</span>.</span>customer_id</span> =</span> ac</span>.</span>id</span></span> WHERE</span> o</span>.</span>created_at</span> > now</span>()</span> -</span> interval </span>'12 months'</span></span> OR</span> o</span>.</span>created_at</span> IS NULL</span>;</span></span> </span> -- layer 3: one row per customer, ready for the dashboard</span></span> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> co</span>.</span>id</span>,</span></span> co</span>.</span>email</span>,</span></span> co</span>.</span>name</span>,</span></span> COUNT</span>(</span>co</span>.</span>order_id</span>) </span>AS</span> orders_12mo,</span></span> COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>) </span>AS</span> revenue_12mo_cents,</span></span> MAX</span>(</span>co</span>.</span>ordered_at</span>) </span>AS</span> last_order_at,</span></span> COUNT</span>(</span></span>) </span>FILTER</span> (</span>WHERE</span> co</span>.</span>order_status</span> =</span> 'refunded'</span>) </span>AS</span> refunds_12mo</span></span> FROM</span> customer_orders co</span></span> GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre> Each layer has one job. "Active customer" is defined exactly once - if marketing changes the ninety-day rule tomorrow, it is one line in one place, and the dashboard query collapses to SELECT FROM customer_summary WHERE id = $1</code>.</p> VIEWs also have the potential to be a real security boundary</strong>. Three hand-written queries means three places where a predicate can be forgotten, and three subtly different result sets a quarter later. With one definition, you have one result set.</p> For simple views, there is no performance penalty either. PostgreSQL inlines them directly into the calling query, so the planner sees through the view and plans as if you had written the underlying SQL by hand. That mechanism, and the cases where it stops working, has a deep-dive of its own</a>. This article assumes it as background and focuses on the other side: what views are made of, and what happens when the schema underneath them moves.</p> VIEWs are just like tables</a> </h2> The trouble starts with the syntax.</p> SELECT * FROM customer_summary</code> reads exactly like SELECT * FROM customers</code>. The catalog reinforces it: pg_class.relkind = 'v'</code> instead of 'r'</code>, but everything else (column lists, grants, comments, even \d</code> output) looks the same. So developers reach for them the same way: joining them, nesting them, wrapping them in other views without thinking about it.</p> A table is a heap with indexes. A view is a stored parse tree that gets expanded into your query at planning time. Most of the time the planner is clever enough that the difference does not matter: simple views are inlined, predicates are pushed down, the resulting plan is what you would have written by hand.</p> The nested view spiral</a> </h3> In practice, what you get is the dependency tree from the introduction (customer_summary</code> on top of customer_orders</code> on top of active_customers</code>) and a new engineer who has no way to know that SELECT * FROM customer_summary WHERE id = $1</code> expands into a three-layer rewrite, a LEFT JOIN</code> against twelve months of orders, and a GROUP BY</code> that the planner cannot push the id</code> predicate through.</p> Nothing in the query, the schema browser, or the ORM signals this. You only learn the shape of the tree by reading every definition top to bottom, which is exactly the work the abstraction was meant to save you.</p> You cannot reason about the performance of a query against a view without reading the view, the views it depends on, and the tables underneath.</p> Views are rewrite rules</a> </h3> A view is a macro. When you reference one, Postgres pastes its body into your query before the planner runs. There is no stored result, no view "object" the executor consults. The pg_class</code> row you see when you query the catalog is an empty shell: it holds the name, the column list, and the grants, but no definition.</p> The definition itself, the SELECT</code> you wrote, is stored separately in pg_rewrite</code>:</p> SELECT</span> ev_class::regclass, ev_type, is_instead</span></span> FROM</span> pg_rewrite</span></span> WHERE</span> ev_class </span>=</span> 'active_customers'</span>::regclass;</span></span></code></pre> ev_class \| ev_type \| is_instead</span></span> ------------------+---------+------------</span></span> active_customers \| 1 \| t</span></span></code></pre> That row is the view. ev_type = 1</code> is SELECT</code>, is_instead = t</code> means "replace, do not augment" (the pg_class</code> shell has no rows to read, so substitution is the only option that makes sense). Between the parser and the planner Postgres runs a stage called the rewriter</strong>, and the rewriter's job is to walk the parse tree, find references to relations that have rules, and substitute the rule body in place. This way a query like SELECT * FROM active_customers WHERE id = $1</code> arrives at the planner already rewritten:</p> -- what the planner actually sees</span></span> SELECT </span></span> FROM</span> (</span></span> SELECT</span> c</span>.</span>id</span>, </span>c</span>.</span>email</span>, </span>c</span>.</span>name</span>, </span>c</span>.</span>status</span>,</span></span> c</span>.</span>last_login_at</span>, </span>c</span>.</span>deleted_at</span></span> FROM</span> customers c</span></span> WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span> AND</span> c</span>.</span>status</span> =</span> 'active'</span></span> AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span></span> ) active_customers</span></span> WHERE</span> id </span>=</span> $</span>1</span>;</span></span></code></pre> Notice that c.</code> is already expanded into an explicit column list. That happened at CREATE VIEW</code> time and was frozen into the stored parse tree; the rewriter does not re-expand it. From here the planner can usually flatten the subquery into the outer query through a pass called subquery pull-up</strong>, leaving id = $1</code> as a predicate next to the others so an index on customers.id</code> is reachable.</p> That flattening is what makes simple views free. When it bails out (because the view body has LIMIT</code>, DISTINCT</code>, an aggregate, a set operation, or a few other shapes) the subquery stays put and outer predicates cannot move past it. The full list of planner barriers and how to spot them in EXPLAIN</code></a> lives in the inlining post.</p> Historically Postgres exposed this same machinery as user-facing CREATE RULE</code>, on the theory that arbitrary query rewriting was a general-purpose feature. It mostly was not. Rules are effectively deprecated outside of views; triggers do the same jobs without the surprises. The rule infrastructure survives because views need it.</span></p> Once you hold that model, every awkward thing about views falls out of it. Two in particular drive the rest of this article.</p> Columns are referenced by attribute number, not name.</strong> The stored parse tree does not remember that it reads customers.email</code>; it remembers that it reads attribute 2 of relation 16385. Rename the column and the view keeps working, because the OID and the position are unchanged. Drop a column in the middle and Postgres refuses upfront, because the stored attribute numbers would no longer line up. Every dependency error later in this article traces back to this.</p> The body is expanded once per reference.</strong> Every mention of the view in a query produces an independent copy of its parse tree in the rewritten plan. For a stable expression that is invisible. For a volatile one like random()</code> or clock_timestamp()</code>, each copy evaluates separately:</p> CREATE VIEW</span> v_rand</span> AS</span></span> SELECT</span> id, random()</span> AS</span> r </span>FROM</span> customers;</span></span> </span> SELECT</span> a</span>.</span>id</span>, </span>a</span>.</span>r</span> AS</span> r_a, </span>b</span>.</span>r</span> AS</span> r_b</span></span> FROM</span> v_rand a</span></span> JOIN</span> v_rand b </span>ON</span> a</span>.</span>id</span> =</span> b</span>.</span>id</span></span> LIMIT</span> 3</span>;</span></span></code></pre> id \| r_a \| r_b</span></span> ----+----------------------+---------------------</span></span> 1 \| 0.84043639656488 \| 0.0833458769902089</span></span> 2 \| 0.009846241116064247 \| 0.6574000469586228</span></span> 3 \| 0.14667469313524628 \| 0.21862693208148087</span></span></code></pre> Same view, same row, two different r</code> values. The rewriter expanded v_rand</code> twice, so the rewritten query contains two independent random()</code> calls, and the planner has no reason to share them. You might hit this expecting a.r = b.r</code> and end up rewriting the view to push the volatile call outside.</p> This is also why CREATE OR REPLACE VIEW</code> can only append columns to the end of the column list. Existing attribute numbers must stay stable for every dependent, so anything that would shift them is forbidden.</span></p> A useful side effect of the same mechanism: views are immune to the search-path attacks that bite SECURITY DEFINER</code> functions. Identifiers are resolved to OIDs at CREATE VIEW</code> time, so a later SET search_path</code> cannot redirect a view to a different table. Functions resolve identifiers per call against the caller's search_path</code> unless you pin it; views never have that exposure.</p> Writable views, the half-kept promise</a> </h3> Funnily enough, I was not aware writable views existed until earlier this year. It came up during a hallway conversation with Ervin Weber</a> and Jesper St John</a> at PgDay Nordic 2026 in Helsinki. Twelve years writing PostgreSQL and the auto-updatable rules had simply never crossed my desk.</span></p> The original promise was symmetry: if a view looks like a table for reads, it should look like one for writes. Postgres delivers this for the easy case (a view over a single base table, no joins, no aggregates, no DISTINCT</code>) and calls them auto-updatable</strong>. You can INSERT</code>, UPDATE</code>, and DELETE</code> through them, and the rewrite rule translates the operation against the underlying table.</p> INSERT INTO</span> active_customers (email, </span>name</span>)</span></span> VALUES</span> (</span>'alice@example.com'</span>, </span>'Alice'</span>);</span></span> -- becomes: INSERT INTO customers (email, name, status, last_login_at, deleted_at)</span></span> -- VALUES ('alice@example.com', 'Alice', DEFAULT, DEFAULT, DEFAULT);</span></span></code></pre> Add a join, an aggregate, a GROUP BY</code>, or anything else outside the auto-updatable rules and the write path goes silent. SELECT</code> still works, but INSERT</code> errors and you have to wire up INSTEAD OF</code> triggers by hand. So customer_orders</code> and customer_summary</code> are read-only by accident, not by design, and the rules for what counts are not visible in the view definition itself. You find out at write time, in production.</p> WITH CHECK OPTION</code> rejects any write that would produce a row the view cannot see, with LOCAL</code> checking only this view's predicate and CASCADED</code> (the default) checking every underlying view up the chain. It is how you hand an application a writable, scoped slice of a table without trusting it to enforce the scope itself.</p> Views and row-level security</a> </h3> RLS policies live on tables, not views. For a regular view that matters less than it sounds: the policies still fire when the view's query hits the base table. What determines whether they fire correctly is whose context the view runs under.</p> By default, a view runs as its owner. Superusers bypass RLS unconditionally; table owners bypass it unless FORCE ROW LEVEL SECURITY</code> is set. A view owned by a privileged role is therefore a hole in any RLS scheme, silently, because nothing in the view definition or the catalog signals it.</p> Since PostgreSQL 15 the fix is a single option:</p> CREATE VIEW</span> my_view</span> WITH</span> (security_invoker </span>=</span> true) </span>AS</span></span> SELECT</span> ... </span>FROM</span> base_table;</span></span></code></pre> With security_invoker</code> set, the view runs as the calling user and RLS evaluates against the actual caller. Before 15, the workaround was a dedicated low-privilege view owner plus FORCE ROW LEVEL SECURITY</code> on the base table.</p> Materialized views are a different problem. REFRESH MATERIALIZED VIEW</code> runs as the owner, so the snapshot contains whatever the owner can see. Every user then reads that same materialized data. There is no per-row filtering because the rows are physically stored; security_invoker</code> has nothing to attach to.</p> The architect's stigma</a> </h3> When I first got into databases, views were the thing senior people warned you off. "Don't use views in production", "views are slow", "views hide the real query". You learned to write the SQL inline, even when the same predicate appeared in twenty places, because the alternative was disapproval at the next code review.</p> In Oracle shops the warning had specific roots. The cost-based optimiser of that era struggled to merge predicates through views with non-trivial projections, joins, or set operations, so a clean three-layer view could expand into a plan that scanned everything and filtered late. The shop-wide answer was almost always materialised views or PL/SQL packages that encapsulated every access path; a simple view sat awkwardly between the two, not cached like a matview, not encapsulated like a package, and with optimiser surprises on top.</span></p> Postgres carried the same baggage by association. Nested-view spirals were real, rule-system surprises were real, and the optimiser of fifteen years ago was less capable than today's. The rule of thumb hardened into doctrine and outlived its conditions.</p> The improvements are real; the reputation has not caught up. Reach for a view in a code review today and someone will still object on principle. Worth knowing before your PR hits review.</p> What happens when you change a table</a> </h2> Everything covered so far stays invisible while the schema underneath holds still. View bodies live in pg_rewrite</code>, columns are pinned to attribute numbers, types are frozen at CREATE VIEW</code> time, and the planner just expands and runs.</p> The moment someone tries to change a column the view tree references, every one of those mechanics surfaces at once. Suppose someone needs to drop the name</code> column from customers</code>:</p> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span></code></pre>ERROR: cannot drop column name of table customers because other objects depend on it</span></span> DETAIL: view active_customers depends on column name of table customers</span></span> HINT: Use DROP ... CASCADE to drop the dependent objects too.</span></span></code></pre> active_customers</code> references name</code> (the c.</code> was expanded at view-creation time), customer_orders</code> inherits the dependency through ac.</code>, and customer_summary</code> is another layer down. The whole tree is frozen on one column.</p> It is not just dropping. Try widening a type:</p> ALTER TABLE</span> customers </span>ALTER</span> COLUMN email </span>TYPE TEXT</span>;</span></span></code></pre>ERROR: cannot alter type of a column used by a view or rule</span></span> DETAIL: rule _RETURN on view active_customers depends on column "email"</span></span></code></pre> The same wall blocks any structural change to a column a view references: drop, rename, type change. Even widening VARCHAR(255)</code> to TEXT</code>, a change that loses no data and adds no constraint, is forbidden because the view's stored definition references the old type OID.</p> The hint suggests CASCADE</code>:</p> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span> CASCADE;</span></span></code></pre>NOTICE: drop cascades to 3 other objects</span></span> DETAIL: drop cascades to view active_customers</span></span> drop cascades to view customer_orders</span></span> drop cascades to view customer_summary</span></span></code></pre> All three views are dropped, not modified, along with their grants and any other dependent objects. On a schema with dozens of interconnected views, CASCADE</code> is nightmare.</p> Never use CASCADE on production views without a full recreation script ready.</strong> CASCADE doesn't modify views, it drops them. All GRANTs, row-level security policies, and downstream dependencies disappear with them. There is no undo. </div> So the manual path is: save every view definition, drop them in reverse dependency order, alter the table, recreate them in forward dependency order, reapply all grants. For three views this is tedious. For thirty views across multiple schemas, it is a full migration project.</p> -- 1. save definitions </span></span> SELECT</span> pg_get_viewdef(</span>'customer_summary'</span>, true);</span></span> SELECT</span> pg_get_viewdef(</span>'customer_orders'</span>, true);</span></span> SELECT</span> pg_get_viewdef(</span>'active_customers'</span>, true);</span></span> </span> -- 2. drop in leaf-first order</span></span> DROP VIEW</span> customer_summary;</span></span> DROP VIEW</span> customer_orders;</span></span> DROP VIEW</span> active_customers;</span></span> </span> -- 3. now you can alter the table</span></span> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span> </span> -- 4. recreate in correct order</span></span> CREATE VIEW</span> active_customers</span> AS</span></span> SELECT</span> id, email, </span>status</span>, last_login_at, deleted_at</span></span> FROM</span> customers</span></span> WHERE</span> deleted_at </span>IS NULL</span></span> AND status =</span> 'active'</span></span> AND</span> last_login_at </span>> now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span> </span> CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span> </span> -- 5. reapply grants and so on</span></span> -- ...</span></span></code></pre>The SELECT * trap</a> </h2> You might think SELECT </code> saves you from column-level dependencies. It does not. It makes things worse.</p> CREATE VIEW</span> all_customers</span> AS</span></span> SELECT FROM</span> customers;</span></span></code></pre> This looks flexible. PostgreSQL expands SELECT </code> at view creation time</strong> and freezes the result. Check what the database actually stored:</p> SELECT</span> pg_get_viewdef(</span>'all_customers'</span>::regclass, true);</span></span></code></pre> SELECT id,</span></span> email,</span></span> name,</span></span> status,</span></span> last_login_at,</span></span> deleted_at</span></span> FROM customers;</span></span></code></pre> The </code> was expanded to the columns that existed when the view was created. Now add a column to the base table:</p> ALTER TABLE</span> customers </span>ADD</span> COLUMN phone </span>TEXT</span>;</span></span> SELECT * FROM</span> all_customers;</span></span></code></pre> The phone</code> column is not there</strong>. The view still returns the original columns. To pick up new columns, you have to CREATE OR REPLACE VIEW all_customers AS SELECT * FROM customers</code>, which re-expands the </code> against the current table definition.</p> Drop a column that was in the original expansion and the migration is blocked the same way it would be with an explicit column list, except now the dependency is hidden in the catalog instead of visible in the view body. SELECT </code> gives the illusion of flexibility while creating the same rigid coupling.</p> This is actually documented behaviour and follows the SQL standard. But it surprises nearly everyone the first time they encounter it, and it is the source of many "my view is missing columns" bug reports.</span></p> Always use explicit column lists in views.</strong> At least then the dependency is visible and the breakage is predictable.</p> The dependency wall, the type-OID coupling, the frozen SELECT </code> expansion: all of it is the same trade-off, applied consistently, trading flexibility for catching breakage early.</p> Where this isn't a PostgreSQL bug</a> </h2> The rigidity is a design choice, not a defect. Other databases made different choices, and none of them are free.</p> Oracle</strong> marks dependent views as INVALID</code> rather than blocking the DDL. The views are recompiled automatically on next access. If they still work, great; if not, you get an error at query time. This sounds better until you realize what it means in practice: you can deploy a migration, get a clean exit code, and not discover that a critical reporting view is broken until a user hits it on Monday morning. The breakage moves from migration time, where you are paying attention, to runtime, where you may not be. For teams that value deployment confidence, this is arguably worse.</p> SQL Server</strong> has sp_refreshview</code>, which recompiles a view's metadata against the current table definitions. You can alter a table and then refresh the dependent views to pick up the changes. But sp_refreshview</code> works on one view at a time. There is no built-in way to refresh an entire dependency chain in the right order. And if a view references a dropped column, sp_refreshview</code> fails; it does not remove the reference for you. You still have to manually edit and recreate the view. It is a convenience, not a solution.</p> PostgreSQL chose compile-time safety: no surprise INVALID</code> views in production, no silently wrong results from a lazy recompilation papering over a structural change. The cost is manual dependency management on every schema change, and it is high enough to push experienced teams away from views entirely.</p> The saving grace: transactional DDL</a> </h2> Before the workarounds, there is one PostgreSQL feature that materially changes the risk calculus: DDL is transactional</strong>. The entire drop-alter-recreate-regrant sequence can live inside a single BEGIN/COMMIT</code>:</p> BEGIN</span>;</span></span> DROP VIEW</span> customer_summary;</span></span> DROP VIEW</span> customer_orders;</span></span> DROP VIEW</span> active_customers;</span></span> </span> ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span> </span> CREATE VIEW</span> active_customers</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span> CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span> </span> GRANT SELECT ON</span> customer_summary </span>TO</span> reporting;</span></span> COMMIT</span>;</span></span></code></pre> If any statement raises (a typo in a recreated view, a missing grant, an unexpected dependency) the whole migration rolls back and the database is exactly as it was. Oracle and SQL Server cannot do this for most DDL; their CREATE VIEW</code> auto-commits, so a partial failure leaves you with a half-migrated schema and no ROLLBACK</code> to lean on. PostgreSQL's "you have to drop and recreate everything" pain is real, but the recovery story is much better than Oracle's or SQL Server's.</p> This does not solve the locking problem. DROP VIEW</code> and CREATE VIEW</code> take AccessExclusiveLock</code> on the view; ALTER TABLE</code> takes one on the table. For the duration of the transaction, anything that touches those objects waits behind it. A long view-rebuild transaction on a busy system is a stop-the-world window. Keep the transaction tight: prepare the new SQL outside the transaction, hold no other locks, and do not run it during peak traffic. Transactional DDL gives you safety, not concurrency.</p> The workarounds</a> </h2> None of these solve the fundamental problem; they make it manageable.</p> Avoid views entirely.</strong> Replace views with application-level query builders or ORM scopes. You lose the shared abstraction: when three services need the same definition of "active customer," each one implements it independently. Definitions drift. For teams with full control over a small codebase, this is a reasonable trade. For shared data platforms, it is not.</p> Script the recreation.</strong> Use pg_get_viewdef()</code> to extract definitions, then write migration scripts that drop and recreate views around schema changes. The problems: no impact analysis (you have to figure out the dependency order yourself), no grant preservation (view definitions do not include permissions), and it does not scale past a handful of views. Your migration framework will not help here either. They run your SQL files in order, but figuring out what</em> SQL to write is entirely on you.</p> Version your views.</strong> Instead of modifying views in place, create active_customers_v2</code> alongside active_customers_v1</code>. Migrate consumers one at a time. Drop the old version when nothing uses it. This works well when views are consumed by multiple independent teams, since you cannot force everyone to migrate in the same deployment window. The trade-off is naming discipline and version proliferation: if customer_orders_v1</code> depends from active_customers_v1</code>, you now need customer_orders_v2</code> too.</p> Schema-based versioning (CREATE SCHEMA api_v2; CREATE VIEW api_v2.active_customers AS ...</code>) is cleaner than suffixes for public-facing APIs. It gives you a natural namespace and lets you SET search_path</code> to switch versions.</span></p> Query pg_depend yourself.</strong> PostgreSQL tracks all object dependencies in the pg_depend</code> system catalog. The catch: views do not depend on tables directly. The rewrite rule that implements the view does, so every traversal goes pg_depend -> pg_rewrite -> pg_class</code>. Direct view dependents of customers</code>:</p> SELECT</span></span> depns</span>.</span>nspname</span> AS</span> dependent_schema,</span></span> depc</span>.</span>relname</span> AS</span> dependent_view,</span></span> COALESCE</span>(</span></span> (</span>SELECT</span> a</span>.</span>attname</span> FROM</span> pg_attribute a</span></span> WHERE</span> a</span>.</span>attrelid</span> =</span> d</span>.</span>refobjid</span> AND</span> a</span>.</span>attnum</span> =</span> d</span>.</span>refobjsubid</span>),</span></span> ''</span></span> ) </span>AS</span> source_column</span></span> FROM</span> pg_depend d</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>refobjid</span> =</span> 'customers'</span>::regclass</span></span> AND</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> d</span>.</span>refclassid</span> =</span> 'pg_class'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> AND</span> depc</span>.</span>oid</span> <></span> 'customers'</span>::regclass</span></span> ORDER BY</span> depns</span>.</span>nspname</span>, </span>depc</span>.</span>relname</span>;</span></span></code></pre> Useful, but only one hop. customer_summary</code> depends on customers</code> transitively through two intermediate views, which this query misses. A recursive CTE walks the chain:</p> WITH RECURSIVE</span> view_deps </span>AS</span> (</span></span> -- direct dependents of the target table</span></span> SELECT DISTINCT</span></span> depc</span>.</span>oid</span> AS</span> view_oid,</span></span> depns</span>.</span>nspname</span> \|\|</span> '.'</span> \|\|</span> depc</span>.</span>relname</span> AS</span> view_name,</span></span> 1</span> AS</span> depth</span></span> FROM</span> pg_depend d</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>refobjid</span> =</span> 'customers'</span>::regclass</span></span> AND</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> </span> UNION</span></span> </span> -- views depending on views we already found</span></span> SELECT</span> depc</span>.</span>oid</span>,</span></span> depns</span>.</span>nspname</span> \|\|</span> '.'</span> \|\|</span> depc</span>.</span>relname</span>,</span></span> vd</span>.</span>depth</span> +</span> 1</span></span> FROM</span> view_deps vd</span></span> JOIN</span> pg_depend d </span>ON</span> d</span>.</span>refobjid</span> =</span> vd</span>.</span>view_oid</span></span> JOIN</span> pg_rewrite r </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span> JOIN</span> pg_class depc </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span> JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span> WHERE</span> d</span>.</span>classid</span> =</span> 'pg_rewrite'</span>::regclass</span></span> AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span> AND</span> depc</span>.</span>oid</span> <></span> vd</span>.</span>view_oid</span> -- skip the view's own _RETURN rule</span></span> )</span></span> SELECT</span> view_name, </span>MIN</span>(depth) </span>AS</span> depth</span></span> FROM</span> view_deps</span></span> GROUP BY</span> view_name</span></span> ORDER BY</span> depth, view_name;</span></span></code></pre> The depc.oid <> vd.view_oid</code> filter matters: each view's _RETURN</code> rule has a self-dependency entry that without the filter sends the walk back to the view it just visited.</p> This works, but at this point you are writing a dependency-analysis tool. Add topological sort, grant capture, matview handling, RLS-policy preservation, and you have small product.</p> Materialized views have it worse.</strong> Everything above applies, plus you lose the cached result set on DROP</code>. After recreating, REFRESH</code> rebuilds from scratch: minutes to hours on large datasets, with no CONCURRENTLY</code> option until a unique index exists. For matviews that back dashboards, this means downtime. RLS is also off the table: REFRESH</code> runs as the matview owner, so the snapshot reflects the owner's access, not the querying user's. </div> Prior art hiding in pg_dump</a> </h2> The mechanism the manual workaround keeps reinventing, rewriting a view's body while preserving its identity, grants, and dependents, already exists inside PostgreSQL. It happens to live in pg_dump</code>, used only when needed.</p> Most of the time pg_dump</code> does the obvious thing: builds a dependency graph, topologically sorts it, and emits each view in base-first order with a normal CREATE VIEW</code>. Dump our four-view chain and that is exactly what you get.</p> The interesting case is when the sort fails. Cycles are uncommon but real: a view body that calls a function whose own body references the view back, a trigger on a base table that reads through a view reading the same table, cross-referencing matviews with RLS policies pointing back at views. When pg_dump</code> hits one, it falls back to the placeholder view trick</strong>: emit one of the views early as a stub with the right column list and types but a dummy body, then come back with CREATE OR REPLACE VIEW</code> to install the real definition once the rest of the cycle exists.</p> -- emitted early, as a placeholder</span></span> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> NULL</span>::</span>integer AS</span> id,</span></span> NULL</span>::</span>text AS</span> email,</span></span> NULL</span>::</span>text AS name</span>,</span></span> NULL</span>::</span>bigint AS</span> orders_12mo,</span></span> NULL</span>::</span>bigint AS</span> revenue_12mo_cents;</span></span> </span> -- emitted later, once dependencies exist</span></span> CREATE OR REPLACE VIEW</span> customer_summary</span> AS</span></span> SELECT</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>,</span></span> COUNT</span>(</span>co</span>.</span>order_id</span>) </span>AS</span> orders_12mo,</span></span> COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>) </span>AS</span> revenue_12mo_cents</span></span> FROM</span> customer_orders co</span></span> GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre> Grants, comments, and policies attach to the stub and survive the rewrite because the OID never changes. CREATE OR REPLACE VIEW</code> mutates the same pg_class</code> row in place. PostgreSQL has been doing this on every pg_dump --schema-only</code> for years. The mechanism is there; it is simply not exposed as user-facing DDL.</p> Before you reach for a VIEW</a> </h2> Views are worth using. The abstraction is real: shared definitions, clean layering, column-level security, and no runtime cost when the planner can inline them. The trouble is that none of the rigidity is visible from the outside. A view looks like a table in the catalog, in \d</code>, in the ORM, and in every query that touches it. The teardown cost only appears the first time someone tries to drop a column, widen a type, or CASCADE</code> their way out of a migration on a Friday afternoon.</p> So before you reach for one, hold the trade-off in mind:</p> SELECT </code> in a view body is a trap.</strong> It freezes the column list at creation time, hides the dependency in the catalog, and still blocks the same DDL an explicit list would. Always write the columns out.</li> Every layer multiplies the teardown cost.</strong> A three-deep view chain means three drops, three recreates, three sets of grants and policies to reapply, in the right order, for any structural change to a column at the bottom. Keep dependency trees shallow, and be honest about whether the abstraction is paying for itself.</li> CASCADE</code> is not a fix.</strong> It drops dependent views along with their grants, RLS policies, and downstream dependencies, with no undo. Never run it in production without a recreation script ready.</li> security_invoker</code> is not the default.</strong> A view runs as its owner, which silently bypasses RLS on base tables when the owner is privileged. On PostgreSQL 15+, WITH (security_invoker = true)</code> is what makes the security boundary real. For materialized views there is no equivalent. Enforce access at the layer that reads the matview, never below it.</li> </ul> When the migration finally has to happen, two things make the cycle survivable. Wrap the whole drop-alter-recreate-regrant sequence in a transaction so a typo or missed grant rolls back cleanly, and keep that transaction tight — AccessExclusiveLock</code> blocks everything else for its duration. Then map the dependency graph before you cut: pg_depend</code> joined through pg_rewrite</code>, walked with a recursive CTE, tells you what actually breaks. Run that query before the migration, not after the incident.</p> The pain traces back to one missing primitive. ALTER VIEW</code> today handles renames, owner and schema changes, column defaults, and options like security_barrier</code>, but nothing structural. CREATE OR REPLACE VIEW</code> can append columns at the end and nothing else. A real ALTER VIEW DROP COLUMN</code>, ADD COLUMN</code>, ALTER COLUMN TYPE</code> is what would make views safe to evolve. The catalog and the in-place-rewrite mechanism are already there, as pg_dump</code>'s placeholder trick demonstrates; the user-facing DDL is what is missing.</p> Even without it, views are still worth using. Just don't pretend they're tables.</p> EDIT (2026-05-17)</em>: The original article understated how RLS interacts with views. Added a section on security_invoker</code> and why the ownership model matters, plus a note on why materialized views are structurally incompatible with RLS.</p> Good CTE, bad CTE 2026-03-29T14:47:00+02:00 The Common Table Expression</a>, or CTE, is often the first feature developers reach for beyond basic SQL, and often the only one. You write a subquery after WITH</code>, give it a name, and use it in the rest of your query. It only exists for the duration of that query.</p> But the popularity of CTEs usually has less to do with modernizing code and more to do with the promise of imperative logic. For many, CTE acts as an easy to understand remedy for 'scary queries' and way how to force execution order on the database. The way how many write queries is as if they tell optimizer "first do this, then do that".</p> This creates a problem. CTEs handle query decomposition, recursion and multi statement DDLs. Planner treats them differently depending how you write and use them though. For long time (prior PostgreSQL 12) CTEs acted as optimization fence. The planner couldn't push predicates into them, couldn't use indexes on the underlying tables. Couldn't do anything that materialize them and scan through the result.</p> PostgreSQL 12 changed this. CTEs now get inlined, materialized, or something in between, depending on how you write them.</p> Sample schema</a> </h2> We will use the same schema as in the article PostgreSQL Statistics: Why queries run slow</a>.</p> CREATE TABLE</span> customers</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span></span> );</span></span> </span> CREATE TABLE</span> orders</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span> amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span> note </span>text</span>,</span></span> created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> CREATE TABLE</span> orders_archive</span> (</span>LIKE</span> orders INCLUDING ALL EXCLUDING </span>IDENTITY</span>);</span></span> </span> INSERT INTO</span> customers (</span>name</span>)</span></span> SELECT</span> 'Customer '</span> \|\|</span> i</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span> </span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span> SELECT</span></span> (random()</span> </span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span> (random()</span> </span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span> (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()4+1)::int],</span></span> CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span> '2022-01-01'</span>::</span>date +</span> (random()</span> </span> 1095</span>)::</span>int</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span> </span> ANALYZE customers;</span></span> ANALYZE orders;</span></span></code></pre> For recursive examples later on we'll also need an employees</code> table with a self-referencing hierarchy:</p> CREATE TABLE</span> employees</span> (</span></span> id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name text NOT NULL</span>,</span></span> manager_id </span>integer REFERENCES</span> employees(id),</span></span> department </span>text NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> employees (</span>name</span>, manager_id, department) </span>VALUES</span></span> (</span>'Alice'</span>, </span>NULL</span>, </span>'Engineering'</span>),</span></span> (</span>'Bob'</span>, </span>1</span>, </span>'Engineering'</span>),</span></span> (</span>'Charlie'</span>, </span>1</span>, </span>'Engineering'</span>),</span></span> (</span>'Diana'</span>, </span>2</span>, </span>'Engineering'</span>),</span></span> (</span>'Eve'</span>, </span>2</span>, </span>'Engineering'</span>),</span></span> (</span>'Frank'</span>, </span>3</span>, </span>'Sales'</span>),</span></span> (</span>'Grace'</span>, </span>3</span>, </span>'Sales'</span>),</span></span> (</span>'Hank'</span>, </span>6</span>, </span>'Sales'</span>),</span></span> (</span>'Ivy'</span>, </span>6</span>, </span>'Sales'</span>);</span></span> </span> ANALYZE employees;</span></span></code></pre>The optimization fence era (pre-PG 12)</a> </h2> As we already covered before PostgreSQL 12, every CTE was materialized. No exceptions. The planner would compute the CTE result set in full, store it in a temporary tuplestore, and then scan that tuplestore whenever the main query referenced the CTE. This made CTEs an optimization fence</strong> because the planner could not look through them.</p> Consider this simple query:</p> EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span> SELECT FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> On PostgreSQL 11 and earlier, the EXPLAIN</code> output would look something like this:</p> QUERY PLAN</span></span> -------------------------------------------------------------------</span></span> CTE Scan on filtered (cost=1840.00..2290.00 rows=2 width=58)</span></span> Filter: (status = 'pending')</span></span> CTE filtered</span></span> -> Seq Scan on orders (cost=0.00..1840.00 rows=10000 width=58)</span></span> Filter: (created_at > '2025-01-01'::date)</span></span></code></pre> Notice what happens here. The CTE runs a sequential scan on orders</code> with the date filter. It materializes all matching rows. Then the outer query applies the status = 'pending'</code> filter after</em> materialization. Even if a composite index on (created_at, status)</code> existed, the planner couldn't use it as it can't see through the CTE boundary to combine the predicates.</p> Why was it designed this way? Two reasons. First, reason was snapshot isolation. Materializing the CTE guaranteed that the result set was computed once, from a single snapshot, regardless of how many times it was referenced. Second, as protection for side-effect edge cases. If a CTE contained a data-modifying statement (INSERT</code>, UPDATE</code>, DELETE</code>), materialising ensured it executed exactly once.</p> The workaround was well-known in the community: rewrite CTEs as subqueries. Subqueries had always been subject to the planner's normal optimisation rules, including predicate pushdown and inlining. The same query written as SELECT * FROM (SELECT * FROM orders WHERE created_at > '2025-01-01') sub WHERE status = 'pending'</code> would produce a much better plan.</div> This led to an entire workaround culture. Developers would write queries with CTEs for readability during development, then rewrite them as nested subqueries for production. The community had a saying: CTEs are optimization fences</em>. It was repeated so often. Many developers still believe it today. But it hasn't been true since PostgreSQL 12.</p> PostgreSQL 12: CTE inlining</a> </h2> PostgreSQL 12 introduced automatic CTE inlining. Non-recursive, side-effect-free, singly-referenced CTEs are now inlined by default. The planner started treating them as subqueries and applies all its normal optimisations. Predicate pushdown, index usage, join reordering apply exactly as if the CTE syntax never existed.</p> The same query from the previous section now produces a completely different plan:</p> EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2355.00 rows=2 width=58)</span></span> Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span></code></pre> The CTE is gone from the plan entirely. Both predicates are merged into a single scan on orders</code>. If there's a suitable index, the planner can use it. The CTE syntax doesn't change the execution plan.</p> PostgreSQL 12 also introduced two new keywords that let you override the planner's decision:</p> MATERIALIZED</code> - forces the CTE to materialize, even when the planner would inline it</li> NOT MATERIALIZED</code> - forces inlining, even when the planner would materialize it</li> </ul> -- force materialization</span></span> EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span> </span> -- Force inlining</span></span> EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> This follows the same principle as VIEW inlining</a>.</p> When does a CTE get materialized?</a> </h2> Case 1: single reference, no side effects (INLINED)</a> </h3> The simplest and most common case. If you reference the CTE exactly once and it contains no side effects, the planner inlines it.</p> EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> ------------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2355.00 rows=2 width=59)</span></span> Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span> (2 rows)</span></span></code></pre> Both predicates are merged. The planner considers all access paths on orders</code> directly.</p> Case 2: multiple references (MATERIALIZED)</a> </h3> When a CTE is referenced more than once, the planner materializes it. This is actually a feature, CTE is computed once and reused. Therefore avoiding redundant work.</p> EXPLAIN </span>WITH</span> summary </span>AS</span> (</span></span> SELECT status</span>, </span>count</span>(</span></span>) </span>AS</span> cnt </span>FROM</span> orders </span>GROUP BY status</span></span> )</span></span> SELECT</span> a</span>.</span>status</span>, </span>b</span>.</span>status</span></span> FROM</span> summary a, summary b</span></span> WHERE</span> a</span>.</span>cnt</span> ></span> b</span>.</span>cnt</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------</span></span> Nested Loop (cost=2355.04..2355.52 rows=5 width=64)</span></span> Join Filter: (a.cnt > b.cnt)</span></span> CTE summary</span></span> -> HashAggregate (cost=2355.00..2355.04 rows=4 width=17)</span></span> Group Key: orders.status</span></span> -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=9)</span></span> -> CTE Scan on summary a (cost=0.00..0.08 rows=4 width=40)</span></span> -> CTE Scan on summary b (cost=0.00..0.08 rows=4 width=40)</span></span> (8 rows)</span></span></code></pre> The CTE Scan</code> nodes appear twice, but the HashAggregate</code> runs only once. For expensive computations referenced multiple times, this is exactly what you want.</p> Case 3: recursive CTE (ALWAYS MATERIALIZED)</a> </h3> Recursive CTEs must maintain a working table between iterations. There's no way to inline them. We'll cover recursion in detail later in the article.</p> EXPLAIN </span>WITH RECURSIVE</span> subordinates </span>AS</span> (</span></span> SELECT</span> id, </span>name</span>, manager_id </span>FROM</span> employees </span>WHERE</span> id </span>=</span> 1</span></span> UNION ALL</span></span> SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span></span> FROM</span> employees e</span></span> JOIN</span> subordinates s </span>ON</span> e</span>.</span>manager_id</span> =</span> s</span>.</span>id</span></span> )</span></span> SELECT FROM</span> subordinates;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> CTE Scan on subordinates (cost=17.21..18.83 rows=81 width=40)</span></span> CTE subordinates</span></span> -> Recursive Union (cost=0.00..17.21 rows=81 width=13)</span></span> -> Seq Scan on employees (cost=0.00..1.11 rows=1 width=13)</span></span> Filter: (id = 1)</span></span> -> Hash Join (cost=0.33..1.53 rows=8 width=13)</span></span> Hash Cond: (e.manager_id = s.id)</span></span> -> Seq Scan on employees e (cost=0.00..1.09 rows=9 width=13)</span></span> -> Hash (cost=0.20..0.20 rows=10 width=4)</span></span> -> WorkTable Scan on subordinates s (cost=0.00..0.20 rows=10 width=4)</span></span> (10 rows)</span></span></code></pre>Case 4: data-modifying CTE (ALWAYS MATERIALIZED)</a> </h3> CTEs that contain INSERT</code>, UPDATE</code>, or DELETE</code> are always materialized. The side effects must execute exactly once, in a predictable order.</p> EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span> DELETE FROM</span> orders </span>WHERE status =</span> 'cancelled'</span> RETURNING </span></span></span> )</span></span> SELECT</span> count</span>(</span></span>) </span>FROM</span> deleted;</span></span></code></pre> QUERY PLAN</span></span> ---------------------------------------------------------------------------</span></span> Aggregate (cost=2670.13..2670.14 rows=1 width=8)</span></span> CTE deleted</span></span> -> Delete on orders (cost=0.00..2105.00 rows=25117 width=6)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25117 width=6)</span></span> Filter: (status = 'cancelled'::text)</span></span> -> CTE Scan on deleted (cost=0.00..502.34 rows=25117 width=0)</span></span> (6 rows)</span></span></code></pre> The CTE Scan</code> is present because the DELETE</code> must be fully executed before the count()</code> can run.</p> Case 5: VOLATILE function (MATERIALIZED)</a> </h3> If a CTE contains a VOLATILE</code> function, the planner materializes it to prevent the function from being evaluated multiple times with potentially different results.</p> EXPLAIN </span>WITH</span> rand </span>AS</span> (</span></span> SELECT</span> id, random()</span> AS</span> r </span>FROM</span> orders</span></span> )</span></span> SELECT FROM</span> rand </span>WHERE</span> r </span><</span> 0</span>.</span>01</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------</span></span> CTE Scan on rand (cost=2105.00..4355.00 rows=33333 width=12)</span></span> Filter: (r < '0.01'::double precision)</span></span> CTE rand</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=100000 width=12)</span></span> (4 rows)</span></span></code></pre> Even though rand</code> is referenced only once, the CTE Scan</code> is there. random()</code> is VOLATILE</code>, which forces materialization.</p> Case 6: STABLE functions (INLINED)</a> </h3> STABLE</code> functions like now()</code> do not</strong> prevent inlining. The reason is that the time is frozen at transaction start.</p> EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span> SELECT * FROM</span> orders</span></span> WHERE</span> created_at </span>> now</span>()</span> -</span> interval </span>'7 days'</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> Seq Scan on orders (cost=0.00..2855.00 rows=2 width=59)</span></span> Filter: ((status = 'pending'::text) AND (created_at > (now() - '7 days'::interval)))</span></span> (2 rows)</span></span></code></pre> No CTE Scan</code>. The planner inlines the CTE and merges both predicates, just like Case 1. now()</code> is STABLE</code> as it returns the same value within a transaction - and the planner's inlining check only looks for VOLATILE</code> functions (via contain_volatile_functions()</code>). STABLE</code> passes that check.</p> Why do people think STABLE</code> blocks inlining? Because before PostgreSQL 12, all</strong> CTEs were materialized regardless of volatility. When PG 12 introduced inlining, the only function-level barrier was VOLATILE</code>. But the old "CTEs are optimization fences" mental model was so deeply ingrained that many developers assumed STABLE</code> was also a problem. It isn't.</p> The function that actually blocks inlining: clock_timestamp()</code>. Unlike now()</code>, it's VOLATILE</code> and returns a different value on every call. A CTE with clock_timestamp()</code> will be materialized. Similarly, random()</code> and nextval()</code> are VOLATILE</code> and force materialization (as shown in Case 5).</p> </div> If you see a CTE with now()</code> being materialized, the cause is something else. Either multiple references, a data-modifying statement, or an explicit MATERIALIZED</code> hint. Don't blame STABLE</code>.</p> Case 7: forcing behaviour with hints</a> </h3> You can always override the planner's decision.</p> -- force materialization on something that would normally be inlined</span></span> EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span> )</span></span> SELECT * FROM</span> filtered </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------</span></span> CTE Scan on filtered (cost=2105.00..2670.58 rows=5290 width=92)</span></span> Filter: (amount > '400'::numeric)</span></span> CTE filtered</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> (5 rows)</span></span></code></pre>-- force inlining on something that would normally be materialized</span></span> EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span> )</span></span> SELECT * FROM</span> filtered a</span></span> JOIN</span> filtered b </span>ON</span> a</span>.</span>customer_id</span> =</span> b</span>.</span>customer_id</span>;</span></span></code></pre> QUERY PLAN</span></span> --------------------------------------------------------------------------------</span></span> Hash Join (cost=2419.21..10686.65 rows=317742 width=118)</span></span> Hash Cond: (orders.customer_id = orders_1.customer_id)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> -> Hash (cost=2105.00..2105.00 rows=25137 width=59)</span></span> -> Seq Scan on orders orders_1 (cost=0.00..2105.00 rows=25137 width=59)</span></span> Filter: (status = 'pending'::text)</span></span> (7 rows) </span></span></code></pre> If the CTE were materialized, you would see a single scan of the orders</code> table, followed by two CTE Scans on the result. Instead, the planner has treated your query as if you had written a standard join between two subqueries.</p> Be careful with NOT MATERIALIZED</code> on multiply-referenced CTEs.</strong> When you force inlining, the subquery runs once for each reference. In the example above, the orders</code> table is scanned twice. Once for a</code> and once for b</code>. For small result sets this might be fine. For large ones, you're doing double the work. Measure before using.</p> </div> Case 8: FOR UPDATE / FOR SHARE (MATERIALIZED)</a> </h3> Row-locking clauses force materialization even on a singly-referenced, side-effect-free CTE. Internally, the planner's contain_dml()</code> check treats FOR UPDATE</code> and FOR SHARE</code> the same as data-modifying statements.</p> EXPLAIN </span>WITH</span> locked </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span> FOR UPDATE</span></span> )</span></span> SELECT * FROM</span> locked </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------</span></span> CTE Scan on locked (cost=2356.37..2921.95 rows=5290 width=92)</span></span> Filter: (amount > '400'::numeric)</span></span> CTE locked</span></span> -> LockRows (cost=0.00..2356.37 rows=25137 width=65)</span></span> -> Seq Scan on orders (cost=0.00..2105.00 rows=25137 width=65)</span></span> Filter: (status = 'pending'::text)</span></span> (6 rows)</span></span></code></pre> Without FOR UPDATE</code>, this CTE would be inlined. The LockRows</code> node and CTE Scan</code> confirm materialization.</p> Decision matrix</a> </h3> Here's the full picture across PostgreSQL versions:</p> Condition</th> PG ≤ 11</th> PG 12–16</th> PG 17–18</th></tr></thead> Single ref, pure SELECT</td> Materialized</td> Inlined</td> Inlined</td></tr> Multiple refs, pure SELECT</td> Materialized</td> Materialized</td> Materialized (better stats)</td></tr> VOLATILE function</td> Materialized</td> Materialized</td> Materialized</td></tr> STABLE function</td> Materialized</td> Inlined</td> Inlined</td></tr> Data-modifying (DML)</td> Materialized</td> Materialized</td> Materialized</td></tr> FOR UPDATE / FOR SHARE</td> Materialized</td> Materialized</td> Materialized</td></tr> Recursive</td> Materialized</td> Materialized</td> Materialized</td></tr> Explicit MATERIALIZED</code></td> -</td> Materialized</td> Materialized</td></tr> Explicit NOT MATERIALIZED</code></td> -</td> Inlined</td> Inlined</td></tr> </tbody></table> The statistics black hole</a> </h2> As mentioned in PostgreSQL Statistics: Why queries run slow</a>, materialized CTEs are one of the places "where no statistics go." This is arguably the biggest practical problem with CTE materialization.</p> When the planner materializes a CTE, the result set is stored in a temporary tuplestore. This tuplestore has no pg_statistic</code> entries - no histograms, no MCVs, no correlation data. The planner has to estimate row counts and value distributions using hardcoded defaults.</p> Let's see this in action. Here's a CTE over 10,000 rows:</p> EXPLAIN </span>WITH</span> all_orders </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> all_orders </span>WHERE status =</span> 'pending'</span> AND</span> amount </span>></span> 400</span>;</span></span></code></pre> QUERY PLAN</span></span> -----------------------------------------------------------------------</span></span> CTE Scan on all_orders (cost=1855.00..4355.00 rows=5290 width=92)</span></span> Filter: ((amount > '400'::numeric) AND (status = 'pending'::text))</span></span> CTE all_orders</span></span> -> Seq Scan on orders (cost=0.00..1855.00 rows=100000 width=59)</span></span> (4 rows)</span></span></code></pre> The planner estimated 5,290 rows. Where does that number come from? The planner has no MCV list for status</code> inside the CTE, no histogram for amount</code>. It falls back to default selectivities of 0.3333</code> for the range comparison on amount</code> and a rough guess for the equality on status</code>, and multiplies them against the 100,000 input rows.</p> If this CTE were inlined, the planner would read the actual statistics from pg_statistic</code> for the orders</code> table and produce estimates based on real data distribution, not defaults.</div> In a simple query this might not matter much. But when a materialized CTE feeds into a join, default-based estimates can cascade. The planner might choose a nested loop where a hash join would be better, or vice versa. It might underestimate memory needs and spill to disk unexpectedly.</p> PG 17: Statistics propagation</a> </h3> PostgreSQL 17 brought two significant improvements to materialized CTEs:</p> Column statistics propagation.</strong> When the planner creates a CTE Scan</code> node, it now propagates column statistics from the underlying query into the scan node. This means n_distinct</code>, MCV lists, and histograms from the source table can inform estimates on the CTE scan.</p> Path key propagation.</strong> Materialized CTEs now preserve sort order information. If the CTE's subquery produces sorted output, the planner knows about it and can skip redundant sorts downstream.</p> These improvements significantly reduce the estimation gap, but they don't eliminate it. Inlined CTEs are still strictly better for planning accuracy, because the planner works directly with the base table statistics rather than propagated copies. If your CTE doesn't need to be materialized, don't force it.</p> When materialization helps</a> </h2> Materialization isn't always bad.</p> Multiple references.</strong> If the CTE result is used in multiple places, materialization computes it once. Without it, the subquery runs once per reference.</p> EXPLAIN </span>WITH</span> monthly_totals </span>AS</span> (</span></span> SELECT</span> date_trunc(</span>'month'</span>, created_at) </span>AS month</span>,</span></span> status</span>,</span></span> sum</span>(amount) </span>AS</span> total</span></span> FROM</span> orders</span></span> GROUP BY</span> 1</span>, </span>2</span></span> )</span></span> SELECT</span> cur</span>.</span>month</span>, </span>cur</span>.</span>status</span>, </span>cur</span>.</span>total</span>,</span></span> prev</span>.</span>total</span> AS</span> prev_month_total,</span></span> cur</span>.</span>total</span> -</span> prev</span>.</span>total</span> AS</span> delta</span></span> FROM</span> monthly_totals cur</span></span> LEFT JOIN</span> monthly_totals prev</span></span> ON</span> cur</span>.</span>month</span> =</span> prev</span>.</span>month</span> +</span> interval </span>'1 month'</span></span> AND</span> cur</span>.</span>status</span> =</span> prev</span>.</span>status</span>; </span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------</span></span> Merge Left Join (cost=3887.46..3990.90 rows=4384 width=136)</span></span> Merge Cond: ((cur.month = ((prev.month + '1 mon'::interval))) AND (cur.status = prev.status))</span></span> CTE monthly_totals</span></span> -> HashAggregate (cost=3105.00..3181.72 rows=4384 width=49)</span></span> Group Key: date_trunc('month'::text, (orders.created_at)::timestamp with time zone), orders.status</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=100000 width=23)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: cur.month, cur.status</span></span> -> CTE Scan on monthly_totals cur (cost=0.00..87.68 rows=4384 width=72)</span></span> -> Sort (cost=352.87..363.83 rows=4384 width=72)</span></span> Sort Key: ((prev.month + '1 mon'::interval)), prev.status</span></span> -> CTE Scan on monthly_totals prev (cost=0.00..87.68 rows=4384 width=72)</span></span> (12 rows)</span></span></code></pre> The aggregation runs once. Both cur</code> and prev</code> read from the materialized result. Without materialization, the entire aggregation would run twice.</p> Expensive VOLATILE expressions.</strong> If a CTE contains calls to volatile functions or expensive computations, materialization ensures they execute exactly once.</p> Data-modifying operations.</strong> The whole point of writable CTEs is that the side effects happen once and the RETURNING</code> data is available downstream. Materialization is not optional here.</p> When inlining isn't enough</a> </h2> The imperative mindset from the introduction, "first do this, then do that", doesn't go away just because the planner inlines your CTEs. And this one is my personal favourite, and source that never stop delivering query refactoring.</p> Developers still structure queries as sequential pipelines, and that structure itself can create performance problems that have nothing to do with materialization.</p> A common pattern is building queries as an assembly line: one CTE filters rows, the next LEFT JOINs related tables and aggregates metadata with GROUP BY</code>, the next filters on the aggregated results. It reads like a clean pipeline, but the GROUP BY</code> in the middle creates a wall the planner can't optimize past.</p> WITH</span> recent_orders </span>AS</span> (</span></span> SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2024-01-01'</span></span> ),</span></span> order_metadata </span>AS</span> (</span></span> SELECT</span></span> o</span>.</span>id</span>,</span></span> bool_or(</span>oa</span>.</span>id</span> IS NOT NULL</span>) </span>AS</span> was_archived,</span></span> count</span>(</span>o2</span>.</span>id</span>) </span>AS</span> related_count</span></span> FROM</span> recent_orders o</span></span> LEFT JOIN</span> orders_archive oa </span>ON</span> o</span>.</span>id</span> =</span> oa</span>.</span>id</span></span> LEFT JOIN</span> orders o2 </span>ON</span> o</span>.</span>customer_id</span> =</span> o2</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> GROUP BY</span> o</span>.</span>id</span></span> )</span></span> SELECT</span> o.</span></span>, </span>m</span>.</span>was_archived</span>, </span>m</span>.</span>related_count</span></span> FROM</span> recent_orders o</span></span> JOIN</span> order_metadata m </span>ON</span> o</span>.</span>id</span> =</span> m</span>.</span>id</span></span> WHERE</span> m</span>.</span>was_archived</span> =</span> false</span></span> AND</span> m</span>.</span>related_count</span> ></span> 0</span>;</span></span></code></pre> Each CTE here is referenced once, so they all get inlined. No materialization, no optimization fence. The planner sees the full query. So what's the problem?</p> The GROUP BY</code> in order_metadata</code>. Even after inlining, the planner cannot push the was_archived = false</code> predicate past the aggregation. It must first LEFT JOIN every filtered order against orders_archive</code> and self-join orders</code>, compute the aggregates for all of them, and only then discard the rows that don't match. If recent_orders</code> returns 50,000 rows but only 200 were ever archived, you're joining and aggregating 49,800 rows for nothing.</p> The fix is to replace the aggregation-then-filter pattern with correlated EXISTS</code> subqueries:</p> SELECT</span> o.</span></span></span> FROM</span> orders o</span></span> WHERE</span> o</span>.</span>created_at</span> ></span> '2024-01-01'</span></span> AND NOT EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders_archive oa </span>WHERE</span> oa</span>.</span>id</span> =</span> o</span>.</span>id</span></span> )</span></span> AND EXISTS</span> (</span></span> SELECT</span> 1</span> FROM</span> orders o2</span></span> WHERE</span> o2</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span> );</span></span></code></pre> EXISTS</code> short-circuits after finding the first matching row. The planner can push created_at > '2024-01-01'</code> all the way down to an index scan on orders</code>, then probe each related table per result. No aggregation, no wasted work.</p> The rule of thumb:</strong> if your CTE contains a GROUP BY</code> or a LEFT JOIN</code> just to compute a boolean ("does this row have related data?"), you've built a wall the planner can't see past. A correlated EXISTS</code> lets the planner push filters down and stop scanning early. This applies whether the CTE is materialized or inlined.</p> </div> Writable CTEs (the power and the traps)</a> </h2> Data-modifying CTEs let you INSERT</code>, UPDATE</code>, or DELETE</code> inside a WITH</code> clause and use the RETURNING</code> data in subsequent CTEs or the main query.</p> EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span> DELETE FROM</span> orders</span></span> WHERE status =</span> 'cancelled'</span></span> AND</span> created_at </span><</span> '2023-01-01'</span></span> RETURNING </span></span></span> ),</span></span> archived </span>AS</span> (</span></span> INSERT INTO</span> orders_archive</span></span> SELECT FROM</span> deleted</span></span> RETURNING id</span></span> )</span></span> SELECT</span> count</span>(</span></span>) </span>FROM</span> archived;</span></span></code></pre> QUERY PLAN</span></span> ----------------------------------------------------------------------------------------------</span></span> Aggregate (cost=2709.19..2709.20 rows=1 width=8)</span></span> CTE deleted</span></span> -> Delete on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> -> Seq Scan on orders (cost=0.00..2355.00 rows=8334 width=6)</span></span> Filter: ((created_at < '2023-01-01'::date) AND (status = 'cancelled'::text))</span></span> CTE archived</span></span> -> Insert on orders_archive (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on deleted (cost=0.00..166.68 rows=8334 width=92)</span></span> -> CTE Scan on archived (cost=0.00..166.68 rows=8334 width=0)</span></span> (9 rows) </span></span></code></pre> This deletes old cancelled orders, moves them to an archive table, and counts how many were archived - all in a single atomic statement, no application-level coordination needed.</p> But there are sharp edges.</p> You cannot read what you just wrote</a> </h3> All sub-statements in a data-modifying CTE see the same snapshot. This means the effects of one CTE are not visible</strong> to other CTEs or the main query when they read the target table</em>. Only the RETURNING</code> clause communicates data between CTE steps.</p> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre>WITH</span> ins </span>AS</span> (</span></span> INSERT INTO</span> orders (customer_id, amount, </span>status</span>, created_at)</span></span> VALUES</span> (</span>1</span>, </span>100</span>.</span>00</span>, </span>'pending'</span>, CURRENT_DATE)</span></span> RETURNING id</span></span> )</span></span> -- this does NOT see the row we just inserted</span></span> SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span> -------</span></span> 31</span></span> (1 row)</span></span></code></pre> The count(1)</code> query sees the pre-insert snapshot. If you need the inserted data, you must use the RETURNING</code> clause from the ins</code> CTE, not re-read the table.</p> Tuple shuffling</a> </h3> A common pattern is using writable CTEs to move rows between tables atomically:</p> WITH</span> moved </span>AS</span> (</span></span> DELETE FROM</span> orders_staging</span></span> RETURNING </span></span></span> )</span></span> INSERT INTO</span> orders</span></span> SELECT * FROM</span> moved;</span></span></code></pre> This deletes all rows from the staging table and inserts them into the production table in a single atomic operation. No window where data exists in both or neither table.</p> Data-modifying CTEs disable parallel query for the entire statement. If you have a complex query that mixes reads and writes, the write CTE prevents parallelism even for the read-only parts.</div> Recursive CTEs are always materialized</a> </h2> Recursive CTEs use an iterative working-table mechanism. Despite the name, they aren't truly recursive. PostgreSQL doesn't "call itself" by creating a nested stack of unfinished queries. Instead, it operates in loops.</p> Execute the non-recursive term (the "seed"). Put results in the working table.</li> Execute the recursive term using the working table as input. New rows become the next</em> working table.</li> Repeat until the recursive term returns no new rows.</li> Return the union of all iterations.</li> </ol> WITH RECURSIVE</span> org_chart </span>AS</span> (</span></span> -- Seed: start from the CEO</span></span> SELECT</span> id, </span>name</span>, manager_id, </span>1</span> AS</span> depth</span></span> FROM</span> employees</span></span> WHERE</span> manager_id </span>IS NULL</span></span> </span> UNION ALL</span></span> </span> -- Recursive term: find direct reports</span></span> SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span>, </span>oc</span>.</span>depth</span> +</span> 1</span></span> FROM</span> employees e</span></span> JOIN</span> org_chart oc </span>ON</span> e</span>.</span>manager_id</span> =</span> oc</span>.</span>id</span></span> )</span></span> SELECT * FROM</span> org_chart </span>ORDER BY</span> depth, </span>name</span>;</span></span></code></pre> id \| name \| manager_id \| depth</span></span> ----+---------+------------+-------</span></span> 1 \| Alice \| \| 1</span></span> 2 \| Bob \| 1 \| 2</span></span> 3 \| Charlie \| 1 \| 2</span></span> 4 \| Diana \| 2 \| 3</span></span> 5 \| Eve \| 2 \| 3</span></span> 6 \| Frank \| 3 \| 3</span></span> 7 \| Grace \| 3 \| 3</span></span> 8 \| Hank \| 6 \| 4</span></span> 9 \| Ivy \| 6 \| 4</span></span> (9 rows)</span></span></code></pre>UNION vs UNION ALL</a> </h3> The choice between UNION</code> and UNION ALL</code> in recursive CTEs matters more than in regular queries.</p> UNION ALL</code> keeps all rows, including duplicates. This is faster but dangerous: if your graph has cycles, the recursion never terminates. PostgreSQL will run until you cancel the query or memory is exhausted.</p> UNION</code> deduplicates at each iteration. This prevents infinite loops in cyclic graphs but adds the cost of hashing and comparing rows at every step.</p> PostgreSQL 14 added SQL-standard SEARCH</code> and CYCLE</code> clauses that replace the manual patterns for controlling traversal order (breadth-first vs. depth-first) and detecting cycles. SEARCH BREADTH FIRST BY</code> / SEARCH DEPTH FIRST BY</code> control the order, while CYCLE</code> automatically detects and marks cycles, which is far cleaner than the old pattern of accumulating an array of visited IDs.</p> For pure hierarchical data (trees without cycles), consider the ltree</code> extension as an alternative to recursive CTEs. It stores the full path as a label tree and supports efficient ancestor/descendant queries with GiST indexes. The trade-off is denormalized storage vs. on-the-fly recursion.</p> </div> The exotic edge cases</a> </h2> Partition pruning lost</a> </h3> When you materialize a CTE over a partitioned table, partition pruning cannot happen on the CTE scan side. The materialized result is a flat tuplestore disconnected from the partition metadata.</p> -- assume orders is range-partitioned by created_at</span></span> WITH</span> recent </span>AS</span> MATERIALIZED (</span></span> SELECT * FROM</span> orders</span></span> )</span></span> SELECT * FROM</span> recent </span>WHERE</span> created_at </span>></span> '2025-06-01'</span>;</span></span></code></pre> The created_at > '2025-06-01'</code> predicate is applied after</em> materialization. All partitions are scanned to build the CTE, even though only one or two would be needed. Use NOT MATERIALIZED</code> (or simply let the planner inline it) to preserve partition pruning.</p> Prepared statements and plan caching</a> </h3> PostgreSQL generates custom plans for the first 5 executions of a prepared statement. After that, it may switch to a generic plan. CTE inlining decisions can differ between custom and generic plans, because generic plans don't know the actual parameter values.</p> This means a CTE that gets inlined during your first 5 calls might start materializing on the 6th or vice versa. If you see sudden plan changes with prepared statements, check whether the CTE inlining behaviour has shifted.</p> work_mem spilling</a> </h3> Materialized CTEs store their results in memory, bounded by work_mem</code>. When the result set exceeds this limit, it silently spills to disk as a temporary file. This is not an error - it just gets slower.</p> Monitor with log_temp_files = 0</code> (logs all temp files) or check EXPLAIN (ANALYZE, BUFFERS)</code> for temp read/write counts.</p> PostgreSQL 18: EXPLAIN now shows memory/disk usage</strong> Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> reports memory and disk usage for Material nodes, including CTE materialization. You can see exactly how much memory a materialized CTE consumed and whether it spilled to disk.</p> </div> CTE and security barrier views</a> </h3> A `security_barrier` view is a "black box" that forces the database to fully resolve the view's internal logic before any outer filters are applied.</div> Security-barrier views already prevent subquery flattening as a security measure (to stop user-defined functions from seeing rows they shouldn't). When you combine a security-barrier view with a CTE, you compound the optimisation barriers. The planner can neither inline the view nor the CTE. If performance matters in this scenario, consider materializing the security-sensitive filtering into a temporary table first. CTE vs. subquery vs. temporary table</a> </h2> CTE, referenced once</strong> the planner inlines it on PG 12+, so it doesn't affect the execution plan. This is the default choice for breaking up complex queries.</p> CTE, referenced multiple times (small result)</strong> represents acceptable cost. Materialization means the subquery runs once. For aggregations or filtered subsets that produce a few hundred rows, the overhead is minimal.</p> As Henrietta Dombrovskaya</a> emphasizes, "The best temporary table is the one you didn't create". Always exhaust your indexing and query-rewriting options before reaching for a temp table, as the DDL overhead often outweighs the execution gains.</div> CTE, referenced multiple times (large result) ss case when you might consider a temporary table instead. A materialized CTE has no indexes and no statistics. A temporary table can have both, and as covered in [Introduction to Buffers](/posts/introduction-to-buffers/), temporary tables use local buffers with simpler locking and no WAL overhead. If you're joining against 100k+ rows from a CTE, create a temp table, add an index, and `ANALYZE` it. Data-modifying operations</strong> with writable CTE. No alternative gives you the atomic, single-statement behavior.</p> Recursive CTEs</strong>. There's no alternative in pure SQL.</p> Large intermediate results needing indexes/statistics</strong>. If you really need them use temporary table. As discussed in Reading Buffer statistics</a>, temporary tables offer full planner support - indexes, statistics, and buffer management - that materialized CTEs simply don't have.</p> Scenario</th> Recommendation</th></tr></thead> Readability, single reference</td> CTE (inlined, free)</td></tr> Compute once, use many times (small)</td> CTE (materialized)</td></tr> Compute once, use many times (large)</td> Temporary table</td></tr> Atomic data modification</td> Writable CTE</td></tr> Hierarchy / graph traversal</td> Recursive CTE</td></tr> Need indexes on intermediate data</td> Temporary table</td></tr> </tbody></table> The PG 18 state of affairs</a> </h2> PostgreSQL 18 continues to refine CTE handling without any revolutionary changes:</p> EXPLAIN shows memory/disk usage</strong> for CTE materialization nodes. You can finally see whether your CTE fit in work_mem</code> or spilled to disk.</li> Better query plans for CTEs on the same table.</strong> The planner is smarter about eliminating redundant scans when multiple CTEs reference the same underlying table.</li> Data-modifying CTE fix for updatable views with rules.</strong> An edge case where writable CTEs interacted incorrectly with views defined using rules has been resolved.</li> CTE inlining is mature.</strong> The core inlining logic hasn't changed since PG 12. What has improved is everything around it - better statistics propagation (PG 17), better materialisation diagnostics (PG 18), better cost estimation.</li> </ul> CTEs are a good tool. Just know when you're holding the sharp end.</p> The hidden cost of PostgreSQL arrays 2026-01-12T20:50:00+00:00 Starting with arrays in PostgreSQL is as simple as declaring a column as integer[]</code>, inserting some values, and you are done.</p> Or building the array on the fly.</p> SELECT</span> '{1,2,3}'</span>::</span>int</span>[];</span></span> SELECT array</span>[1,2,3];</span></span></code></pre> int4</span></span> ---------</span></span> {1,2,3}</span></span> (1 row)</span></span> </span> array</span></span> ---------</span></span> {1,2,3}</span></span> (1 row)</span></span></code></pre> The official documentation</a> provides a good introduction. But beneath this straightforward interface lies a set of more complex properties than most of us realise. Arrays in PostgreSQL are not just "lists" in a field. They have their own memory management strategy, their own index logic, and a lot of edge-case scenarios.</p> As it goes with boringSQL</strong> deep-dives, this article will explore the corners of array functionality that might break your production.</p> The document model temptation</a> </h2> Wait? Are we going to talk about JSONB arrays? Not at all. The whole concept of arrays in RDBMSs is actually document storage in disguise</strong>.</p> In database design, locality ensures faster retrieval times by keeping related data close on physical storage.</span>Whether you use a distinct integer[]</code> type or a JSON list [1, 2, 3]</code>, you are making the exact same architectural decision: you are prioritising locality over normalisation</strong>.</p> When you store tag_ids</code> in an array, you are embedding related data directly into a row - just like a NoSQL database might embed subdocuments. This is not inherently wrong. Document databases exist for good reasons: they eliminate joins, simplify reads, and map naturally to application objects.</p> But PostgreSQL is a relational database. It was designed around the relational model, where:</p> foreign keys</strong> enforce referential integrity</a></li> joins</strong> connect normalised tables</li> updates</strong> modify individual rows, not entire lists</li> </ul> Arrays give you document-model convenience, but you lose relational promises. There are no foreign keys and no ON DELETE referential_action</code> (like CASCADE) for array elements. If you delete a tags</code> entry, the orphaned ID will remain in your array forever.</p> The rule of thumb</strong> is that if you find yourself in need of referential integrity - you most likely want a link table, not an array. Arrays are for data that shares the same lifecycle as the parent row. Not for relationships spanning across different tables.</p> A practical example is the author of a blog post (one author might write multiple other posts), whereas a whitelist of IP addresses for a service account is only applicable to the given entity.</p> With JSONB being so flexible, you might wonder why we still bother with quirkier native types. The answer lies in the 'boring' part of the database: predictability and efficiency. An integer[]</code> column guarantees that every element is an integer — similar to how enums</a> enforce a fixed set of allowed values.</p> Arrays are also more storage-efficient for primitives because they don't carry the metadata overhead of JSON objects.</p> The syntax gotchas</a> </h2> This post assumes basic knowledge of arrays. We won't cover the basics.</p> Arrays don't have to start at 1</a> </h3> By default, SQL arrays start at 1. And seemingly, there is nothing wrong with iterating through them in a fashion similar to:</p> FOR</span> i </span>IN</span> 1</span> .. array_length(fruits, </span>1</span>) </span>LOOP</span></span> RAISE NOTICE </span>'Index % contains: %'</span>, i, fruits[i];</span></span> END LOOP</span>;</span></span></code></pre> that is until you find an array with the arbitrary bounds. Which PostgreSQL allows.</p> SELECT</span> '[-5:-3]={10,20,30}'</span>::</span>int</span>[];</span></span></code></pre> To make sure you iterate correctly through any given array, always use array_lower()</code> and array_upper()</code> in PL/pgSQL</p> SELECT</span> array_lower(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> array_lower</span></span> -------------</span></span> -5</span></span> (1 row)</span></span></code></pre> or generate_subscripts()</code> in SQL.</p> SELECT</span> generate_subscripts(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> generate_subscripts</span></span> ---------------------</span></span> -5</span></span> -4</span></span> -3</span></span> (3 rows)</span></span></code></pre>Missing dimensions</a> </h3> When creating a table, you might expect strict typing. That's true for everything — except the array dimensions. You might think integer[][]</code> enforces a 2D matrix. Except it does not. The []</code> syntax is effectively syntactic sugar. PostgreSQL does not enforce the number of dimensions of sub-arrays at the schema level at all by default.</p> CREATE TABLE</span> dimension_test</span> (</span></span> matrix </span>integer</span>[][] </span></span> );</span></span> </span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3,4}}'</span>);</span></span> </span> -- this is not going to fail</span></span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{1,2,3}'</span>);</span></span> </span> -- 3D matrix works too</span></span> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{{1,2},{3,4}}, {{5,6},{7,8}}}'</span>);</span></span></code></pre> If you want to enforce a specific array dimension, you cannot rely on the type definition. Instead, you must use a CHECK</code> constraint.</p> CREATE TABLE</span> strict_matrix</span> (</span></span> -- dims: ensure it's 2-Dimensional</span></span> -- array_length: make sure it's exactly 3x3</span></span> board </span>integer</span>[]</span> CHECK</span> (array_ndims(board) </span>=</span> 2</span> AND</span> array_length(board, </span>1</span>) </span>=</span> 3</span>)</span></span> );</span></span></code></pre>INSERT INTO strict_matrix VALUES (</span></span> ARRAY[</span></span> [0, 1, 0],</span></span> [1, 1, 0],</span></span> [0, 0, 1]</span></span> ]</span></span> );</span></span></code></pre> The only exception is that PostgreSQL enforces uniformity of arrays on every nesting level. This means it rejects sub-arrays with different sizes.</p> INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span> ERROR: malformed </span>array</span> literal: </span>"{{1,2}, {3}}"</span></span> LINE</span> 1</span>: </span>INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span> ^</span></span> DETAIL: Multidimensional arrays must have sub</span>-</span>arrays </span>with</span> matching dimensions.</span></span></code></pre>Slicing arrays</a> </h3> When accessing array values, it's important to consider that the syntax [1]</code> and [1:1]</code> are different. While the first one is an accessor, the second one acts like a constructor.</p> select</span> matrix[1][1]</span> from</span> dimension_test ;</span></span> matrix</span></span> --------</span></span> 1</span></span> (</span>1</span> row</span>)</span></span></code></pre> When slicing an array, even if the slice is a single-element, it will be returned as a single-element array, not a scalar value.</p> select</span> matrix[1:1][1:1], matrix[1][1:1], matrix[1:1][1]</span> from</span> dimension_test ;</span></span> matrix \| matrix \| matrix</span></span> --------+--------+--------</span></span> {{</span>1</span>}} \| {{</span>1</span>}} \| {{</span>1</span>}}</span></span> (</span>1</span> row</span>)</span></span></code></pre>The ugly</a> </h3> Accessing array values has a forgiving behaviour, making it more difficult to find underlying bugs:</p> -- out-of-bound access returns NULL</span></span> SELECT</span> (</span>ARRAY</span>[1,2,3])[10];</span></span> array</span></span> --------</span></span> (</span>null</span>)</span></span> (</span>1</span> row</span>)</span></span></code></pre>-- out of bounds slicing returns an empty array</span></span> SELECT</span> (</span>ARRAY</span>[1,2,3])[5:10];</span></span> array</span></span> -------</span></span> {}</span></span> (</span>1</span> row</span>)</span></span></code></pre> But the single most confusing aspect that might trip you up coming from other programming languages is the fact that PostgreSQL treats multi-dimensional arrays as a single matrix, not an array of arrays.</p> -- wrong dimensionality </span></span> SELECT</span> (</span>ARRAY</span>[[1,2],[3,4]])[1];</span></span> array</span></span> --------</span></span> (</span>null</span>)</span></span> (</span>1</span> row</span>) </span></span></code></pre> While in other languages you expect {1,2}</code> as a result, PostgreSQL can't give it to you. It tries to return the first cell and fails (because the index is not complete).</p> And to paraphrase Fletcher's "Double Bubble" analogy (which also went very wrong — extra points for getting the reference), you can't fix this by using slice notation.</p> select</span> (</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1];</span></span> int4</span></span> ---------</span></span> {{</span>1</span>,</span>2</span>}}</span></span> (</span>1</span> row</span>)</span></span></code></pre> The only way to solve this puzzle is to unnest</code> the slice and re-aggregate the results.</p> SELECT</span> array_agg(val) </span>FROM</span> unnest((</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1]) val;</span></span> array_agg</span></span> -----------</span></span> {</span>1</span>,</span>2</span>}</span></span> (</span>1</span> row</span>)</span></span></code></pre> Just be aware that `array_agg` does not guarantee the order of aggregated elements unless you use an `ORDER BY` clause. While it usually works with simple `unnest` queries, relying on implicit ordering can be risky. </div> The other alternative is to cast it to JSONB and back. Not pretty no matter how you look at it. If you need to work with complex multi-dimensional structures where each sub-array has independent meaning, just use JSONB. It will do exactly what you expect.</p> Indexing arrays</a> </h2> While you can use a B-tree index on an array column, it won't help you unless you are looking for whole-array equality and sorting an array by dictionary rules — and even on regular columns, PostgreSQL can ignore your indexes</a> when it decides a sequential scan is cheaper.</p> -- which is bigger? B is correct</span></span> -- A: {1, 1000, 1000}</span></span> -- B: {2, 0}</span></span></code></pre> Rendering B-tree indexes useless for any real-world index operations.</p> When working with arrays, you actually need GIN</strong> (Generalized Inverted Index). If a B-tree index is a phone book, GIN is an index at the back of the book. To query one or more elements, you need to find all possible locations and then intersect them to find the locations that match.</p> CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags);</span></span></code></pre> GIN indexes are designed for set operations, making presence the key feature, while ignoring order. Here are operators that GIN provides for arrays:</p> Containment</strong> @></code> - matches rows that include ALL of the selected items.</p> -- match ALL tags</span></span> tags @</span>></span> '{urgent, bug}'</span></span></code></pre> Overlap</strong> &&</code> - does the row include ANY of the selected items.</p> -- match bug or feature</span></span> tags && </span>'{bug, feature}'</span></span></code></pre> There's also <@</code> and =</code> (equality), but they should be easy to understand.</p> Two sides of ANY</code></a> </h3> This is where it gets interesting. While you might often hear (myself included) that you shouldn't use dynamic SQL for IN</code> lists and should use ANY</code> instead, there are some dangers you need to be aware of.</p> The ANY</code> operator behaves very differently depending on which side of the comparison the array sits.</p> The advice to use ANY</code> holds true when you are passing lists into the database</strong>. Instead of generating a query with 100 distinct parameters (WHERE id IN ($1, $2, ... $100)</code>), which bloats your query cache and forces hard-parses, you should pass a single array parameter.</p> -- good: one parameter, one query plan</span></span> SELECT * FROM</span> users </span>WHERE</span> id </span>=</span> ANY($</span>1</span>::</span>int</span>[]);</span></span></code></pre> The trap is assuming this syntax is equally efficient when querying array columns. It is not. If you use ANY</code> to check if a value exists inside a table column, you are effectively asking the database to loop.</p> -- bad: GIN does not support ANY; turning this into a seq scan</span></span> SELECT * FROM</span> tickets </span>WHERE</span> 'feature'</span> =</span> ANY(tags);</span></span></code></pre> When you write WHERE 'feature' = ANY(tags)</code>, you are not actually using an array operator. What you wrote is a scalar integer equality operator =</code> applied inside a loop construct. Since the scalar operator =</code> is not part of array_ops</code>, the planner assumes the index cannot help and falls back to a sequential scan.</p> The correct way to rewrite the query is:</p> -- good again</span></span> SELECT * FROM</span> tickets </span>WHERE</span> tags @</span>> ARRAY</span>['feature'];</span></span></code></pre>Fast updates and the trade-off</a> </h3> Because a GIN index is built to work with sets, it is expensive to maintain. With a B-tree index, one row equals one index entry. In a GIN index, one row equals N index entries, where N is the number of elements in your array.</p> This leads to write multiplication. To prevent this, PostgreSQL defaults to using a "fast update" mechanism. This is a strategy where new entries are added to a pending list (an unsorted temporary buffer) and only merged into the main index structure later (during VACUUM).</p> Given the unsorted nature of a GIN index, it is an exception to VACUUM is a Lie</a>, as VACUUM actually does perform structural maintenance here.</span>While this makes INSERT operations manageable, it can slow down SELECTs. Every time you query the index, PostgreSQL must scan the organised main index plus the entire messy pending list. If that list grows large, your query performance might degrade.</p> If you operate a read-heavy workflow with infrequent writes, you should disable this to guarantee consistent reading performance.</p> CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags) </span>WITH</span> (fastupdate </span>= off</span>);</span></span></code></pre>Storage and modification</a> </h2> Now it's time to return to the document model</a>. In PostgreSQL, rows are immutable (MVCC); there is no such thing as an "in-place update", and arrays are stored as atomic values. The consequence: to modify a single element of an array, PostgreSQL must copy and rewrite the entire row</strong>.</p> UPDATE</span> user_activity</span></span> SET</span> event_ids </span>=</span> event_ids </span>\|\|</span> 10001</span></span> WHERE</span> user_id </span>=</span> 50</span>;</span></span></code></pre> Every single append rewrites the entire array, which in effect results in a rewrite of the entire row.</p> TOASTed</a> </h3> When any array grows large enough (> 2 KB — see below), PostgreSQL moves it automatically to a separate storage area using TOAST</a>. While this keeps the data row lean, it turns array updates into a severe performance bottleneck.</p> The difference this introduces is subtle but comes with a big impact. While a standard MVCC update simply copies the row version on the main heap, updating a TOASTed array forces PostgreSQL to fetch all external chunks, decompress the entire object into memory, apply the change, and then recompress and write the new full-size blob back to the TOAST table. This turns a simple modification into a CPU and I/O-intensive operation that rewrites the entire dataset rather than just the delta.</p> The threshold is derived from TOAST_TUPLES_PER_PAGE</code> (default: 4), ensuring 4 tuples fit on a page.</br></br>Threshold: ~2 KB</span>Where does the 2 KB value come from? The TOAST threshold is calculated to ensure at least four tuples can fit on a single 8 KB heap page</a>. PostgreSQL uses this to balance efficiency against the overhead of TOAST indirection.</p> The compressions</a> </h3> Before version 14, PostgreSQL relied on pglz</code> — an algorithm prioritising compression ratio over speed. This made the "decompress-modify-compress" cycle of TOAST painful.</p> PostgreSQL 14 introduced LZ4 as an alternative:</p> ALTER TABLE</span> articles </span>ALTER</span> COLUMN tags </span>SET COMPRESSION</span> lz4;</span></span></code></pre> LZ4 is significantly faster for both compression and decompression, with only slightly lower compression ratios. If you are working with large arrays, switching to LZ4 is one of the easiest ways to reduce the CPU penalty of TOAST.</p> When a large array might make sense</a> </h3> You might have gained the impression that arrays are bad. When evaluating the use of arrays, the real question isn't how big the array is</strong> but rather how often do you modify it</strong>? An array of 10,000 elements that you write once and is read-only for the rest of its lifecycle is a completely valid use case. An array of 50 elements that you append to on every incoming request is the real villain here.</p> If you combine this with compression, you might get an interesting mix.</p> DROP TABLE IF EXISTS</span> compression_test;</span></span> CREATE TABLE</span> compression_test</span> (</span></span> id </span>serial PRIMARY KEY</span>,</span></span> compressed_floats float4[], </span></span> raw_floats float4[]</span></span> );</span></span> </span> -- do not compress raw_floats</span></span> ALTER TABLE</span> compression_test </span>ALTER</span> COLUMN raw_floats </span>SET</span> STORAGE </span>EXTERNAL</span>;</span></span> </span> -- insert semi-random data with low cardinality</span></span> INSERT INTO</span> compression_test (compressed_floats, raw_floats)</span></span> SELECT</span> semi_random_arr, semi_random_arr</span></span> FROM</span> (</span></span> SELECT ARRAY</span>(</span></span> SELECT</span> floor</span>(random()</span> </span> 50</span>)::float4 </span></span> FROM</span> generate_series</span>(</span>1</span>, </span>10000</span>)</span></span> ) </span>as</span> semi_random_arr</span></span> ) </span>as</span> generator;</span></span> </span> SELECT</span> </span></span> 'Compressed (EXTENDED)'</span> as</span> strategy,</span></span> pg_size_pretty(pg_column_size(compressed_floats)::</span>bigint</span>) </span>as</span> size_on_disk</span></span> FROM</span> compression_test</span></span> UNION ALL</span></span> SELECT</span> </span></span> 'Raw (EXTERNAL)'</span>,</span></span> pg_size_pretty(pg_column_size(raw_floats)::</span>bigint</span>)</span></span> FROM</span> compression_test;</span></span></code></pre> strategy \| size_on_disk</span></span> -----------------------+--------------</span></span> Compressed (EXTENDED) \| 15 kB</span></span> Raw (EXTERNAL) \| 39 kB</span></span> (2 rows) </span></span></code></pre>Bulk loading with arrays</a> </h2> Up until now, it might seem that arrays don't actually bring many benefits. While they can have rough edges around storage, they are incredibly useful for transport.</p> For inserting 5,000 rows, unnest</code> beats both an application-side loop and a massive VALUES (...), (...)</code> string.</p> INSERT INTO</span> measurements (sensor_id, </span>value</span>, captured_at)</span></span> SELECT FROM</span> unnest(</span></span> $</span>1</span>::</span>int</span>[], </span>-- array of sensor IDs</span></span> $</span>2</span>::</span>float</span>[], </span>-- array of values</span></span> $</span>3</span>::</span>timestamptz</span>[]</span> -- array of timestamps</span></span> );</span></span></code></pre> All that is needed is one network round trip, and one query to parse and plan. PostgreSQL handles the arrays row by row internally for you. This works both for UPSERTs and MERGE</a>.</p> The cases for special arrays</a> </h2> Standard PostgreSQL arrays are polymorphic types (anyarray</code>). This provides a powerful feature that allows a single function definition to operate on many different data types. They have to handle integers, strings, timestamps, and custom types equally well. But if you have specific data types, you can unlock significant performance gains by using specialised extensions.</p> The intarray</code> extension</a> </h3> If you are dealing exclusively with 4-byte integers (int4</code>/integer</code>), the built-in array operations are leaving performance on the table. The intarray</a> extension provides specialised functions and index operators that are significantly faster than the generic implementation.</p> To use it, you must explicitly enable it:</p> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> intarray;</span></span></code></pre> The difference in developer ergonomics is immediate. To sort an array in standard SQL, you are forced to unnest, order, and re-aggregate. With intarray, you get native functions like sort() and uniq().</p> -- standard arrays</span></span> SELECT</span> array_agg(val </span>ORDER BY</span> val)</span></span> FROM</span> unnest(</span>'{3, 1, 2}'</span>::</span>int</span>[]) val;</span></span> </span> -- intarray</span></span> SELECT</span> sort(</span>'{3, 1, 2}'</span>::</span>int</span>[]);</span></span></code></pre> Beyond raw management functions, it introduces a specialised query syntax that simplifies complex boolean logic. Instead of chaining multiple overlap (&&</code>) and containment (@></code>) checks, you can express your requirements in a single "query string" using the @@</code> operator.</p> -- standard arrays</span></span> SELECT * FROM</span> staff</span></span> WHERE</span> available_days @</span>></span> '{1}'</span> -- must include Mon</span></span> AND</span> available_days && </span>'{6, 7}'</span> -- must include Sat OR Sun</span></span> AND NOT</span> (available_days @</span>></span> '{2}'</span>); </span>-- must NOT include Tue</span></span> </span> -- intarray</span></span> SELECT * FROM</span> staff</span></span> WHERE</span> available_days @@ </span>'1 & (6 \| 7) & !2'</span>;</span></span></code></pre> The only catch is the type restriction. intarray</code> is strictly limited to signed 32-bit integers. If your values exceed 2 billion, you are back where you started.</p> AI with pgvector</code></a> </h3> You cannot talk about arrays in 2026 without mentioning pgvector</a>. While it markets itself as a "vector store", internally it is simply an array of floats with a different mathematical focus.</p> Standard arrays are binary</strong>: they care about Exact Matches (Overlap &&</code>, Containment @></code>). Vectors are all about fuzzy distance</strong> (Cosine <=></code>, Euclidean <-></code>).</p> If you are building search or recommendation features, pgvector allows you to treat your array column not as a list of "facts" (tag A, tag B), but as coordinates in semantic space.</p> Nevertheless, the architectural decision is exactly the same as using a standard array: you are trading strict structure for convenience. Since there is no way to "join" two rows based on how similar they are, you store the vector directly on the row. You accept a larger table size in exchange for the ability to ask, "What is close to this?".</p> PS: Thanks to Matthias Feist for inspiring this article.</p> RegreSQL: Regression Testing for PostgreSQL Queries 2025-11-13T22:47:00+00:00 TL;DR</strong> - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production.</em></p> As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification.</p> For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</em></a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries.</p> I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level.</p> Introduction</a> </h2> The RegreSQL</a></strong> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later).</p> But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere?</p> This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations.</p> The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management.</p> Basic regression testing</a> </h2> Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like</p> $ regresql text</span></span> Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> </span> Results: 8 passed, 0 failed, 8 skipped (0.00s)</span></span></code></pre> In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for .sql</code> files and attempts to run all queries against the configured PostgreSQL connection.</p> The individual files can contain either single or multiple sql queries. Like following example</p> -- name: top-artists-by-album</span></span> -- Get the list of the N artists with the most albums</span></span> SELECT</span></span> artist</span>.</span>name</span>,</span></span> count</span>(</span></span>) </span>AS</span> albums</span></span> FROM</span></span> artist</span></span> LEFT JOIN</span> album </span>USING</span> (artist_id)</span></span> GROUP BY</span></span> artist</span>.</span>name</span></span> ORDER BY</span></span> albums </span>DESC</span></span> LIMIT</span> :n;</span></span></code></pre> The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now).</p> This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines.</p> Performance regression testing</a> </h2> Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts.</p> RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated</p> $ regresql baseline</span></span> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> Creating baselines directory: regresql/baselines</span></span> Creating directory 'regresql/baselines'</span></span> </span> Creating baselines for queries:</span></span> </span> ./</span></span> Created baseline: album-by-artist_list-albums-by-artist.1.json</span></span> Created baseline: album-by-artist_list-albums-by-artist.2.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.1.json</span></span> Created baseline: album-tracks_list-tracks-by-albumid.2.json</span></span> Created baseline: artist_top-artists-by-album.1.json</span></span> Created baseline: genre-topn_genre-top-n.top-1.json</span></span> Created baseline: genre-topn_genre-top-n.top-3.json</span></span> Created baseline: genre-tracks_tracks-by-genre.json</span></span> </span> Baselines have been created successfully!</span></span> Baseline files are stored in: regresql/baselines</span></span></code></pre> the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place.</p> To demonstrate this, let's review the test output with the baselines.</p> Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span> </span> Running regression tests...</span></span> </span> ✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span> ✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span> ⚠️ Sequential scan detected on table 'artist'</span></span> Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> ✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span> </span> ✓ artist_top-artists-by-album.1.json (0.00s)</span></span> ✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: album, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> ✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span> ✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, artist</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> ✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: artist, genre</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> ⚠️ Multiple sort operations detected (2 sorts)</span></span> Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span> ⚠️ Nested loop join with sequential scan detected</span></span> Suggestion: Add index on join column to avoid repeated sequential scans</span></span> </span> ✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span> ✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s)</span></span> ⚠️ Multiple sequential scans detected on tables: genre, track</span></span> Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span> </span> Results: 16 passed (0.00s)</span></span></code></pre> As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready".</p> In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below.</p> -- name: query_name</span></span> -- metadata: key1=value1, key2=value2</span></span> SELECT</span> ...;</span></span></code></pre> At this point RegreSQL recognizes</p> notest</code> to skip the query testing altogether (not just cost tracking)</li> nobaseline</code> to skip cost tracking</li> noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li> and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li> </ul> -- name: query_name</span></span> -- regresql: notest, nobaseline</span></span> -- regresql: noseqscanwarn</span></span> -- regresql: difffloattolerance:0.25</span></span> -- query that can vary in cost by 20% without being considered a failure</span></span> SELECT</span> ...;</span></span></code></pre>ORM enters the room</a> </h2> ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data:</p> orders</span> =</span> (</span></span> session.query(Order)</span></span> .filter(Order.user_id</span> ==</span> user_id)</span></span> .options(</span></span> joinedload(Order.user),</span></span> joinedload(Order.shipping_address),</span></span> selectinload(Order.items)</span> # NEW: Load order items</span></span> )</span></span> .all()</span></span> )</span></span></code></pre> That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan.</p> RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system:</p> @event.listens_for</span>(engine,</span> "before_cursor_execute"</span>)</span></span> def</span> capture_sql</span>(conn, cursor, statement,</span> </span>args):</span></span> captured_queries.append(statement)</span></span></code></pre> Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development.</p> Test Data Management</a> </h2> Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky.</p> Traditional approach to create test data might involve</p> Database dumps</strong> become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li> SQL scripts</strong> might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li> Factories in application code</strong> might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li> Shared test database</strong> is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li> </ul> What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000).</p> This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale.</p> RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example</p> fixture</span>:</span> basic_users</span></span> description</span>:</span> a handful of test users</span></span> cleanup</span>:</span> rollback</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> alice@example.com</span></span> name</span>:</span> Alice Anderson</span></span> created_at</span>:</span> 2024-01-15</span></span> -</span> id</span>:</span> 2</span></span> email</span>:</span> bob@example.com</span></span> name</span>:</span> Bob Builder</span></span> created_at</span>:</span> 2024-02-20</span></span></code></pre> To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture</p> fixtures</span>:</span></span> -</span> basic_users</span></span> </span> "1"</span>:</span></span> email</span>:</span> alice@example.com</span></span> </span> "2"</span>:</span></span> email</span>:</span> bob@example.com</span></span></code></pre> And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators</strong> including</p> sequences, random integer, decimal, string, uuid, email and name generators</li> date_between for generating random timestamps within a range</li> foreign key references to be able to reuse data from other table's fixtures</li> range to select value from predefined sources</li> Go template support</li> </ul> fixture</span>:</span> realistic_orders</span></span> generate</span>:</span></span> -</span> table</span>:</span> customers</span></span> count</span>:</span> 1000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> email</span>:</span></span> generator</span>:</span> email</span></span> domain</span>:</span> shop.example.com</span></span> name</span>:</span></span> generator</span>:</span> name</span></span> type</span>:</span> full</span></span> created_at</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span> </span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 5000</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> customer_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1000</span></span> amount</span>:</span></span> generator</span>:</span> decimal</span></span> min</span>:</span> 10.00</span></span> max</span>:</span> 999.99</span></span> precision</span>:</span> 2</span></span> order_date</span>:</span></span> generator</span>:</span> date_between</span></span> start</span>:</span> "2023-01-01"</span></span> end</span>:</span> "2024-12-31"</span></span></code></pre> This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like.</p> The fixtures are also stackable</strong> and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse.</p> fixture</span>:</span> orders_with_shipping</span></span> depends_on</span>:</span></span> -</span> basic_users</span></span> </span> data</span>:</span></span> -</span> table</span>:</span> orders</span></span> rows</span>:</span></span> -</span> id</span>:</span> 101</span></span> user_id</span>:</span> 1</span> # References Alice from basic_users</span></span> total</span>:</span> 99.99</span></span> status</span>:</span> shipped</span></span></code></pre> Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation.</p> fixture</span>:</span> mixed_setup</span></span> description</span>:</span> Combine SQL with YAML and generated data</span></span> cleanup</span>:</span> rollback</span></span> </span> # SQL executes first (either as file or inline)</span></span> sql</span>:</span></span> -</span> file</span>:</span> sql/setup_schema.sql</span></span> -</span> inline</span>:</span> "INSERT INTO config (key, value) VALUES ('version', '1.0');"</span></span> </span> # followed YAML data</span></span> data</span>:</span></span> -</span> table</span>:</span> users</span></span> rows</span>:</span></span> -</span> id</span>:</span> 1</span></span> email</span>:</span> admin@example.com</span></span> </span> # and finally generated data</span></span> generate</span>:</span></span> -</span> table</span>:</span> orders</span></span> count</span>:</span> 100</span></span> columns</span>:</span></span> id</span>:</span></span> generator</span>:</span> sequence</span></span> start</span>:</span> 1</span></span> user_id</span>:</span></span> generator</span>:</span> int</span></span> min</span>:</span> 1</span></span> max</span>:</span> 1</span></span></code></pre> RegreSQL provides commands to inspect and validate your fixtures</p> # List all available fixtures</span></span> regresql</span> fixtures list</span></span> </span> # Show fixture details and dependencies</span></span> regresql</span> fixtures show realistic_orders</span></span> </span> # Validate fixture definitions</span></span> regresql</span> fixtures validate</span></span> </span> # Show dependency graph</span></span> regresql</span> fixtures deps</span></span> </span> # Apply fixture manually (for debugging)</span></span> regresql</span> fixtures apply basic_users</span></span></code></pre> The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves.</p> (EDIT 2025-11-20)</strong> The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use</p> Fixed generated fixtures re-used between tests</li> Use RegreSQL to test fixtures before/after migration</li> </ul> I'm working on next release of RegreSQL to address DX of fixtures usage.</p> RegreSQL future</a> </h2> Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL</strong> brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development.</p> At the same time RegreSQL</strong> is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable.</p> RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>.</p> Beyond Start and End: PostgreSQL Range Types 2025-11-02T21:40:00+00:00 One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them.</p> Bug Not Invented Here</a> </h2> But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets.</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> </span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> -- define the hold period explicitly</span></span> hold_started_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP,</span></span> hold_expires_at </span>TIMESTAMPTZ NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span> </span> CREATE INDEX</span> seat_holds_on_seat_id</span> ON</span> seat_holds(seat_id);</span></span> CREATE INDEX</span> seat_holds_on_expiration</span> ON</span> seat_holds(hold_expires_at);</span></span></code></pre> While the table design looks perfectly reasonable, it has one serious flaw</strong> - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee.</p> If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain.</p> Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention</strong> situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.).</p> And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level.</p> -- needed to add GiST support to equality of integers (for seat_id)</span></span> CREATE</span> EXTENSION </span>IF NOT EXISTS</span> btree_gist;</span></span> </span> ALTER TABLE</span> seat_holds </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> tsrange(hold_started_at, hold_expires_at) </span>WITH</span> &&</span></span> );</span></span></code></pre> Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - the database now rejects the overlap outright instead of merely making it rare. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load.</p> But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box.</p> Introducing the Data Range Types</a> </h2> You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in?</p> This brings us to the core of the matter: PostgreSQL's native Range Types</strong>.</p> PostgreSQL provides a set of built-in range types, all following the pattern of type and range:</p> int4range</code> for integer</li> int8range</code> for bigint</li> numrange</code> for numeric</li> tsrange</code> for timestamp without time zone</li> tstzrange</code> for timestamp with time zone (which we briefly saw above)</li> daterange</code> for date</li> </ul> And it does not stop there. You can easily define your own custom range types</strong> over any basic data type.</p> First win: cleaner schema</a> </h2> When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code.</p> When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct. The application code no longer needs to manage two separate boundaries.</p> CREATE TABLE</span> seat_holds_native</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span> user_session_id UUID </span>NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span> );</span></span></code></pre> This is the power of a first-class data type</strong>. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly.</p> Second win: Atomicity guaranteed</a> </h2> While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint</strong> as demonstrated previously.</p> ALTER TABLE</span> seat_holds_native </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span> EXCLUDE </span>USING</span> gist (</span></span> seat_id </span>WITH =</span>,</span></span> hold_period </span>WITH</span> &&</span></span> );</span></span></code></pre> This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once:</p> seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li> hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li> </ol> Finally EXCLUDE USING gist</code> is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint. This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.).</p> Range boundaries: A quick math refresher</a> </h2> Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics.</p> PostgreSQL ranges can have four different boundary types:</p> Inclusive boundaries -</strong> [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to".</p> -- [10, 20] includes both 10 and 20</span></span> -- represents 10 ≤ x ≤ 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>);</span></span></code></pre> Exclusive boundaries -</strong> (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality.</p> -- (10, 20) excludes both 10 and 20</span></span> -- represents 10 < x < 20</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'()'</span>);</span></span> </span></code></pre> Mixed boundaries -</strong> [)</code> or (]</code> You can mix and match. The most common and default pattern</strong> in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next".</p> As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods.</p> -- these ranges are adjacent, not overlapping</span></span> -- Week 1</span></span> SELECT</span> '[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Week 2</span></span> SELECT</span> '[2025-11-08, 2025-11-15)'</span>::tstzrange;</span></span></code></pre> With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments.</p> Canonicalization and Range Set Operations</a> </h2> Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges.</p> For discrete</strong> range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper).</p> PostgreSQL uses a canonicalization function</strong> to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing.</p> -- [10, 20] includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span> </span> -- (9, 21) includes integers 10, 11, ..., 20.</span></span> SELECT</span> int4range(</span>9</span>, </span>21</span>, </span>'()'</span>) </span>AS</span> original_range;</span></span> -- Result: [10,21)</span></span></code></pre> Exception are continuous ranges</strong> (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning.</p> The operator toolkit</a> </h2> The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier.</p> Overlap operator - &&</code></strong> As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points.</p> -- find holds active at any point between 10:00 and 11:00</span></span> SELECT FROM</span> seat_holds_native </span>WHERE</span> hold_period && </span>'[2025-12-25 10:00, 2025-12-25 11:00)'</span>::tstzrange;</span></span></code></pre> Contains operator - @></code></strong> For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type).</p> -- find holds that are active at the specific momement</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '2025-11-05 15:00'</span>::</span>timestamptz</span>;</span></span> </span> -- find holds that are active at the specific time range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '[2025-12-25 10:00, 2025-12-25 10:15)'</span>::tstzrange;</span></span></code></pre> Contained By operator - <@</code></strong> In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right.</p> -- find holds that are within November '25</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span><</span>@ </span>'[2025-11-01, 2025-12-01)'</span>::tstzrange;</span></span></code></pre> Strictly Before/After operators - <<</code> and >></code></strong> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries).</p> -- find holds that finished strictly before 10 November</span></span> SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</span></span></code></pre> Boundary Extension operators - &<</code> and &></code></strong> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range).</p> -- find holds that end before or at the same time as reference range ends</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span><</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span> </span> -- find holds that start at or after reference range starts</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span>></span>'[2025-11-08 09:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre> Finally the Adjecent operator - -\|-</code></strong> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict.</p> -- find holds that are immediately adjacent (touching) to given range</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>-</span>\|</span>-</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre>To Infinity and Beyond</a> </h2> Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>.</p> Let's start with infinite bounds</strong>, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time).</p> -- range that never expires (upper bound is infinite)</span></span> SELECT</span> '[2025-11-01 10:00, infinity)'</span>::tstzrange;</span></span> </span> -- range that has always been valid (lower bound is infinite)</span></span> SELECT</span> '[-infinity, 2025-11-01 10:00)'</span>::tstzrange;</span></span> </span> -- range covering all time</span></span> SELECT</span> '[-infinity, infinity)'</span>::tstzrange;</span></span></code></pre> This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription.</p> -- lifetime subscription that never expires</span></span> INSERT INTO</span> subscriptions (user_id, plan, active_period)</span></span> VALUES</span> (</span>42</span>, </span>'lifetime'</span>, </span>'[2025-11-01, infinity)'</span>);</span></span> </span> -- all active subscriptions right now</span></span> SELECT * FROM</span> subscriptions</span></span> WHERE</span> active_period @</span>> NOW</span>();</span></span></code></pre> Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent.</p> At any point you can validate whatever range has infinite bounds:</p> SELECT</span></span> lower_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> lower_is_infinite,</span></span> upper_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> upper_is_infinite;</span></span></code></pre>Understanding NULL vs empty: Schrödinger's Range</a> </h2> Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty.</p> Let's see this in practice:</p> -- NULL range: we don't know what the period is</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>42</span>, </span>'abc-123'</span>, </span>NULL</span>);</span></span> </span> -- empty range: we know the period is explicitly "nothing"</span></span> INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span> VALUES</span> (</span>43</span>, </span>'def-456'</span>, </span>'empty'</span>);</span></span></code></pre> The main difference between them is when it comes to handling.</p> SELECT NULL</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: NULL (not true, not false—we don't know)</span></span> </span> SELECT</span> 'empty'</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span> -- Result: false (we know it doesn't overlap)</span></span></code></pre> And you can check for them in your queries using built-in function isempty</code>.</p> -- check for NULL (like any column)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>IS NULL</span>;</span></span> </span> -- check for empty (special function)</span></span> SELECT * FROM</span> seat_holds_native </span>WHERE</span> isempty(hold_period);</span></span></code></pre> In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail.</p> Practical Integer ranges for Tiered pricing</a> </h2> While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing.</p> CREATE TABLE</span> quantity_discounts</span> (</span></span> discount_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> product_id </span>INTEGER NOT NULL</span>,</span></span> quantity_range INT4RANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- no overlapping tiers</span></span> EXCLUDE </span>USING</span> GIST (</span></span> product_id </span>WITH =</span>,</span></span> quantity_range </span>WITH</span> &&</span></span> )</span></span> );</span></span> </span> INSERT INTO</span> quantity_discounts (product_id, quantity_range, discount_percentage) </span>VALUES</span></span> -- 1-9 units: no discount</span></span> (</span>1</span>, </span>'[1,10)'</span>, </span>0</span>.</span>00</span>),</span></span> -- 10-49 units: 5% off</span></span> (</span>1</span>, </span>'[10,50)'</span>, </span>5</span>.</span>00</span>),</span></span> -- 50-99 units: 10% off</span></span> (</span>1</span>, </span>'[50,100)'</span>, </span>10</span>.</span>00</span>),</span></span> -- 100+ units: 15% off</span></span> (</span>1</span>, </span>'[100,1000)'</span>, </span>15</span>.</span>00</span>);</span></span> </span> -- verify what discount we offer for ordering 75 units?</span></span> SELECT</span> discount_percentage</span></span> FROM</span> quantity_discounts</span></span> WHERE</span> product_id </span>=</span> 1</span></span> AND</span> quantity_range @</span>></span> 75</span>;</span></span> -- Result: 10.00</span></span></code></pre>Making Bad Data Impossible</a> </h2> If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database.</p> CREATE TABLE</span> promotional_campaigns</span> (</span></span> campaign_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span>,</span></span> budget_range NUMRANGE </span>NOT NULL</span>,</span></span> discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span> </span> -- campaigns must be at least 1 days long</span></span> CONSTRAINT</span> campaigns_minimum_duration</span></span> CHECK</span> (</span>upper</span>(active_period) </span>-</span> lower</span>(active_period) </span>>=</span> INTERVAL </span>'1 days'</span>),</span></span> </span> -- budget must be between $1000 and $100000</span></span> CONSTRAINT</span> campaigns_valid_budget</span></span> CHECK</span> (budget_range </span><</span>@ numrange(</span>1000</span>, </span>100000</span>, </span>'[]'</span>)),</span></span> </span> -- active period must not be empty</span></span> CONSTRAINT</span> campaigns_valid_period</span></span> CHECK</span> (</span>NOT</span> isempty(active_period))</span></span> );</span></span></code></pre> While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions).</p> Multiranges: When one range is not enough</a> </h2> Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column.</p> PostgreSQL 14 introduced multirange types for all the built-in range types:</p> int4multirange</code>, int8multirange</code>, nummultirange</code></li> tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li> </ul> The real power of multiranges lies in schema density</strong> and query efficiency</strong>. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas.</p> Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data.</p> CREATE TABLE</span> user_periods_single_range</span> (</span></span> period_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> user_id </span>INTEGER NOT NULL</span>,</span></span> active_period TSTZRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_single_range_gist_idx</span></span> ON</span> user_periods_single_range</span></span> USING</span> gist (active_period);</span></span> </span> -- 20 fragmented periods for user_id 42 (20 rows total)</span></span> INSERT INTO</span> user_periods_single_range (user_id, active_period)</span></span> SELECT</span></span> 42</span>,</span></span> tstzrange(</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> \|\|</span> ' days'</span>)::interval,</span></span> '2025-01-01'</span>::</span>timestamptz +</span> (i </span></span> 2</span> +</span> 1</span> \|\|</span> ' days'</span>)::interval</span></span> )</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>20</span>) </span>AS</span> s(i);</span></span></code></pre> compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data.</p> CREATE TABLE</span> user_periods_multirange</span> (</span></span> user_id </span>INTEGER PRIMARY KEY</span>,</span></span> active_periods TSTZMULTIRANGE </span>NOT NULL</span></span> );</span></span> </span> CREATE INDEX</span> user_periods_multirange_gist_idx</span></span> ON</span> user_periods_multirange</span></span> USING</span> gist (active_periods);</span></span> </span> -- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row</span></span> INSERT INTO</span> user_periods_multirange (user_id, active_periods)</span></span> SELECT</span></span> user_id,</span></span> -- range_agg function automatically handles merging adjacent ranges</span></span> range_agg(active_period)::TSTZMULTIRANGE</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span> user_id </span>=</span> 42</span></span> GROUP BY</span> user_id;</span></span></code></pre> consider now the difference between</p> EXPLAIN</span></span> SELECT</span> period_id, user_id</span></span> FROM</span> user_periods_single_range</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_period @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------</span></span> Bitmap Heap Scan on user_periods_single_range (cost=4.19..13.67 rows=1 width=12)</span></span> Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span> Filter: (user_id = 42)</span></span> -> Bitmap Index Scan on user_periods_single_range_gist_idx (cost=0.00..4.19 rows=6 width=0)</span></span> Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> and same version of the consolidated data</p> EXPLAIN ANALYZE</span></span> SELECT</span> user_id, active_periods</span></span> FROM</span> user_periods_multirange</span></span> WHERE</span></span> user_id </span>=</span> 42</span></span> AND</span> active_periods @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre> QUERY PLAN</span></span> -------------------------------------------------------------------------------------------------------------</span></span> Index Scan using user_periods_multirange_pkey on user_periods_multirange (cost=0.15..8.17 rows=1 width=36)</span></span> Index Cond: (user_id = 42)</span></span> Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre> Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements.</p> One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps.</p> Creating custom range types</a> </h2> While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges.</p> To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values:</p> -- function to calculate difference between two IP addresses using bigint</span></span> CREATE OR REPLACE FUNCTION</span> inet_diff</span>(x </span>INET</span>, y </span>INET</span>)</span></span> RETURNS</span> FLOAT8 </span>AS</span> $$</span></span> SELECT</span> (</span></span> (host(x)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>) </span>-</span></span> (host(y)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>)</span></span> )::float8;</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span> </span> -- custom inetrange type</span></span> CREATE TYPE</span> inetrange</span> AS RANGE</span> (</span></span> subtype </span>= inet</span>,</span></span> subtype_diff </span>=</span> inet_diff</span></span> );</span></span> </span> -- convert CIDR ranges to inetranges</span></span> CREATE OR REPLACE FUNCTION</span> cidr_to_inetrange</span>(</span>cidr CIDR</span>)</span></span> RETURNS</span> inetrange </span>AS</span> $$</span></span> SELECT</span> inetrange(</span></span> host(network($</span>1</span>))::</span>inet</span>,</span></span> host(broadcast($</span>1</span>))::</span>inet</span>,</span></span> '[]'</span></span> );</span></span> $$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span></code></pre> Now you can use these custom types just like the built-in ones:</p> CREATE TABLE</span> ip_blocklists</span> (</span></span> blocklist_name </span>TEXT PRIMARY KEY</span>,</span></span> blocked_range inetrange </span>NOT NULL</span></span> );</span></span> </span> INSERT INTO</span> ip_blocklists (blocklist_name, blocked_range) </span>VALUES</span></span> (</span>'attack #434401'</span>, cidr_to_inetrange(</span>'192.168.1.0/24'</span>)),</span></span> (</span>'attack #434401 (1)'</span>, </span>'[203.0.113.50,203.0.113.99]'</span>),</span></span> (</span>'attack #434401 (2)'</span>, </span>'[203.0.113.143,203.0.113.159]'</span>);</span></span></code></pre> and locate which attack has been assigned to particular malicious IP address.</p> SELECT</span> blocklist_name, blocked_range</span></span> FROM</span> ip_blocklists</span></span> WHERE</span> blocked_range @</span>></span> '192.168.1.25'</span>::</span>INET</span>;</span></span></code></pre> blocklist_name \| blocked_range</span></span> ----------------+-----------------------------</span></span> attack #434401 \| [192.168.1.0,192.168.1.255]</span></span></code></pre> But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block.</p> Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges.</p> SELECT</span> typname </span>FROM</span> pg_type </span>WHERE</span> typname </span>LIKE</span> 'inet%range'</span>;</span></span></code></pre> typname</span></span> ----------------</span></span> inetmultirange</span></span> inetrange</span></span></code></pre> Danger zone! When creating a custom range type the subtype_diff</strong> function is more than just simple helper. It plays important role in indexing and query performance</strong>. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges.</p> In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans.</p> Performance deep dive: GiST vs GIN</a> </h2> Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt.</p> Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree)</strong> is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index)</strong> would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space.</p> The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with</strong> EXCLUDE constraints.</strong> GiST is mandatory there.</p> Therefore while GiST index is going to be preferred for cases like</p> CREATE TABLE</span> seat_holds</span> (</span></span> hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> seat_id </span>INTEGER NOT NULL</span>,</span></span> hold_period TSTZRANGE </span>NOT NULL</span></span> );</span></span></code></pre> the GIN index is actually preferred for</p> CREATE TABLE</span> venue_blackouts</span> (</span></span> venue_id </span>INTEGER</span>,</span></span> blocked_dates DATERANGE </span>NOT NULL</span></span> );</span></span></code></pre> The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day.</p> The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available.</p> Conclusion</a> </h2> From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design.</p> But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables</strong>. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article.</p> For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes, and it will save you a debugging session later.</p> Time to Better Know The Time in PostgreSQL 2025-04-06T00:00:00+00:00 To honor the name of the site (boringSQL) let's deep dive into a topic which might sound obvious, but it might be never ending source of surprises and misunderstanding.</p> Simple things</a> </h2> We can start with the simple statement like</p> SELECT</span> '2025-03-30 00:30'</span> as</span> t;</span></span></code></pre> Result:</p> t</span></span> ------------------</span></span> 2025-03-30 00:30</span></span></code></pre> which gives you (and I do hope that's not a big surprise) a simple text literal, rather than anything to do with date and time. As soon as you use the string literal in queries similar to</p> SELECT * FROM</span> events </span>WHERE</span> start_at </span>></span> '2025-03-30 00:30'</span>;</span></span></code></pre> PostgreSQL will implicitly convert the string into timestamp without time zone. The reason why it can happen is the fact start_at</code> is most likely timestamp based field, allowing automatic cast to match the column's data type. Which is unlike the query</p> SELECT</span> '2025-03-30 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre> Result:</p> ERROR: invalid input syntax for type interval: "2025-03-30 01:00"</span></span></code></pre> which will simply not work, as PostgreSQL can't perform automatic cast (which as we will cover later might be surprising behavior, but that's how things are). The correct way to get this example query to work is to use either one of two ways (both functionally equivalent but different notation).</p> -- PostgreSQL cast notation</span></span> SELECT</span> '2025-03-30 01:00'</span>::</span>timestamp -</span> interval </span>'15 minutes'</span>;</span></span> </span> -- SQL standard explicit type notation</span></span> SELECT timestamp</span> '2025-03-03 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre>PostgreSQL timestamp vs timestamptz</a> </h2> The next possible source of confusion when working with time in PostgreSQL is presence of two distinct data types:</p> timestamp</code> (or timestamp without time zone</code>)</li> timestamptz</code> (or timestamp with time zone</code>) Despite what the names suggest, the key difference isn't whether they store timezone information</strong>, but rather how they handle it during storage and retrieval.</li> </ul> IMPORTANT: before we cover the details, remember to always use timestamptz</code>. As official Don't Do This</a> page shows, using timestamp</code> is like storing picture of the clock, instead of point in time. And while this article might use timestamp</code> it's purely for demonstration purposes.</p> </blockquote> All you need to remember is the fact timestamp</code> stores the exact datetime value as entered</strong> without any timezone context or adjustments. It's essentially a snapshot of a calendar date and wall clock time, disconnected from any particular geographic location.</p> -- this stores what you provided</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp</span>;</span></span></code></pre> Result:</p> timestamp</span></span> ---------------------</span></span> 2025-03-30 01:30:00</span></span></code></pre> On the other hand timestamptz</code> normalizes all input to UTC internally</strong> and then converts values to the session's timezone for display. This provides geographic context to your datetime values.</p> -- this converts your input to UTC based on your session timezone</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+01</span></span></code></pre> And this is where potential confusion might start. Let's take this example</p> -- With UTC timezone</span></span> SET</span> timezone </span>=</span> 'UTC'</span>;</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+00</span></span></code></pre>-- With Tokyo timezone</span></span> SET</span> timezone </span>=</span> 'Asia/Tokyo'</span>;</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre> Result:</p> timestamptz</span></span> -----------------------</span></span> 2025-03-30 01:30:00+09</span></span></code></pre> which gives you correct representation based on the session timezone. This comes in handy when we add the actual storage.</p> -- Create table and view with Berlin timezone</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> CREATE TABLE</span> time_flies</span>(moment </span>timestamp with time zone</span>);</span></span> INSERT INTO</span> time_flies </span>VALUES</span> (</span>'2025-03-30 00:30'</span>);</span></span> SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre> Result:</p> moment</span></span> ------------------------</span></span> 2025-03-30 00:30:00+01</span></span></code></pre>-- View the same data with New York timezone</span></span> SET</span> timezone </span>=</span> 'America/New_York'</span>;</span></span> SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre> Result:</p> moment</span></span> ------------------------</span></span> 2025-03-29 19:30:00-04</span></span></code></pre> This demonstrates the advantage of using automatic time zone conversion, allowing you to avoid not only time-zone bugs, but also DST related issues. While most of us might be fast asleep during our local DST changes, it's no fun to account for the time difference in a wrong. Using timestamptz</code> is sure way how to avoid it. Let's take an example of recent (at the time of writing the article) DST change.</p> -- Set Berlin timezone (during DST change)</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> </span> -- Using timestamp (without timezone awareness)</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre> Result:</p> end_time</span></span> ---------------------</span></span> 2025-03-30 02:15:00</span></span></code></pre>-- Using timestamptz (with timezone awareness)</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre> Result:</p> end_time</span></span> ------------------------</span></span> 2025-03-30 03:15:00+02</span></span></code></pre>AT TIME ZONE</a> </h2> Another powerful but potentially confusing operator in PostgreSQL is AT TIME ZONE</code> which allows you to convert timestamps between different time zones, but its behavior differs depending on the input data type.</p> When applied to timestamp (without time zone)</strong> When you apply AT TIME ZONE to a timestamp, PostgreSQL interprets the input as being in the specified time zone</strong> and converts it to a timestamptz:</p> -- First, set UTC timezone</span></span> SET</span> timezone</span>=</span>'UTC'</span>;</span></span> </span> -- Interpret '2025-03-30 01:30' as if it were in the 'Europe/Paris' time zone</span></span> SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp AT TIME ZONE</span> 'Europe/Paris'</span>;</span></span></code></pre> Result:</p> timezone</span></span> ------------------------</span></span> 2025-03-30 00:30:00+00</span></span></code></pre> What it does is effectively "Take this wall clock time, consider it as being in the specified time zone, and give me the corresponding moment in time (as a timestamptz)". The representation here is based on session time zone settings.</p> When applied to timestamptz (with time zone)</strong> Conversely, when you apply AT TIME ZONE to a timestamptz, PostgreSQL converts the timestamp to the specified time zone and returns a timestamp without time zone</strong>:</p> -- Set timezone</span></span> SET</span> timezone</span>=</span>'America/Port_of_Spain'</span>;</span></span> </span> -- Convert the timestamptz to how it would appear on wall clocks in Tokyo</span></span> SELECT</span> '2025-03-30 01:30+01:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span>;</span></span></code></pre> Result:</p> timezone</span></span> ---------------------</span></span> 2025-03-30 09:30:00</span></span></code></pre> This operation says: "Take this moment in time and show me what wall clock time it would be in the specified time zone."</p> Practical usage</strong> Probably the only use when correctly using timestamps</code> is to provide "wall clock" representation of the times at various time zones at once for (view) representational purposes.</p> -- What time is the company-wide meeting in various offices?</span></span> SELECT</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AS</span> meeting_time_utc,</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Europe/London'</span> AS</span> london_office_time,</span></span> '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span> AS</span> tokyo_office_time;</span></span></code></pre> Result:</p> meeting_time_utc \| london_office_time \| tokyo_office_time</span></span> ------------------------+---------------------+---------------------</span></span> 2025-03-30 15:00:00+02 \| 2025-03-30 14:00:00 \| 2025-03-30 22:00:00</span></span></code></pre> It's important to note that in API based clients, it's always better to represent full time notation and leave the clients with the representational layer. Similar to that, unless you need to work with multiple time zones, and you depend on the correct local time representation it's always better to use local session timezone</code> setting instead of using AT TIME ZONE</code> operator. This applies for both time input and output.</p> Common mistake</strong> Let's consider the example where you might consider "magically casting" already stored time using AT TIME ZONE</code>.</p> -- Setup example</span></span> SET</span> timezone</span>=</span>'Europe/Berlin'</span>;</span></span> CREATE TABLE</span> different_moments</span>(moment1 </span>timestamptz</span>, moment2 </span>timestamptz</span>);</span></span> INSERT INTO</span> different_moments (moment1) </span>values</span> (</span>'2025-03-30 01:30'</span>);</span></span> </span> -- Apply double time zone conversion</span></span> UPDATE</span> different_moments </span>SET</span> moment2 </span>=</span> moment1 </span>AT TIME ZONE</span> 'Europe/Berlin'</span> AT TIME ZONE</span> 'America/New_York'</span>;</span></span> </span> -- View the results</span></span> SELECT * FROM</span> different_moments;</span></span></code></pre> Result:</p> moment1 \| moment2</span></span> ------------------------+------------------------</span></span> 2025-03-30 01:30:00+01 \| 2025-03-30 07:30:00+02</span></span></code></pre> Unless you are aware of the implicit conversion between timestamps</code> and timestamp</code> and vice-versa you might be set for a lot of surprises. In this particular case the first conversion performed redundant conversion and removed the time zone offset. While second "forced" it to NYC time, while the subsequent select rendered it in local session timezone.</p> Timestamps and the storage</a> </h2> While you will use timestamp with time zone</code> from now on, PostgreSQL still comes with several nuances related to the way it can store the data. First lesser known feature might be timestamp precision. Let's consider following table.</p> -- Table with various timestamp precision specifications</span></span> CREATE TABLE</span> precision</span> (</span></span> t1 </span>timestamp with time zone</span>, </span>-- default precision (6)</span></span> t2 </span>timestamp</span>(</span>0</span>)</span> with time zone</span>, </span>-- seconds precision</span></span> t3 </span>timestamp</span>(</span>2</span>)</span> with time zone</span>, </span>-- centiseconds precision</span></span> t4 </span>timestamp</span>(</span>4</span>)</span> with time zone</span> -- 10 microseconds precision</span></span> );</span></span></code></pre> Effectively the timestamp</code> comes with default precision of 6 digits (microseconds), but you can specify anywhere from 0 to 6 for both timestamp</code> and timestamptz</code> types. At the same time it does not have any impact on the storage (8 bytes) regardless the precision specified.</p> -- Insert the same timestamp into all columns</span></span> INSERT INTO precision VALUES</span> (current_timestamp, current_timestamp, current_timestamp, current_timestamp);</span></span> SELECT * FROM precision</span>;</span></span></code></pre> Result:</p> -[ RECORD 1 ]---------------------</span></span> t1 \| 2025-04-03 21:19:20.952354+02</span></span> t2 \| 2025-04-04 21:19:21+02</span></span> t3 \| 2025-04-04 21:19:20.95+02</span></span> t4 \| 2025-04-04 21:19:20.9524+02</span></span></code></pre> Other than precision there are some other interesting aspects of the timestamps storage in PostgreSQL:</p> It has finite range</strong> of timestamptz '4713-01-01 BC'</code> and timestamptz '294276-12-31'</code></li> and at the same time supports positive and negative infinity</strong> with 'infinity'::timestamptz</code> and '-infinity'::timestamptz</code> that might provide alternative to NULL</code> for open ended intervals. If you choose to use infinity instead of NULL</code> values, you can test whatever the provided date/timestamp is finite value or not using isfinite(timestamp)</code>.</li> </ul> Getting the current time</a> </h2> Not many developers are aware that there are different ways to get the current datetime in SQL. These methods not only differ in syntax but also in their underlying logic.</p> CURRENT_TIMESTAMP vs NOW()</a> </h3> CURRENT_TIMESTAMP</strong> is part of the SQL Standard and interestingly, it's defined as both a "time-varying variable" and a "datetime value function". In PostgreSQL, you can use it with or without parentheses, making both these forms valid:</p> SELECT</span> CURRENT_TIMESTAMP;</span></span> SELECT</span> CURRENT_TIMESTAMP</span>(</span>2</span>); </span>-- (2) specifies precision</span></span></code></pre> Along with CURRENT_TIMESTAMP, the SQL Standard also defines CURRENT_TIME</code> and CURRENT_DATE</code> for working with individual time and date components respectively.</p> NOW()</code>, while not part of the SQL Standard, is a widely used alternative across many database systems. Unlike CURRENT_TIMESTAMP, it's strictly a function and therefore always requires parentheses when called. Its straightforward syntax - NOW()</code> - makes it a popular choice among developers.</p> Transaction Behavior</a> </h3> Both NOW()</code> and CURRENT_TIMESTAMP</code> share the same transaction behavior: they return the timestamp from the start of the current transaction, not the moment of execution:</p> BEGIN</span>;</span></span> SELECT NOW</span>(); </span>-- Returns transaction start time</span></span> SELECT</span> pg_sleep(</span>5</span>); </span>-- Wait 5 seconds</span></span> SELECT NOW</span>(); </span>-- Still returns the same time</span></span> COMMIT</span>;</span></span></code></pre> This behavior ensures data consistency within transactions but might be unexpected when measuring elapsed time.</p> Alternative Time Functions</a> </h3> When you need the actual current time regardless of transaction status, CLOCK_TIMESTAMP()</code> is your best choice. It returns the precise moment of execution, making it particularly useful for measuring real elapsed time. Here's how it differs from NOW()</code>:</p> BEGIN</span>;</span></span> SELECT</span> pg_sleep(</span>1</span>);</span></span> SELECT NOW</span>()</span> AS</span> transaction_time,</span></span> CLOCK_TIMESTAMP()</span> AS</span> actual_time;</span></span> COMMIT</span>;</span></span></code></pre> Result:</p> transaction_time \| actual_time</span></span> ------------------------------+-------------------------------</span></span> 2025-04-03 19:28:25.310254+00 \| 2025-04-03 19:28:26.311917+00</span></span></code></pre> Another useful function is STATEMENT_TIMESTAMP()</code>, which captures the time when the current statement began executing. This differs subtly from CLOCK_TIMESTAMP()</code>, which gives you the exact moment of function execution. The difference becomes clear in this example:</p> SELECT</span></span> pg_sleep(</span>2</span>),</span></span> STATEMENT_TIMESTAMP()</span> AS statement</span>,</span></span> CLOCK_TIMESTAMP()</span> AS</span> wall_time;</span></span></code></pre> Result:</p> pg_sleep \| statement \| wall_time</span></span> ----------+-------------------------------+------------------------------</span></span> \| 2025-04-03 19:32:15.984459+00 \| 2025-04-03 19:32:17.98661+00</span></span></code></pre> These timestamp functions serve different purposes and are invaluable when you need to measure transaction timing or analyze statement-level performance. Each provides a different perspective on time within your database operations.</p> Intervals</a> </h2> Guess if you have worked with time in PostgreSQL you have come across with intervals. You probably just entered the it using something like interval '1 year 2 months 3 days 4 hours'</code>. Technical that's representation driven by setting intervalstyle</code> which (as in this case) is set to postgres_verbose</code>. But you have much more options.</p> -- PostgreSQL default style</span></span> SET</span> intervalstyle </span>=</span> 'postgres'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> -------------------------------</span></span> 1 year 2 mons 3 days 04:00:00</span></span></code></pre>-- SQL standard style</span></span> SET</span> intervalstyle </span>=</span> 'sql_standard'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> ------------------</span></span> +1-2 +3 +4:00:00</span></span></code></pre>-- ISO 8601 style</span></span> SET</span> intervalstyle </span>=</span> 'iso_8601'</span>;</span></span> SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre> Result:</p> interval</span></span> ------------</span></span> P1Y2M3DT4H</span></span></code></pre> With the session intervalstyle</code> you only change the interval format representation, not the actual value. Nevertheless the interval representation you might run into an interval definition that might not longer make it obvious (both in input or output) to interpret. That's where function justify_interval</code> comes in play - helping you to normalize time expression into conventional formats.</p> -- Normalize complex interval</span></span> SELECT</span> justify_interval(interval </span>'15 months 40 days 30 hours'</span>);</span></span></code></pre> Result:</p> justify_interval</span></span> ---------------------------------</span></span> @ 1 year 4 mons 11 days 6 hours</span></span></code></pre>-- Practical example: normalize project durations</span></span> SELECT</span></span> project_name,</span></span> total_hours </span>\|\|</span> ' hours'</span> AS</span> raw_duration,</span></span> justify_interval(interval </span>'1 hour'</span> </span> total_hours) </span>AS</span> normalized_duration</span></span> FROM</span></span> (</span>VALUES</span> (</span>'Database Migration'</span>, </span>1850</span>),</span></span> (</span>'API Development'</span>, </span>720</span>),</span></span> (</span>'UI Redesign'</span>, </span>340</span>))</span></span> AS</span> projects(project_name, total_hours);</span></span></code></pre> Result:</p> project_name \| raw_duration \| normalized_duration</span></span> --------------------+--------------+--------------------------</span></span> Database Migration \| 1850 hours \| @ 2 mons 17 days 2 hours</span></span> API Development \| 720 hours \| @ 1 mon</span></span> UI Redesign \| 340 hours \| @ 14 days 4 hours</span></span></code></pre> One of the important specific with internal normalization is the fact it uses 30-days time periods as a month definition.</p> -- Month definition in normalize intervals</span></span> SELECT</span> justify_interval(interval </span>'30 days'</span>);</span></span></code></pre> Result:</p> justify_interval</span></span> ------------------</span></span> @ 1 mon</span></span></code></pre> Which might be understandable, but you need to beware of the edge cases when using days and months intervals.</p> -- Edge case comparison: month vs 30 days</span></span> SELECT</span></span> date</span> '2025-01-31'</span> +</span> interval </span>'1 month'</span> as</span> example1,</span></span> date</span> '2025-01-31'</span> +</span> interval </span>'30 days'</span> as</span> example2;</span></span></code></pre> Result:</p> -[ RECORD 1 ]-----------------</span></span> example1 \| 2025-02-28 00:00:00</span></span> example2 \| 2025-03-02 00:00:00</span></span></code></pre> And one last thing, before wrapping up the interval - you can (same as with dates/timestamp) get specific parts using extract</code> function.</p> -- Extract specific parts from intervals</span></span> SELECT</span></span> extract(</span>days from</span> interval </span>'1 year 3 months 21 days'</span>) </span>AS days</span>,</span></span> extract(</span>hours from</span> interval </span>'25:30:45'</span>) </span>AS hours</span>;</span></span></code></pre> Result:</p> days \| hours</span></span> ------+-------</span></span> 21 \| 25</span></span></code></pre>Time ranges</a> </h2> While working with timestamp comes natural, it's time ranges which often feel like ugly ducklings. And quite unfairly so - timestamp ranges in PostgreSQL are powerful yet underutilized features that elegantly solve common time-based challenges. PostgreSQL offers dedicated range types that are perfect for modeling time periods, reservations, schedules, and any situation where you need to track intervals with defined start and end points:</p> tsrange</code> - range of timestamp without time zone</li> tstzrange</code> range of timestamp with time zone And to follow the earlier advice - just as you should default to timestamps</code>, always use tstzrange</code> for time ranges unless you have VERY specific reason for it. Timestamp ranges aren't merely convenient syntax - they provide a complete set of operations for determining relationships between time periods and enable powerful constraints that handle complex business rules without reinventing the wheel. Timestamp ranges can be created using the range constructor syntax or string syntax.</li> </ul> -- Range constructor syntax</span></span> SELECT</span> tstzrange(</span></span> '2025-04-01 09:00:00+02'</span>::</span>timestamptz</span>,</span></span> '2025-04-01 17:30:00+02'</span>::</span>timestamptz</span>,</span></span> '[)'</span></span> ) </span>AS</span> workday;</span></span> </span> -- String syntax</span></span> SELECT</span> '[2025-04-01 09:00:00+02, 2025-04-01 17:30:00+02)'</span>::tstzrange </span>AS</span> workday;</span></span></code></pre> Result:</p> workday</span></span> ["2025-04-01 07:00:00+00","2025-04-01 15:30:00+00")</span></span></code></pre> The default boundary notation for timestamp ranges is [)</code> - lower bound inclusive (includes the start time) and upper bound exclusive (excludes the end time). For time-based applications, this convention is particularly valuable. Consider scheduling consecutive one-hour meetings from 9:00 to 10:00 and 10:00 to 11:00. With [)</code> notation, these are represented as [09:00, 10:00)</code> and [10:00, 11:00)</code>, creating perfect adjacency without overlap or gaps. If you used inclusive bounds [09:00, 10:00]</code> and [10:00, 11:00]</code>, the moment 10:00 would technically belong to both ranges - a logical impossibility for scheduling. This natural alignment with how we conceptualize time slots makes [)</code> notation the default and recommended choice for most of the application use cases. Timestamp ranges support numerous operations for checking relationships:</p> -- 1. Check if a timestamp is within a range</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange @</span>></span> '2025-04-01 12:30'</span>::</span>timestamptz AS</span> is_during_workday;</span></span></code></pre> Result:</p> is_during_workday</span></span> ------------------</span></span> t</span></span></code></pre>-- 2. Check if two ranges overlap</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange &&</span></span> '[2025-04-01 11:00, 2025-04-01 14:00)'</span>::tstzrange </span>AS</span> meetings_overlap;</span></span></code></pre> Result:</p> meetings_overlap</span></span> -----------------</span></span> t</span></span></code></pre>-- 3. Extract time range bounds</span></span> SELECT</span></span> lower</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> start_time,</span></span> upper</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> end_time;</span></span></code></pre> Result:</p> start_time \| end_time</span></span> --------------------------+--------------------------</span></span> 2025-04-01 09:00:00+00 \| 2025-04-01 17:00:00+00</span></span></code></pre> As well as obvious manipulations:</p> -- 1. Merge adjacent ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange </span>+</span></span> '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> full_day;</span></span></code></pre> Result:</p> full_day</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 09:00:00+00","2025-04-01 17:00:00+00")</span></span></code></pre>-- 2. Find intersection between overlapping ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 14:00)'</span>::tstzrange </span></span></span> '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> overlap_period;</span></span></code></pre> Result:</p> overlap_period</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 12:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre>-- 3. Find gap between non-overlapping ranges</span></span> SELECT</span></span> '[2025-04-01 09:00, 2025-04-01 11:00)'</span>::tstzrange </span>-</span></span> '[2025-04-01 14:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> gap;</span></span></code></pre> Result:</p> gap</span></span> -------------------------------------------------------------</span></span> ["2025-04-01 11:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre> And if you followed the previously mentioned logic, you wouldn't double guess tstzrange</code> handling of the DST transitions.</p> -- Testing DST transition handling</span></span> SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span> SELECT</span></span> '[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange </span>AS</span> dst_transition_range,</span></span> upper</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>-</span></span> lower</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>AS</span> actual_duration;</span></span></code></pre> Result:</p> dst_transition_range \| actual_duration</span></span> -----------------------------------------------------+-----------------</span></span> ["2025-03-30 01:00:00+01","2025-03-30 03:00:00+02") \| @ 1 hour</span></span></code></pre> The area where time ranges win in almost all scenarios is indexing. If you consider a manual implementation of the range using start_at</code> and end_at</code> columns you typically rely on B-tree indexes. While they might work for simple queries on just one of the values, they fall short for time ranges. Consider this example:</p> -- Traditional approach with two columns and B-tree indexes</span></span> SELECT * FROM</span> events_columns</span></span> WHERE</span> start_at </span><=</span> '2025-04-01 17:00'</span></span> AND</span> end_at </span>>=</span> '2025-04-01 09:00'</span>;</span></span></code></pre> For this query, PostgreSQL might use one of the indexes if it's selective enough, but the fundamental problem remains: each B-tree index can only efficiently filter on its own column. The planner might use the start_at index to find events starting before 17:00, or the end_at index to find events ending after 09:00, but it can't use both indexes simultaneously to find the intersection. With range types, we can use GiST</strong> (Generalized Search Tree) indexes that are specifically designed for range operations:</p> -- Create specialized GiST index for time range operations</span></span> CREATE INDEX</span> idx_events_range</span> ON</span> events_range </span>USING</span> GIST(time_period);</span></span></code></pre> This allows you to transform the sample query into:</p> -- Range-based approach using the && (overlap) operator</span></span> SELECT * FROM</span> events_range</span></span> WHERE</span> time_period && tstzrange(</span>'2025-04-01 09:00'</span>, </span>'2025-04-01 17:00'</span>);</span></span></code></pre> The performance gap between these approaches is substantial – range queries with GiST indexes typically outperform</strong> the traditional approach by 2-10x on large datasets. This difference becomes even more pronounced as your data grows.</p> Time to wrap up</a> </h2> Working with time in PostgreSQL has more edge cases than it first appears. To summarize:</p> Always use timestamptz instead of timestamp</strong></li> Be intentional about time zone handling</strong></li> Leverage PostgreSQL's specialized time tools</strong></li> Mind the edge cases</strong></li> </ul> Time handling has more edge cases than most software engineering problems, but PostgreSQL's time types cover nearly all of them. Understanding these "boring" concepts will save you from the pitfalls covered above.</p> VIEW inlining in PostgreSQL 2025-02-08T00:00:00+00:00 Database VIEWs are powerful tools that often don't get the attention they deserve when building database-driven applications. They make our database work easier in several ways:</p> They let us reuse common query patterns instead of writing them over and over</li> They give us a place to define business rules once and use them everywhere</li> They help us write cleaner, more organized queries</li> </ul> Let's see how this works with a practical example. Imagine we want to work with active users</em> - users who have used the application within the last 7 days. Instead of writing this condition in every query, we can define a view:</p> CREATE VIEW</span> active_users</span> AS</span></span> SELECT</span></span> </span></span> FROM</span> users;</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span>;</span></span></code></pre> Now we can easily use this view whenever we need active users. For example, if we want to find active users from Germany:</p> SELECT</span></span> user_id</span></span> FROM</span> active_users</span></span> WHERE</span></span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> While this simple example shows how views can make developers' lives easier by organizing and reusing common logic, there's more to the story. In this article, we'll explore something even more interesting: how PostgreSQL can optimize these views through a process called "inlining" - making them not just convenient, but fast too.</p> What is VIEW inlining</a> </h2> When you use a view, it acts like a building block in SQL queries. Taking our previous example, PostgreSQL effectively transforms the query by replacing the view with its underlying subquery.</p> SELECT</span></span> user_id</span></span> FROM</span> (</span></span> SELECT</span></span> </span></span> FROM</span> users</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span> ) active users</span></span> WHERE</span></span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> While we write the query using active_users</code> VIEW, PostgreSQL query planner will see it as an opportunity to optimize it further. Instead of treating the sub-query as separate step (and retrieving large subset of the users), it effectively transforms the query and inlines</strong> the view into the query itself. Behind the scenes, PostgreSQL will execute the query similar to:</p> SELECT</span></span> user_id</span></span> FROM</span> users</span></span> WHERE</span></span> last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span> AND</span> country </span>=</span> 'Germany'</span>;</span></span></code></pre> The inlining process</strong> allows the PostgreSQL query planner to optimize the entire query as a single unit. This allows it to select the best indexes, possible join strategies and other aspects together, rather than having to execute the individual parts separately and then filtering the data that won't be otherwise used. This is exactly what VIEWs are for - we get the best of both worlds - clean, modular code for developers and optimal performance for the database.</p> Inlining in action</a> </h2> The easiest way how to preview the VIEW inlining is to run EXPLAIN</p> Seq Scan on users (cost=0.00..19.20 rows=1 width=4)</span></span> Filter: ((country = 'Germany'::text) AND (last_login > (CURRENT_DATE - '7 days'::interval)))</span></span></code></pre> The Filter part here is the one showing how the filtering conditions from both the query and the view got merged together by the query planner.</p> This works especially well with complex queries involving joins, aggregations and filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p> CREATE VIEW</span> order_analytics</span> AS</span></span> SELECT</span></span> o</span>.</span>customer_id</span>,</span></span> c</span>.</span>country</span>,</span></span> DATE_TRUNC(</span>'month'</span>, </span>o</span>.</span>created_at</span>) </span>as month</span>,</span></span> COUNT</span>(</span></span>) </span>as</span> orders,</span></span> SUM</span>(</span>o</span>.</span>total_amount</span>) </span>as</span> revenue</span></span> FROM</span> orders o</span></span> JOIN</span> customers c </span>ON</span> c</span>.</span>id</span> =</span> o</span>.</span>customer_id</span></span> GROUP BY</span> 1</span>, </span>2</span>, </span>3</span>;</span></span></code></pre> When you use this VIEW in the query</p> SELECT</span> customer_id, revenue</span></span> FROM</span> order_analytics</span></span> WHERE</span> country </span>=</span> 'Germany'</span></span> AND month >=</span> '2024-01-01'</span></span> AND</span> revenue </span>></span> 1000</span>;</span></span></code></pre> you will see how PostgreSQL effectively inlines the filter conditions where they belong.</p> HashAggregate (cost=205.97..207.97 rows=200 width=16)</span></span> Group Key: o.customer_id, c.country, (date_trunc('month'::text, o.created_at))</span></span> Filter: (sum(o.total_amount) > 1000)</span></span> -> Hash Join (cost=16.12..185.25 rows=1235 width=20)</span></span> Hash Cond: (o.customer_id = c.id)</span></span> -> Seq Scan on orders o</span></span> Filter: (created_at >= '2024-01-01'::date)</span></span> -> Hash</span></span> -> Seq Scan on customers c</span></span> Filter: (country = 'Germany'::text)</span></span></code></pre> This works especially well with increasing query complexity. Particulary involving joins and further filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p> CREATE VIEW completed_orders AS</span></span> SELECT</span></span> o.id,</span></span> o.customer_id,</span></span> o.total_amount,</span></span> o.created_at,</span></span> o.status,</span></span> p.name as product_name,</span></span> c.email,</span></span> c.country</span></span> FROM orders o</span></span> JOIN customers c ON c.id = o.customer_id</span></span> JOIN order_items oi ON oi.order_id = o.id</span></span> JOIN products p ON p.id = oi.product_id</span></span> WHERE</span></span> o.status = 'completed';</span></span> </span> SELECT</span></span> </span></span> FROM completed_orders</span></span> WHERE</span></span> country = 'Germany';</span></span> </span> CREATE INDEX idx_orders_customer_id ON orders(customer_id); CREATE INDEX idx_order_items_order_id ON order_items(order_id); CREATE INDEX idx_order_items_product_id ON order_items(product_id);</span></span></code></pre> as demonstrated in the query plan</p> Nested Loop (cost=0.30..19.70 rows=1 width=176)</span></span> -> Nested Loop (cost=0.15..13.52 rows=1 width=148)</span></span> Join Filter: (o.id = oi.order_id)</span></span> -> Nested Loop (cost=0.15..12.43 rows=1 width=144)</span></span> -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=80)</span></span> Filter: (status = 'completed'::text)</span></span> -> Index Scan using customers_pkey on customers c (cost=0.15..8.17 rows=1 width=68)</span></span> Index Cond: (id = o.customer_id)</span></span> Filter: (country = 'Germany'::text)</span></span> -> Seq Scan on order_items oi (cost=0.00..1.04 rows=4 width=8)</span></span> -> Index Scan using products_pkey on products p (cost=0.15..6.17 rows=1 width=36)</span></span> Index Cond: (id = oi.product_id)</span></span></code></pre>Planner barriers</a> </h2> While PostgreSQL is very good at inlining views, certain operations create "planner barrier" that prevent the optimization. In case of views some of the conditions that will cause it are</p> DISTINCT ON operations</li> Window functions (OVER clauses)</li> Set operations (UNION/INTERSECT/EXCEPT)</li> More complex aggregations</li> Common Table Expressions that are materialized (either with MATERIALIZED hint or by the decision of the query planner</a>)</li> Use of VOLATILE functions</li> And in some cases complex subqueries</li> </ul> When planner is not able to inline VIEWs it might lead to the unnecessary performance and resource impact. I.e. each sub query might materialize results separately, leading to the already retrieved rows to be discarded by another part of the query as one of the most common examples.</p> When using EXPLAIN this is demostated by one of the following options:</p> Subquery scan on...</code> nodes</li> Materialization nodes</li> Separate aggregation/sorting steps before the main execution</li> </ul> As mentioned above the window function acts as such a planner barrier.</p> CREATE VIEW order_insights AS</span></span> SELECT</span></span> o.customer_id,</span></span> o.total_amount,</span></span> o.created_at,</span></span> ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as order_sequence,</span></span> SUM(o.total_amount) OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as running_total</span></span> FROM orders o;</span></span> </span> EXPLAIN</span></span> SELECT * FROM order_insights</span></span> WHERE customer_id = 1</span></span> AND total_amount > 500;</span></span></code></pre> Giving us easy to spot Subquery scan</code> node.</p> Subquery Scan on order_insights (cost=1.06..1.10 rows=1 width=84)</span></span> Filter: (order_insights.total_amount > '500'::numeric)</span></span> -> WindowAgg (cost=1.06..1.08 rows=1 width=84)</span></span> -> Sort (cost=1.06..1.06 rows=1 width=44)</span></span> Sort Key: o.created_at</span></span> -> Seq Scan on orders o (cost=0.00..1.05 rows=1 width=44)</span></span> Filter: (customer_id = 1)</span></span></code></pre>Runtime Materialization (Not MATERIALIZED VIEWs)</a> </h2> Let's clear up a possible confusion - we're not talking about CREATE MATERIALIZED VIEW here. This is about PostgreSQL's runtime decision to cache view results in memory during query execution. Query planner might use explicit materialization when it needs to reference view results multiple times or prevent repeated expensive computations. The query planner shows this through a Materialize node:</p> CREATE VIEW</span> customer_summary</span> AS</span></span> SELECT</span></span> customer_id,</span></span> COUNT</span>(</span></span>) </span>as</span> orders,</span></span> SUM</span>(total_amount) </span>as</span> total_spent</span></span> FROM</span> orders</span></span> GROUP BY</span> 1</span>;</span></span> </span> EXPLAIN </span>SELECT</span></span> a</span>.</span>customer_id</span>,</span></span> a</span>.</span>total_spent</span>,</span></span> b</span>.</span>total_spent</span> as</span> other_customer_spent</span></span> FROM</span> customer_summary a</span></span> JOIN</span> customer_summary b </span>ON</span> b</span>.</span>total_spent</span> ></span> a</span>.</span>total_spent</span>;</span></span></code></pre> Giving the perfect example of materialization.</p> Nested Loop (cost=46.00..200.75 rows=3333 width=68)</span></span> Join Filter: (b.total_spent > (sum(orders.total_amount)))</span></span> -> HashAggregate (cost=23.00..24.25 rows=100 width=44)</span></span> Group Key: orders.customer_id</span></span> -> Seq Scan on orders (cost=0.00..18.00 rows=1000 width=15)</span></span> -> Materialize (cost=23.00..25.75 rows=100 width=32)</span></span> -> Subquery Scan on b (cost=23.00..25.25 rows=100 width=32)</span></span> -> HashAggregate (cost=23.00..24.25 rows=100 width=44)</span></span> Group Key: orders_1.customer_id</span></span> -> Seq Scan on orders orders_1 (cost=0.00..18.00 rows=1000 width=15)</span></span></code></pre> Next to the materialization - the materialized VIEWs</strong> (CREATE MATERIALIZED VIEW</code>) are for practical purposes "just another table" they present an ultimate planner barrier. PostgreSQL must scan the materialized data directly, trading query flexibility for performance</p> Tips for writing inlining friendly VIEWs</a> </h2> Well, there's not a single "right" way how to write views. And don't forget - VIEWs should make your database easier to work with, not harder. When designing views, focus on single responsibility - each view should handle one aspect of business logic rather than trying to optimize everything at once. Simple views are more likely to get inlined by PostgreSQL's query planner.</p> There are several common patterns that prevent inlining</p> Avoid window functions, distinct and set operations completely</li> Split complex logic into multiple views if possible</li> Minimize subquery usage</li> Consider materialized views for aggregations</li> </ol> View dependencies are a critical consideration. Even minor schema changes can trigger cascading view modifications across your database. Instead of creating deep view hierarchies, split complex logic into smaller, composable views. For critical views that many applications depend on, consider versioning (v1, v2) rather than modifying existing ones.</p> When in doubt, write the plain SQL first, then extract common patterns into views. This helps avoid overengineering and keeps your database schema maintainable.</p> Inlining is only half the story. The other half is what views are made of in the catalog, and why every column change to the underlying table turns into a drop-and-recreate exercise. That trade-off, and the workarounds, is covered in Strong views on PostgreSQL VIEWs</a>.</p> DELETEs are difficult 2024-11-23T00:00:00+00:00 Your database is ticking along nicely - until a simple DELETE brings it to its knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT operations, we often overlook the hidden complexities of DELETE. Yet, removing unnecessary data is just as critical. Outdated or irrelevant data can bloat your database, degrade performance, and make maintenance a nightmare. Worse, retaining some types of data without valid justification might even lead to compliance issues.</p> At first glance, the DELETE command seems straightforward. Even the PostgreSQL documentation</a> provides simple examples like:</p> DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span> DELETE FROM</span> films;</span></span></code></pre> These queries might work effortlessly on your development machine, where only a few hundred records exist. But what happens when you try running a similar DELETE in production, where datasets are orders of magnitude larger?</p> In this article, we’ll uncover why DELETE operations demand careful consideration and explore how to handle them effectively.</p> What really happens when you DELETE the data?</a> </h2> At first glance, a DELETE query might seem straightforward. However, once the query is executed, a series of intricate steps occur:</p> Row Identification</strong>: Similar to a SELECT operation, the query identifies rows visible to the current transaction (considering MVCC) and checks for locks.</li> Lock Acquisition</strong>: The database acquires row-level exclusive locks to prevent other operations on the targeted rows.</li> BEFORE DELETE trigger</strong>: If a BEFORE DELETE trigger is defined, it is executed at this point.</li> Marking Rows as Deleted</strong>: Instead of being physically removed, the rows are marked as deleted in the current transaction, rendering them invisible to future queries (depending on transaction isolation). If table has large data objects, TOAST table will have to be involved too.</li> Index Updates</strong>: The corresponding index entries are also marked for deletion (if applicable).</li> Cascaded Actions</strong>: Cascading operations, such as ON DELETE CASCADE, are performed on related tables.</li> AFTER DELETE Trigger</strong>: If an AFTER DELETE trigger is defined, it is executed.</li> Write-Ahead Log (WAL)</strong>: Changes are recorded in the WAL-first at the row level, followed by index-level updates.</li> </ol> Only when the transaction is committed do these changes become permanent and visible to transactions starting afterward. However, even at this point, the data is not physically removed</strong>. This is how bloat</strong> is created.</p> Until the autovacuum</strong> process or a manual VACUUM operation reclaims the space, the “deleted” data remains. This leftover data contributes to bloat, which can degrade query performance over time.</p> The key question now is whether DELETEs are truly the hardest operation we can subject our database to. The answer? Quite possibly. While UPDATEs come close in complexity, they’re typically designed in ways that make them less challenging:</p> UPDATEs usually modify only a limited number of columns, reducing the potential number of index updates lower.</li> Not all UPDATEs trigger a full row (COLD) update, where the old row is marked as dead and a new row is created. With careful table and query design, you can minimise these cases. HOT (Heap-Only Tuple) updates</a>, for instance, are easier to achieve with fixed-length columns.</li> Unlike DELETEs, UPDATEs don’t trigger cascaded actions - they only involve triggers that are explicitly defined.</li> </ul> And then comes AUTOVACUUM</a> </h2> When autovacuum kicks in (and you really want it to kick in) - typically triggered by thresholds for the number of dead tuples or changes to the table - a significant amount of work is required to clean them up. Let’s break it down step by step:</p> The process begins by scanning the table. While it’s not always a full table scan, the autovacuum checks the visibility map and pages where dead tuples might exist. This can happen incrementally, allowing even large tables to be processed - provided your autovacuum settings allow it to run frequently enough.</li> Each tuple is checked to ensure it’s no longer visible to any active or pending transactions.</li> Dead tuples that pass the visibility check are physically removed from the table.</li> Corresponding index entries for the removed tuples are updated.</li> The now-empty space is marked for reuse in future INSERT or UPDATE operations.</li> Table statistics are updated to reflect the current state, helping the query planner make better decisions.</li> The changes, including tuple removals and index updates, are logged in the Write-Ahead Log (WAL) for durability and replication.</li> If the table has TOASTed data (large objects), the associated TOAST tables are processed.</li> The visibility map is updated to mark cleaned pages as fully visible again.</li> Autovacuum resets thresholds to determine when the next vacuum operation should occur.</li> </ul> This process continues until it reaches the configured vacuum cost limit (in the case of autovacuum), at which point it pauses or stops. While autovacuum helps keep your database in check, it’s clear that reclaiming dead tuples is no small task - and it underscores why DELETE operations can have lasting effects on database performance.</p> While the AUTOVACUUM</code> might sounds as a bad news, without it, your database would quickly become bloated with dead tuples, leading to degraded performance, slower queries, increased storage usage, and even the potential for out-of-disk errors as unused space cannot be reclaimed.</p> Further considerations</a> </h2> For what seems like a simple DELETE, a surprising amount of work has already taken place, but the complexity doesn’t stop there. DELETE operations can introduce additional challenges, particularly when replication, resource contention, or the size of the operation comes into play.</p> In environments with replication to hot standby or replicas, DELETEs become more time-sensitive. The transaction cannot complete until the corresponding WAL (Write-Ahead Log) records have been written to disk on the standby. This is a fundamental requirement for maintaining data consistency in high-availability setups, where at least one standby server is typically involved. Additionally, if the standby is actively serving read operations, it must account for DELETEs before confirming the changes, potentially introducing further delays.</p> The size of the DELETE operation also plays a critical role. Small DELETEs, such as removing a single row, tend to have minimal impact. However, as the size of the operation grows, so does the volume of WAL records generated. Large DELETEs can overwhelm the system, slowing down transactions and straining the replication process. Standby servers must work harder to process the incoming WAL stream, which can bottleneck performance if their throughput is insufficient.</p> Resource contention adds yet another layer of complexity, particularly for large DELETEs. Generating WAL records, handling regular transactional workloads, and running background processes can collectively push the system towards I/O saturation. This creates competition for CPU and memory resources, leading to slower operations across the board.</p> Finally, once the data is marked for deletion, the autovacuum process must eventually step in to physically remove it. This introduces its own set of challenges, as autovacuum must deal with the same resource contention and I/O demands, compounding the overall impact of the initial DELETE operation.</p> Soft-deletes are not the solution</a> </h2> Soft-deletes might seem like an easy way to sidestep the complexities of traditional DELETE operations. After all, updating a deleted_at</code> field is straightforward and unlikely to trigger a COLD update. However, soft-deletes are not a true mechanism for data removal, and they come with their own set of complications.</p> While soft-deletes can provide a simple way to implement “undo” functionality, they raise serious questions about data consistency. For instance, do you only mark the main entity as deleted, or do you also cascade the status to all related records in referenced tables? Failing to cascade properly can leave your database in an inconsistent state, making it difficult to maintain data integrity.</p> Soft-deletes also require consideration in your application logic. Every query must include appropriate filters to exclude “deleted” rows, which can complicate query design and increase the risk of oversights. One missed filter could expose data that should no longer be visible, leading to potential security or business logic issues.</p> Finally, soft-deletes don’t solve the problem - they do merely postpone it. The data is still in your database, consuming storage and potentially contributing to performance degradation over time. Sooner or later, you’ll need to deal with the actual removal of this data, bringing you back to the same challenges that DELETEs pose in the first place.</p> At the time of writing this article we can only speculate how much will support of temporal PRIMARY KEY and UNIQUE constriants in PostgreSQL 18 will transform the balances in future. But given the complexity of the feature I wouldn't bet on it just yet.</p> Batching is the answer</a> </h2> Giving PostgreSQL time to process and catch up with large-scale changes is critical when dealing with operations like DELETEs. The core issue here is the duration and magnitude of the transaction. The shorter the transaction and the fewer changes made, the better PostgreSQL can manage and reconcile those changes.</strong> This principle is universal across all database operations and underscores the importance of minimizing the impact of individual transactions.</p> While you can optimize certain aspects, like row identification (using indexes, clustering, or similar techniques), larger datasets demand a more strategic approach - batching. For example, deleting 1 million rows in a single transaction is a textbook case of what not to do. Instead, splitting the operation into smaller batches, such as deleting 10,000 rows across 100 iterations, is far more effective.</p> Will this method be faster than performing one massive DELETE? Likely not, especially if you include a wait time between batches to allow PostgreSQL to handle other workloads. However, the trade-off is worthwhile. By batching, you give PostgreSQL more breathing room to manage changes without overwhelming regular transactional workloads - unless, of course, you’ve scheduled dedicated maintenance time for the operation.</p> How to Batch DELETEs</a> </h3> The easiest way to batch DELETEs is to use a subquery or a Common Table Expression (CTE)</a> to limit the number of rows affected in each iteration. For example, instead of executing a bulk DELETE like this:</p> DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span></code></pre> You can break the operation into smaller chunks. Using a query like the following, you can repeatedly delete rows in manageable batches (for instance, using \watch in psql to automate iterations):</p> DELETE FROM</span> films</span></span> WHERE</span> ctid </span>IN</span> (</span></span> SELECT</span> ctid </span>FROM</span> films</span></span> WHERE</span> kind </span><></span> 'Musical'</span></span> LIMIT</span> 250</span></span> );</span></span></code></pre> The use of ctid</code> in this example is PostgreSQL system column which provides a unique identifier for each row. By selecting ctid</code> values in the subquery, you can limit the number of rows affected in each iteration. This approach is more efficient than using LIMIT</code> directly in the main query, as it avoids the need to re-scan the table for each batch.</p> If you don't feel comfortable with ctid</code> (which might deserve article on its own) you can use regular lookup by primary key and LIMIT</code>.</p> Planning for Autovacuum</a> </h3> Batching alone doesn’t directly solve the issue of autovacuum catching up with the changes. You’ll need to plan for that separately. Adjusting autovacuum settings or triggering manual VACUUM</code> and VACUUM ANALYZE</code> runs can help manage bloat created during the DELETE process. However, disabling autovacuum is rarely advisable unless you’ve carefully planned for manual maintenance throughout the batch operations. Skipping this step risks leaving behind performance-impacting bloat that will require even more effort to address later.</p> Drop whole range of data with partitioning</a> </h2> Data that is naturally segmented - for example by time of the creation - makes it an excellent candidate for removal through partitioning. Partitioning allows you to bypass DELETE operations altogether by simply dropping or truncating the relevant partitions. This approach is far more efficient and avoids the overhead of scanning, locking, and marking rows as deleted, effectively eliminating the problem with the bloat.</p> While partitioning adds some complexity to schema design and query planning, it can provide significant performance benefits for DELETE-heavy workloads, especially when combined with automated partition management.</p> Conclusion</a> </h2> DELETE operations are often a source of unpleasant surprises - not just by affecting performance and creating bloat, but by striking back at the times we least expect. To handle them effectively, focus on strategies such as batching, monitoring autovacuum, or leveraging partitioning for large datasets. By considering DELETE operations during schema design, you can maintain an efficient database, reduce maintenance headaches, and ensure it continues to run smoothly as your data grows.</p> Text identifiers in PostgreSQL database design 2024-11-09T00:00:00+00:00 Whether you are designing a standalone application or a microservice, you will inevitably encounter the topic of sharing identifiers. Whether it’s URLs of web pages, RESTful API resources, JSON documents, CSV exports, or something else, the identifier of specific resources will be exposed.</p> /orders/123</span></span> /products/345/variants/1</span></span></code></pre> While an identifier is just a number and does not carry any negative connotations, there are valid reasons why you might want to avoid exposing them. These reasons include:</p> Security and Data Exposure</strong>: Numerical identifiers are sequential and predictable, which can expose information about the underlying data source (e.g., the volume of the data) and provide a basis for ID enumeration.</li> Privacy and Confidentiality</strong>: Concerns may arise about concealing the volume of referenced data. For example, the number of customers, clients, or orders might be information a business prefers to keep private.</li> Non-descriptive Nature</strong>: Integers as identifiers can lead to confusion. An ID like 123 does not convey any additional information, making debugging edge cases more challenging.</li> </ol> These and other reasons (like SEO optimisation) have led to the increased use of text-based identifiers. Their readability and versatility make them ideal for external data sharing.</p> However, in database (or data model) design, the advantages of text identifiers are often overshadowed by the problems they introduce. While text identifiers improve interoperability, they frequently come with performance and storage trade-offs. In contrast, integers are naturally faster and more efficient to process, resulting in lower storage requirements and faster indexing, sorting, and searching-tasks that computers are optimised for.</p> In this article, we will explore scenarios where using text identifiers directly in the database design might seem natural and discuss strategies for using them effectively.</p> What Makes Text Identifiers Appealing?</a> </h3> Let’s be honest-text identifiers are popular for a reason. For humans, they are much more readable and, in some cases, add extra context. (Raise your hand if you don’t appreciate Heroku’s quirky and memorable names!) If chosen carefully, they can also be easier to recall.</p> Text identifiers can embed additional context. Consider the order number APAC-20241103-8237, which encodes both the region and the order date. Another popular reason for using text identifiers is their uniqueness in distributed environments.</p> They’re especially handy when people need to interact with them directly. For example, customers copying an order number from an email or support teams discussing an issue benefit from a readable, meaningful identifier. It’s simpler, more intuitive, and less likely to cause headaches when someone tries to recall or share it.</p> When an Identifier Isn’t Just an Identifier</a> </h3> Problems with text identifiers arise when they are used as natural keys in your data or database model. Despite their benefits, text identifiers often make poor primary keys for several reasons:</p> Context Changes</strong>: The additional context provided by text identifiers will likely change, necessitating updates. Despite assurances to the contrary, changes are inevitable.</li> Sorting Issues</strong>: Sorting text identifiers can be tricky, particularly with locale-based sorting or when numbers are embedded within the identifier (e.g., order1434 vs. order349).</li> </ul> The ultimate issue is efficiency:</p> Text identifiers typically require more storage space</strong> than numeric ones. Each character in a text field occupies more bytes than a simple integer, leading to larger database sizes and slower performance, especially during indexing or handling large datasets.</li> Databases are optimised for numeric operations, making searches, joins, and indexing on text fields inherently slower</strong>. This performance gap can significantly affect large datasets, impacting application efficiency.</li> Text identifiers complicate managing relationships between tables</strong>. The additional storage requirements affect not only the source table but also all referencing entities. Multiply this increased storage by the number of referencing tables, and you’ll get a sense of the overall impact.</li> </ul> As databases grow, these issues become more pronounced. The combination of increased storage needs and slower operations contributes to database bloat</strong>, which can degrade system performance. Additionally, bloated indexes can mislead the query planner into making suboptimal choices, such as favouring sequential scans over index scans, complicating regular operations further.</p> Aren't UUIDs the Solution to All This?</a> </h3> It depends. While UUIDs offer numerical representation and are excellent for unique key generation across distributed systems, they are not always the best choice:</p> In comparison to BIGINT, there are few real-world scenarios that necessitate the full range of UUIDs. Premature optimisation most often isn’t worthwhile for most solutions.</li> UUIDs’ 16-byte (128-bit) storage size can be less efficient than many text identifiers. Even BIGINT, at 8 bytes (64 bits), is more storage-efficient. This inefficiency extends to indexes, joins, and other operations.</li> </ul> A matter of opinion: UUIDs are plain ugly.</li> </ul> Moreover, different versions of UUIDs offer varying benefits. For example, UUIDv1 includes a timestamp component, making it somewhat sortable, while UUIDv4 is entirely random. Even sortable UUIDs may not offer significant advantages over more streamlined options like BIGINT or carefully structured text identifiers.</p> In most cases, the trade-offs in performance and readability make UUIDs less appealing unless their globally unique nature is essential across distributed systems.</p> Real-Life Examples</a> </h3> Let’s move beyond theory and explore practical examples:</p> CREATE TABLE</span> sessions</span> (</span></span> token </span>TEXT PRIMARY KEY</span>,</span></span> user_id </span>INT NOT NULL REFERENCES</span> users(user_id),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> products</span> (</span></span> sku </span>TEXT PRIMARY KEY</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> documents</span> (</span></span> document_id </span>TEXT PRIMARY KEY</span>,</span></span> ...</span></span> );</span></span></code></pre> In these cases, using text identifiers as primary keys might seem logical because:</p> They serve as natural keys for the entity.</li> </ul> They are likely propagated outside the service scope, hence needing storage anyway.</li> They are not expected to change.</li> The storage impact for a single table seems negligible.</li> </ul> However, this logic falters under scrutiny. Text identifiers often propagate beyond the service scope, leading to pressure to update them. Updating primary keys is no trivial matter. Consider:</p> While SKUs ideally never change, real-world scenarios like rebranding, product consolidation, or supplier changes can necessitate updates. Despite their appeal, SKUs are poor primary key candidates.</li> Randomly generated text identifiers (like session tokens) will… TBD.</li> </ul> The real issue arises when referencing these identifiers across tables:</p> CREATE TABLE</span> session_logs</span> (</span></span> log_id </span>BIGINT GENERATED ALWAYS AS IDENTITY</span>,</span></span> token </span>TEXT NOT NULL REFERENCES sessions</span>(token),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> product_reviews</span> (</span></span> review_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> product_sku </span>TEXT NOT NULL REFERENCES</span> products(sku),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> customer_orders</span> (</span></span> order_id </span>TEXT PRIMARY KEY</span>,</span></span> customer_id </span>TEXT NOT NULL REFERENCES</span> customers(customer_id),</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> document_revisions</span> (</span></span> revision_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> document_id </span>TEXT NOT NULL REFERENCES</span> documents(document_id),</span></span> ...</span></span> );</span></span></code></pre> In such scenarios, text identifiers can become problematic. Firstly, you lose the ability to modify them</strong>. Secondly, the increased storage requirements</strong> become evident. For instance, an additional 100 bytes per reference across a million records results in an extra 100 MB of storage for just one reference.</p> However, storage isn’t the biggest concern; indexing</strong> is. In PostgreSQL, indexing text fields - whether as primary or foreign keys - requires significantly more space than indexing numeric fields. This leads to bloated indexes, slower lookups, and more fragmented operations, particularly in queries relying on foreign key relationships. As a result, the query planner might resort to full table scans instead of index scans, further degrading performance.</p> These performance issues often remain undetected in development or testing environments but can cause significant disruptions in production.</p> Reintroducing Text Identifiers Effectively</a> </h3> The goal of this post is not to discourage the use of text identifiers entirely but to highlight why they are unsuitable as primary keys. Here are some strategies to handle them effectively:</p> 1. Introduce a Surrogate Key</a> </h4> One straightforward solution is to introduce a surrogate primary key, replacing references to text identifiers:</p> CREATE TABLE</span> products</span> (</span></span> product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> sku </span>TEXT NOT NULL</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE INDEX</span> products_by_sku</span> ON</span> products(sku);</span></span> </span> CREATE TABLE</span> product_reviews</span> (</span></span> review_id </span>SERIAL PRIMARY KEY</span>,</span></span> product_id </span>INT REFERENCES</span> products(product_id),</span></span> ...</span></span> );</span></span></code></pre> This approach maintains efficient retrieval by SKU via an index.</p> 2. Use Mapping Tables for Greater Flexibility</a> </h4> Mapping tables allow you to:</p> Update identifiers without affecting the parent entity.</span></span> * Maintain a history of text identifiers linked to a specific entity.</span></span></code></pre>CREATE TABLE</span> products</span> (</span></span> product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> label </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> product_skus</span> (</span></span> product_sku_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span> product_id </span>INT REFERENCES</span> products(product_id),</span></span> sku </span>TEXT NOT NULL</span>,</span></span> created_at </span>TIMESTAMPTZ DEFAULT</span> CURRENT_TIMESTAMP,</span></span> deleted_at </span>TIMESTAMPTZ</span></span> );</span></span> </span> CREATE UNIQUE INDEX</span> unique_product_skus</span> ON</span> product_skus (product_id, sku) </span>WHERE</span> deleted_at </span>IS NULL</span>;</span></span></code></pre> This approach accommodates changing SKUs without compromising data integrity. A related use case could be linking a mapping table to a product variant:</p> CREATE TABLE</span> product_variants</span> (</span></span> variant_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> product_id </span>INT NOT NULL REFERENCES</span> products(product_id),</span></span> sku </span>TEXT NOT NULL</span>,</span></span> name TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span></code></pre>3. Decompose the Text Identifier’s Meaning</a> </h4> Text identifiers often embed meaningful information, such as regions, dates, or categories. By decomposing the identifier, you can store this contextual data separately, improving both flexibility and performance.</p> For example, instead of storing an order identifier like ORD-EMEA-00789 directly, you can design a more robust schema:</p> CREATE TABLE</span> orders</span> (</span></span> order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> region_id </span>INT NOT NULL REFERENCES</span> regions(region_id),</span></span> order_date </span>DATE NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> CREATE TABLE</span> regions</span> (</span></span> region_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> name TEXT NOT NULL</span>,</span></span> code </span>TEXT NOT NULL</span>,</span></span> ...</span></span> );</span></span> </span> INSERT INTO</span> regions (</span>name</span>, code) </span>VALUES</span></span> (</span>'Europe, the Middle East, and Africa'</span>, </span>'EMEA'</span>),</span></span> (</span>'Asia Pacific'</span>, </span>'APAC'</span>);</span></span> </span> To</span> generate a user</span>-</span>friendly order </span>number</span>, you can </span>create</span> a </span>function</span>:</span></span> </span> CREATE OR REPLACE FUNCTION</span> get_order_number</span>(p_order_id </span>INT</span>)</span></span> RETURNS TEXT AS</span> $$</span></span> DECLARE</span></span> v_region_code </span>TEXT</span>;</span></span> v_formatted_order_id </span>TEXT</span>;</span></span> v_order_number </span>TEXT</span>;</span></span> BEGIN</span></span> SELECT</span> r</span>.</span>code</span> INTO</span> v_region_code</span></span> FROM</span> orders o</span></span> JOIN</span> regions r </span>ON</span> o</span>.</span>region_id</span> =</span> r</span>.</span>region_id</span></span> WHERE</span> o</span>.</span>order_id</span> =</span> p_order_id;</span></span> </span> IF NOT</span> FOUND </span>THEN</span></span> RETURN NULL</span>;</span></span> END IF</span>;</span></span> </span> v_formatted_order_id :</span>=</span> TO_CHAR(p_order_id, </span>'FM00000'</span>);</span></span> v_order_number :</span>=</span> 'ORD-'</span> \|\|</span> v_region_code </span>\|\|</span> '-'</span> \|\|</span> v_formatted_order_id;</span></span> </span> RETURN</span> v_order_number;</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql;</span></span></code></pre> This method enables you to store and retrieve structured data efficiently while still providing a readable and meaningful identifier for external use.</p> 4. Reversible Text IDs</a> </h4> For scenarios requiring enhanced security or privacy, where identifiers should not be easily enumerable or predictable, you can use reversible text-based IDs. These allow you to present users with seemingly random text representations while maintaining efficient numerical storage internally.</p> Sqids</a></strong> is a project that can help achieve this. It generates URL-friendly unique identifiers from numbers and can encode multiple numerical identifiers into a single string. Here’s an example using the Sqids project:</p> [42] -> JgaEBgznCpUZo3Kk</span></span> [42, 430004] -> lTiYlvsGkh59m1PQ</span></span></code></pre> The generated identifiers are reversible with knowledge of the shared alphabet, allowing you to decode requests without hitting the database, which can be beneficial in high-throughput environments. This technique is particularly useful for user, account, or session identifiers, balancing the need for security with operational efficiency.</p> However, it’s crucial to remember that this is not a substitute for robust security practices. Proper authentication and authorisation mechanisms are still necessary to secure your application.</p> Use text identifiers where they genuinely help, and stick to numeric primary keys where they don't - that keeps the schema both efficient and maintainable.</p> 5. Leverage Generate columns</a> </h4> In edge case scenarios, where keeping text identifiers with embedded context is necessary, generated columns</strong> in PostgreSQL can be a valuable feature. Since version 12, PostgreSQL allows defining columns whose values are automatically computed from other columns in the table. This ensures consistency without manual intervention.</p> For example, you can define a function to handle the formatting logic:</p> CREATE OR REPLACE FUNCTION</span> get_formatted_order_id</span>(p_region_id </span>INT</span>, p_order_id </span>INT</span>)</span></span> RETURNS TEXT AS</span> $$</span></span> DECLARE</span></span> v_region_code </span>TEXT</span>;</span></span> BEGIN</span></span> v_region_code :</span>= CASE</span> p_region_id</span></span> WHEN</span> 1</span> THEN</span> 'EMEA'</span></span> WHEN</span> 2</span> THEN</span> 'APAC'</span></span> ELSE</span> 'OTHER'</span></span> END</span>;</span></span> </span> RETURN</span> 'ORD-'</span> \|\|</span> v_region_code </span>\|\|</span> '-'</span> \|\|</span> LPAD(p_order_id::</span>TEXT</span>, </span>5</span>, </span>'0'</span>);</span></span> END</span>;</span></span> $$ </span>LANGUAGE</span> plpgsql IMMUTABLE;</span></span></code></pre> The function is marked IMMUTABLE because PostgreSQL requires generated columns to use only immutable functions-those guaranteed to return the same result for the same input every time.</p> CREATE TABLE</span> orders</span> (</span></span> order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span> region_id </span>INT NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL</span>,</span></span> formatted_order_id </span>TEXT GENERATED ALWAYS AS</span> (</span></span> get_formatted_order_id(region_id, order_id)</span></span> ) STORED</span></span> );</span></span></code></pre> This setup ensures that formatted_order_id</code> is automatically updated whenever region_id</code> or order_id</code> changes. The column is physically stored, enhancing read performance for frequently queried data.</p> Using generated columns can simplify maintaining consistency in derived values like formatted text identifiers. However, be mindful of their characteristics, such as:</p> Automated Updates</strong>: The system automatically recomputes the column’s value when the referenced columns change.</li> Immutability Requirements</strong>: Only immutable functions can be used, ensuring reliable and consistent computation.</li> </ul> Wrapping up</a> </h3> Text identifiers will stay with us, and that's great. They’re human-readable, memorable, and can pack a lot of meaningful context into a simple string. They make external interactions smoother, whether it’s a customer referencing an order number or a support team tracing an issue. They even add a bit of charm and personality to otherwise sterile identifiers.</p> However it's important to keep their usage in control. The nice rule I've heard is</p> Use text identifiers when communicating externally</strong> - for example, in URLs or API responses</li> Internally, always rely on numerical IDs</strong> - surrogate keys like INT or BIGINT (even UUID if you go for planet dominanance) where necessary, to maintain database efficiency and integrity.</li> </ul> This approach allows you to leverage the strengths of text identifiers for external communication while keeping your database optimised for performance and scalability. By storing text identifiers in dedicated fields and using numeric primary keys for internal operations, you achieve a the right balance and do best to maintain the system performance.</p> We need to talk about ENUMs 2024-09-04T00:00:00+00:00 Designing a database schema, whether for a new application or a new feature, always raises a lot of questions. The choices you make can have a big impact on how well your database performs and how easy it is to maintain and scale. Whether you’re just getting started with PostgreSQL or consider yourself a seasoned pro, it’s easy to rely on old habits or outdated advice. In this article, I want to take a fresh look at one of those topics that often sparks debate: the use of ENUMs in PostgreSQL.</p> I have to admit, not so long ago, I would advise "don't use ENUMs" without thinking about it too much. Relying only on random articles and some personal but outdated experience, this had been my go-to answer for some years. And while there were several limitations of ENUMs in PostgreSQL in the (distant) past, the support has improved a long time ago.</p> The improvements are so long-standing that:</p> PostgreSQL 9.1 (released in 2011) introduced the option to add new values to ENUMs without a table rewrite</strong>. From what I can say this fact alone remained the source of biggest misconception when thinking about ENUMs.</li> Renaming of values</strong> was added in version 10 (2017).</li> The ability to ALTER TYPE ... ADD VALUE</code> in a transaction block was introduced in PostgreSQL 12 (2019).</li> </ul> And new features coming (at the time of writing this article) with PostgreSQL 17</strong> allow the use of newly added values within the same transaction block (previously not possible without an explicit commit).</p> Therefore, I want to correct even myself and say—let's give ENUMs another chance. This article will go into detail and help us correctly decide when it makes sense to use them or not.</p> How are ENUMs Implemented?</a> </h2> Every stored ENUM value occupies 4 bytes on disk. This is the size of the OID, representing the actual ENUM value, stored as a row within the pg_enum</code> table. You can see this yourself by querying the table directly, but it will come by default with no data defined.</p> # select * from pg_catalog.pg_enum;</span></span> </span> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> -----+-----------+---------------+-----------</span></span> (0 rows)</span></span></code></pre> But once you define a new ENUM data type, using:</p> CREATE TYPE</span> order_status</span> AS</span> ENUM (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>);</span></span></code></pre> You will get the actual OIDs.</p> # select * from pg_catalog.pg_enum;</span></span> </span> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> --------+-----------+---------------+------------</span></span> 211018 \| 211016 \| 1 \| new</span></span> 211020 \| 211016 \| 2 \| pending</span></span> 211022 \| 211016 \| 3 \| processing</span></span> 211024 \| 211016 \| 4 \| shipped</span></span> 211026 \| 211016 \| 5 \| delivered</span></span></code></pre> The table straightforwardly identifies both the type and individual values, but also the enumsortorder</code>, which is the foundation for BEFORE/AFTER</code> new value(s) definition.</p> ALTER TYPE</span> order_status </span>ADD VALUE</span> 'cancelled'</span> BEFORE</span> 'shipped'</span>;</span></span></code></pre> Resulting in:</p> oid \| enumtypid \| enumsortorder \| enumlabel</span></span> --------+-----------+---------------+------------</span></span> 211018 \| 211016 \| 1 \| new</span></span> 211020 \| 211016 \| 2 \| pending</span></span> 211022 \| 211016 \| 3 \| processing</span></span> 211024 \| 211016 \| 4 \| shipped</span></span> 211026 \| 211016 \| 5 \| delivered</span></span> 211027 \| 211016 \| 3.5 \| cancelled</span></span></code></pre> To confirm the actual storage size of the ENUM value, you can use a sample table, the previously defined type order_status</code>, a single data row, and the pg_column_size</code> function.</p> CREATE TABLE</span> orders</span> (</span></span> id </span>SERIAL PRIMARY KEY</span>,</span></span> status</span> order_status </span>NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE</span></span> );</span></span> </span> INSERT INTO</span> orders (</span>status</span>, order_date) </span>VALUES</span></span> (</span>'new'</span>, </span>'2024-09-01'</span>);</span></span></code></pre> This gives you confirmation of the actual 4-byte storage.</p> # SELECT id, status, pg_column_size(status) AS status_size</span></span> </span> id \| status \| status_size</span></span> ----+--------+-------------</span></span> 1 \| new \| 4</span></span></code></pre> Returning to the pg_enum</code> table, the structure of the table provides other important clues:</p> enumlabel</code> is of type name</code>, which is effectively a 63-byte varchar for storing system identifiers.</li> The UNIQUE CONSTRAINT pg_enum_typid_label_index</code> prevents us from creating two values with the same name.</li> However, given the nature of VARCHAR</code>, it is case-sensitive, allowing you to define new</code> and NEW</code> as two distinct values.</li> </ul> Life with ENUMs</a> </h2> Now that we understand how ENUMs are implemented, let's review the lifecycle of such a data type in a typical database.</p> As demonstrated above, creation of new ENUMs</strong> is simple, and since version 12 (and expanded in 17), it's possible even in transaction blocks, making it easier to work with database migration tools without needing to explicitly worry about transaction management.</p> Adding new values</strong> does not require a table rewrite and can be done without further hesitation. You can also safely rename existing values</strong>. Similarly, you can use ENUMs in arrays, indexes, or compare them as needed (based on their order).</p> ALTER TYPE name ADD VALUE</span> [ IF NOT EXISTS ] new_enum_value [ { BEFORE \| AFTER } neighbor_enum_value ]</span></span> </span> ALTER TYPE name</span> RENAME </span>VALUE</span> existing_enum_value </span>TO</span> new_enum_value</span></span></code></pre> It's also important to mention that ENUMs allow you to use a default value, like this:</p> status order_status NOT NULL DEFAULT 'new'</span></span></code></pre> Unfortunately, this is where the flexibility ends at the moment. There's no easy way to remove or re-order existing values (please, don't modify pg_enum</code>—or any system tables, for that matter). The topic of removing existing ENUM value(s) can easily open up a whole discussion about the best way to 'deprecate' and drop a value over time. The only suitable way to perform these operations is by creating a new type and altering the column. Please be aware of the ways described in How not to change PostgreSQL column type</a></strong> will apply in this scenario.</p> Using CHECK Constraints as an Alternative to ENUMs</a> </h2> While ENUMs are a powerful tool in PostgreSQL for enforcing a fixed set of values in a column, they come with the limitations mentioned above. An alternative approach to achieving similar functionality is to use a CHECK constraint with a TEXT column. This method offers more flexibility when managing the allowed values, especially in scenarios where the set of valid values might change frequently.</p> A CHECK constraint can be applied to a column to enforce that its value must be one of a predefined set of values. Here’s how you can define a table using a CHECK constraint as an alternative to an ENUM:</p> CREATE TABLE</span> orders</span> (</span></span> id </span>SERIAL PRIMARY KEY</span>,</span></span> status TEXT NOT NULL</span>,</span></span> order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE,</span></span> CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>))</span></span> );</span></span></code></pre> Modification of the constraints is possible, but you will always have to DROP CONSTRAINT</code> and then ADD CONSTRAINT</code>.</p> ALTER TABLE</span> orders </span>DROP CONSTRAINT</span> orders_status_check;</span></span> </span> ALTER TABLE</span> orders </span>ADD CONSTRAINT</span> orders_status_check </span>CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>, </span>'returned'</span>));</span></span></code></pre> While this operation is not as 'heavyweight' as changing the data type, please be aware that an ACCESS EXCLUSIVE</strong> lock will be acquired for the entire table—meaning no other operations can be performed on the table. With a growing table size and increasing concurrency, this might still lead to application-level downtime.</p> Another limitation is the lack of direct support for sorting, meaning you will have to implement it manually in every query or abstract the sorting detail using a view.</p> SELECT</span> id, </span>status</span>, order_date</span></span> FROM</span> orders</span></span> ORDER BY</span></span> ARRAY_POSITION(</span></span> ARRAY</span>['new', 'pending', 'processing', 'shipped', 'delivered', 'returned'],</span></span> status</span></span> );</span></span></code></pre>Reference Tables as the Real Alternative</a> </h2> It might be surprising, but one of the motivations for using either ENUMs or CHECK constraints is to avoid additional JOINs. It's no surprise that JOINs are a source of pain for many people who struggle with SQL, and therefore they try to limit their use. While there might be negligible impact on performing such an operation, there are no other reasons not to use reference tables—it’s not storage (unless you over-optimise with smallint</code> and extremely large datasets) and definitely not operational considerations.</p> In fact, reference tables are often better in most cases. They not only address all the use cases supported by both ENUMs and CHECK constraints, but they can also resolve the limitations of these methods.</p> Creating and Using a Reference Table</a> </h3> Let’s create a reference table similar to the ENUM data type we’ve discussed above:</p> CREATE TABLE</span> order_statuses</span> (</span></span> status_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> status_name </span>TEXT NOT NULL</span></span> );</span></span> </span> CREATE UNIQUE INDEX</span> unique_status_name</span> ON</span> order_statuses (</span>LOWER</span>(status_name));</span></span> </span> INSERT INTO</span> order_statuses (status_name) </span>VALUES</span></span> (</span>'new'</span>),</span></span> (</span>'pending'</span>),</span></span> (</span>'processing'</span>),</span></span> (</span>'shipped'</span>),</span></span> (</span>'delivered'</span>),</span></span> (</span>'cancelled'</span>);</span></span></code></pre>Updating the Orders Table to Use the Reference Table</a> </h3> To use the reference table, you’ll need to alter your existing orders</code> table. This involves dropping the current status</code> column if it’s using an ENUM or CHECK constraint, and replacing it with a foreign key that references the order_statuses</code> table:</p> ALTER TABLE</span> orders </span>DROP</span> COLUMN </span>IF EXISTS status</span>;</span></span> </span> ALTER TABLE</span> orders </span>ADD</span> COLUMN status_id </span>INT</span>,</span></span> ADD CONSTRAINT</span> fk_order_status</span></span> FOREIGN KEY</span> (status_id) </span>REFERENCES</span> order_statuses(status_id);</span></span></code></pre>Impact of Using Reference Tables</a> </h3> While reference tables do not provide anything for free, such as ENUM ordering or simple CHECK constraints, they offer far greater flexibility. Here are some examples of what you can achieve with reference tables:</p> Custom Sorting Logic:</strong> You can implement specific sorting by adding an additional sort_order</code> column to the order_statuses</code> table. This allows you to easily change the order of statuses without altering the schema.</li> Deprecating Values:</strong> By adding a deleted_at</code> column (or a similar field), you can deprecate certain status values. Combined with a trigger, you can prevent these deprecated values from being used in new data while maintaining historical records.</li> Renaming Values:</strong> Unlike ENUMs, renaming a status is straightforward - simply update the value in the order_statuses</code> table.</li> Internationalisation:</strong> If your application needs to support multiple languages, you can extend the order_statuses</code> table with additional columns for different languages or even create a separate lookup table that stores translations.</li> Additional Metadata:</strong> Reference tables can store extra information about each status, such as descriptions, colours (for UI purposes), or associated icons, which can be particularly useful in applications with rich user interfaces.</li> </ul> Having mentioned all advantages of using Reference Tables, it's also important to mention the fact it's not straightforward to set the default values for such a references. Definitely not in obvious way which ENUMs provide. The real alternative is to create table/column without default value and alter it specifically for given database using ALTER COLUMN ... SET DEFAULT</code></p> ALTER TABLE</span> orders</span></span> ALTER</span> COLUMN status_id </span>SET DEFAULT</span> (</span>SELECT</span> status_id </span>FROM</span> order_statuses </span>WHERE</span> status_name </span>=</span> 'new'</span>);</span></span></code></pre>Performance Considerations: ENUMs, CHECK Constraints, and Reference Tables</a> </h2> Thanks to their internal representation as OIDs, ENUMs offer predictable performance and indexing strategies, often outperforming text-based CHECK constraints. Since ENUM comparisons rely on integers, they tend to be faster than string comparisons required for CHECK constraints. Reference tables, on the other hand, inherently require a JOIN, which introduces the overhead of accessing two tables and potentially working with multiple indexes.</p> While I initially planned to publish benchmarks comparing the performance of all three approaches, I decided not to include specific numbers. It’s true that ENUMs consistently performed the fastest, and reference tables were somewhat slower due to the necessary JOIN. However, as with most benchmarks, the scenarios were artificial. The actual performance difference, though measurable, is unlikely to be significant in practical applications. In any reasonably complex query, other factors are far more likely to impact performance than whether you use ENUMs or not.</p> Conclusion: Use ENUMs Where They Make Sense</a> </h2> As with many decisions in software development, the use of ENUMs comes down to "it depends." This article has provided clarity on how ENUMs work, where their limitations lie, and how to make an informed decision about when to use them.</p> To recap:</p> ENUMs</strong> are ideal for small, static sets of values that require strict type enforcement and minimal changes, preferably data types that are not directly tied to the business logic.</li> CHECK constraints</strong> offer more flexibility than ENUMs, making them a good choice when the allowed values might change, but the data type remains simple, and there's no</li> Reference tables</strong> provide the most flexibility and scalability, making them the go-to choice for dynamic or complex value sets and when you want to avoid the limitations of ENUMs and CHECK constraints.</li> </ul> But in the end, no plan ever survives first contact with reality. It’s essential to revisit and reconsider your decisions as your application evolves, ensuring that your database schema continues to serve the needs of your project as effectively as possible.</p> And while it's not meant as advice - personally I'm still going to gravitate towards the advice "don't use ENUMs" though. I'm yet to see value-based field that does not change over time.</em></p> Beyond Simple Upserts with MERGE in PostgreSQL 2024-08-25T00:00:00+00:00 Understanding how comfortable someone is with databases and SQL often comes down to the features they use. In PostgreSQL, one such feature that distinguishes more advanced users is the MERGE</code> command, introduced in version 15 and expanded in version 17 (in beta at the time of writing this article). Before MERGE</code>, developers typically relied on INSERT ... ON CONFLICT DO UPDATE</code> for upserts—a method introduced in PostgreSQL 9.5 that has since become a staple in many developers' toolkits.</p> While ON CONFLICT</code> offers a straightforward solution for simple upsert scenarios, it can quickly become limiting as business logic grows in complexity. This is where the MERGE</code> command excels. Introduced in the SQL:2003 standard, MERGE</code> allows for more sophisticated data synchronisation tasks by combining multiple operations—such as conditional inserts, updates, and deletes—into a single, atomic statement.</p> In this article, we’ll explore the capabilities of the MERGE</code> command, comparing it with traditional upsert methods and examining how it can streamline database operations. Through practical examples, we'll illustrate how MERGE</code> can simplify even the most complex workflows, making it an indispensable tool for developers working with PostgreSQL.</p> Hands-on Example: Managing a Score Points System for a Mobile Game</a> </h2> Imagine you’re managing a tournament score system for an online game, where players earn special status by participating in weekly tournaments. The simple business logic would be:</p> Players can earn score points by participating in the tournament.</li> Players must participate in all tournaments to maintain their score (i.e., maintain the streak).</li> Players who participate in a second consecutive tournament can achieve the status of veteran</code> (if they reach 1,000 score points) or gain the status of star</code> (if they reach 100 or more score points).</li> </ul> The score points are updated weekly, based on the outcome of the previous week's participation.</p> We will start with a simple database schema consisting of a table tournament_scores</code> that tracks (as the name suggests) only active users.</p> CREATE TABLE</span> tournament_scores</span> (</span></span> player_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span> player_name </span>TEXT UNIQUE NOT NULL</span>,</span></span> score </span>INT NOT NULL DEFAULT</span> 0</span>,</span></span> status TEXT NOT NULL DEFAULT</span> 'newbie'</span> CHECK</span> (</span>status IN</span> (</span>'newbie'</span>, </span>'veteran'</span>, </span>'star'</span>))</span></span> );</span></span></code></pre> And populate it with some sample data:</p> INSERT INTO</span> tournament_scores (player_name, score, </span>status</span>)</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>900</span>, </span>'newbie'</span>), </span>-- Regular player, close to Veteran promotion</span></span> (</span>'PlayerTwo'</span>, </span>1200</span>, </span>'veteran'</span>), </span>-- Already a Veteran player</span></span> (</span>'PlayerThree'</span>, </span>300</span>, </span>'newbie'</span>); </span>-- Regular player with a lower score</span></span></code></pre>Upsert using ON CONFLICT</code></a> </h2> When activity data is received, there are multiple ways to process it. ON CONFLICT</code> is used here for demonstration purposes.</p> INSERT INTO</span> tournament_scores (player_name, score)</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ON</span> CONFLICT (player_name)</span></span> DO </span>UPDATE SET</span></span> score </span>=</span> tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>,</span></span> status = CASE WHEN</span> (</span>tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span>;</span></span></code></pre> This performs the basic requirements, updating the scores and possibly evaluating the status for existing tournament users. To remove users who no longer participated (and hence broke the streak) and apply other conditional logic, you would need to use separate statements.</p> Handling Upserts with MERGE</code></a> </h2> Now, let’s introduce the MERGE</code> command and implement the same logic as above.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span></span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span> status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre> Here, we define the table tournament_scores</code> as the target</strong> and the list of VALUES as the source</strong>. Using a conditional clause, we define the logic for both matched and unmatched entries, giving us both INSERT and UPDATE paths.</p> The evaluation paths in this case are called when_clauses</code>. The MERGE</code> statement allows you to specify multiple clauses with different conditions, for example, allowing you to award users star</code> status if they gain more than 100 score points within a given tournament.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span></span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span> status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre> Here’s a summary of how it works:</p> The MERGE</code> command evaluates the when_clauses</code> in the order they are written.</li> If a row matches the condition specified in a clause, the action defined in that clause is performed, and the row is no longer eligible to be matched against subsequent when_clauses</code>.</li> </ol> While we introduced multiple evaluation paths, the MERGE</code> command in PostgreSQL goes beyond that and expands the WHEN NOT MATCHED</code> clause, which effectively becomes WHEN NOT MATCHED [BY TARGET]</code>. PostgreSQL allows you to specify WHEN NOT MATCHED BY SOURCE</code> to perform the necessary merge statement for the data not present in the source table.</p> Handling DELETEs with MERGE</code></a> </h2> The functionality completely missing from regular upserts with ON CONFLICT</code> is the ability to delete missing entries. In our sample scenario, we want to penalise players who broke their streak and did not participate in last week's tournament by effectively deleting their entries from the target table.</p> MERGE INTO</span> tournament_scores ts</span></span> USING</span> (</span></span> VALUES</span></span> (</span>'PlayerOne'</span>, </span>50</span>), </span>-- Add points for PlayerOne</span></span> (</span>'PlayerTwo'</span>, </span>120</span>), </span>-- Add points for PlayerTwo</span></span> (</span>'PlayerFour'</span>, </span>70</span>) </span>-- New player without an account, PlayerFour</span></span> ) </span>AS</span> v(player_name, score_added)</span></span> ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span> WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span> WHEN MATCHED THEN</span></span> UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span></span> WHEN NOT MATCHED THEN</span></span> INSERT</span> (player_name, score)</span></span> VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>)</span></span> WHEN NOT MATCHED BY</span> SOURCE </span>THEN</span></span> DELETE</span>;</span></span></code></pre> The introduction of the DELETE</code> merge operation complements all possible MERGE</code> outcomes:</p> The first is merge_update</code>, applicable to WHEN MATCHED</code> and WHEN NOT MATCHED</code> clauses, consisting of regular UPDATE</code> operations.</li> The second is merge_insert</code> for the WHEN NOT MATCHED</code> clause, allowing (as the name implies) data to be inserted.</li> The DELETE</code> we introduce is part of merge_delete</code>.</li> </ul> Technically, there's also the ability to specify DO NOTHING</code> for all available when_clauses</code>, similar to upserts using ON CONFLICT</code>.</p> Using MERGE</code> Output</a> </h2> Starting from PostgreSQL 17, the MERGE</code> command has been extended to support RETURNING</code> clauses to process merged data further. When an INSERT</code> or UPDATE</code> action is performed, the new values of the target table's columns are used. When a DELETE</code> is performed, the old values of the target table's columns are used.</p> Conclusion</a> </h2> The MERGE</code> command significantly enhances the ability to handle complex INSERT</code> and UPDATE</code> logic by allowing multiple operations within a single query while also capturing even the most intricate scenarios. In this article, we have explored the syntax and common use cases of MERGE</code>. For further exploration, note that the source</code> dataset is where you can perform any data transformations required, making MERGE</code> particularly well-suited for ETL operations, data archival, cleanup tasks, and more.</p> Incorporating MERGE</code> into your PostgreSQL toolkit can simplify database operations, reduce the risk of data inconsistencies, and streamline your</p> Gentle Introduction to Window Functions in PostgreSQL 2024-07-07T00:00:00+00:00 Understanding the relationship between data points is crucial. For instance, you might need to identify the most recent orders for each customer or track changes in sensor readings over time. Unlike aggregate functions, which summarise data into a single row, it is window functions that allow you to analyse data while preserving each row’s details. This is the core of the logic, but don’t worry if you struggle to imagine the difference, as we will cover all of it in this article.</p> PostgreSQL supports SQL window functions, facilitating complex calculations across related rows within a table. These functions are particularly useful for tasks such as ranking entries, calculating running totals, finding moving averages, and comparing individual entries. Mastering window functions can significantly enhance your data analysis capabilities.</p> You can easily use similar window functions as you would with aggregation. Let’s take a simple example of sensor readings:</p> CREATE TABLE</span> sensor_readings</span> (</span></span> sensor_id </span>bigint</span>,</span></span> reading_value </span>decimal</span>,</span></span> reading_time </span>timestamp with time zone default</span> current_timestamp</span></span> );</span></span> </span> INSERT INTO</span> sensor_readings (sensor_id, reading_value, reading_time) </span>VALUES</span></span> (</span>1</span>, </span>32</span>.</span>7</span>, </span>'2024-07-01 11:24:34'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 11:29:01'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 12:03:59'</span>),</span></span> (</span>1</span>, </span>33</span>.</span>0</span>, </span>'2024-07-03 10:12:15'</span>),</span></span> (</span>2</span>, </span>32</span>.</span>8</span>, </span>'2024-07-01 13:17:01'</span>),</span></span> (</span>2</span>, </span>35</span>.</span>8</span>, </span>'2024-07-02 09:18:11'</span>),</span></span> (</span>3</span>, </span>29</span>.</span>1</span>, </span>'2024-07-01 13:54:03'</span>),</span></span> (</span>3</span>, </span>30</span>.</span>3</span>, </span>'2024-07-01 14:12:09'</span>),</span></span> (</span>3</span>, </span>31</span>.</span>5</span>, </span>'2024-07-02 16:07:43'</span>);</span></span></code></pre> When you start with the aggregation functions, it is straightforward for anybody familiar with SQL:</p> SELECT</span></span> sensor_id,</span></span> AVG</span>(reading_value)</span></span> FROM</span> sensor_readings</span></span> GROUP BY</span> sensor_id;</span></span></code></pre> with the output</p> sensor_id \| avg</span></span> -----------+---------------------</span></span> 3 \| 30.3000000000000000</span></span> 2 \| 34.3000000000000000</span></span> 1 \| 32.9750000000000000</span></span></code></pre> While similar, using the window function AVG we can get almost same result:</p> SELECT</span></span> sensor_id,</span></span> reading_value,</span></span> AVG</span>(reading_value) </span>OVER</span> (</span>PARTITION BY</span> sensor_id) </span>AS</span> avg_reading_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> such as</p> sensor_id \| reading_value \| avg_reading_value</span></span> -----------+---------------+---------------------</span></span> 1 \| 32.7 \| 32.9750000000000000</span></span> 1 \| 33.1 \| 32.9750000000000000</span></span> 1 \| 33.1 \| 32.9750000000000000</span></span> 1 \| 33.0 \| 32.9750000000000000</span></span> 2 \| 32.8 \| 34.3000000000000000</span></span> 2 \| 35.8 \| 34.3000000000000000</span></span> 3 \| 29.1 \| 30.3000000000000000</span></span> 3 \| 30.3 \| 30.3000000000000000</span></span> 3 \| 31.5 \| 30.3000000000000000</span></span></code></pre> This reiterates the fundamental difference between the two sets of functions. As mentioned earlier, while the results for sensor_id</code> are the same in both cases, the aggregate function</strong> summarised it into a single row (grouped by sensor_id</code>), whereas the window function</strong> provides the value for the set of rows in the partition defined by sensor_id</code>.</p> After showing the average in a window function, it’s important to note that traditional aggregation functions might not be the most helpful in the context of window function logic. Despite functions like AVG</code>, COUNT</code>, and SUM</code>—which are the most used aggregation functions—being available as window functions, we used the above only to demonstrate the difference.</p> OVER clause</a> </h2> Before diving into the individual functions, let’s cover the syntax first. From the sample query above, you already get the basic syntax of the window functions, with the OVER</code> clause being a primary identification of windowing functionality.</p> window_function ([expression...]) OVER window_definition</span></span></code></pre> In our example, the basic window functions can be similar to the aggregate ones— like AVG</code>, SUM</code> and COUNT</code> - and a number of the functions we will cover shortly.</p> The window definition part specifies how the window function will see the data it works over. It can include:</p> Partitioning</strong> to divide the result sets into separate partitions (think groups in aggregate functions).</li> Definition of the ordering</strong> of the rows within each partition.</li> Specification of how to apply framing</strong> of the subset of rows for each row’s calculation.</li> </ol> From all the components of the window syntax, only partitioning is mandatory.</p> If we revisit our first window function example above, you can identify the partitioning</strong> part (PARTITION BY sensor_id</code>). As already mentioned, you can easily compare the partitioning logic to the grouping used in aggregate functions, with the same properties—like partitioning data segments by multiple fields, using functions and other logic.</p> With partitioning comes hand in hand ordering</strong> of the data. When you start thinking of row properties, instead of grouping data into a single row, order starts to make a difference. Let’s take the following example:</p> window_function ([expression...]) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time)</span></span></code></pre> This will provide different data compared to unsorted ones. If you struggle to find the application for this, think of the moving average or row numbering.</p> The last component of the window definition is framing</strong>, allowing you to further limit the data over which the window function is calculated. There are two framing expressions: ROWS BETWEEN</code> and RANGE BETWEEN</code>. Using the framing, we can turn AVG from the above example to provide our first real use case when you might want to use window functions.</p> SELECT</span></span> reading_time,</span></span> sensor_id,</span></span> reading_value,</span></span> AVG</span>(reading_value) </span>OVER</span> (</span></span> PARTITION BY</span> sensor_id</span></span> ORDER BY</span> reading_time</span></span> ROWS BETWEEN</span> 1</span> PRECEDING AND</span> 1</span> FOLLOWING</span></span> ) </span>AS</span> moving_avg_reading_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> Implementing the moving average of the individual readings, taking into account a maximum of 3 rows including the current one.</p> reading_time \| sensor_id \| reading_value \| moving_avg_reading_value</span></span> ------------------------+-----------+---------------+--------------------------</span></span> 2024-07-01 11:24:34+02 \| 1 \| 32.7 \| 32.9000000000000000</span></span> 2024-07-02 11:29:01+02 \| 1 \| 33.1 \| 32.9666666666666667</span></span> 2024-07-02 12:03:59+02 \| 1 \| 33.1 \| 33.0666666666666667</span></span> 2024-07-03 10:12:15+02 \| 1 \| 33.0 \| 33.0500000000000000</span></span> 2024-07-01 13:17:01+02 \| 2 \| 32.8 \| 34.3000000000000000</span></span> 2024-07-02 09:18:11+02 \| 2 \| 35.8 \| 34.3000000000000000</span></span> 2024-07-01 13:54:03+02 \| 3 \| 29.1 \| 29.7000000000000000</span></span> 2024-07-01 14:12:09+02 \| 3 \| 30.3 \| 30.3000000000000000</span></span> 2024-07-02 16:07:43+02 \| 3 \| 31.5 \| 30.9000000000000000</span></span></code></pre>Exploring Window functions</a> </h2> Now that we have a foundational understanding of window functions and their components, let’s dive into specific window functions. We’ll cover some of the most commonly used functions, such as ROW_NUMBER()</code>, RANK()</code>, DENSE_RANK()</code>, LAG()</code>, LEAD()</code>, FIRST_VALUE()</code>, LAST_VALUE()</code> and more. For each function, we’ll provide examples to illustrate their practical applications.</p> ROW_NUMBER</a> </h3> The ROW_NUMBER()</code> function assigns a unique sequential integer to rows within a partition of a result set, starting with one for the first row in each partition.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> ROW_NUMBER</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> row_num</span></span> FROM</span> sensor_readings;</span></span></code></pre> This makes it easy to find the first/last entries for a specified window. As an example, you can experiment with getting only the last reading for each hour.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value</span></span> FROM</span> (</span></span> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> ROW_NUMBER</span>() </span>OVER</span> (</span></span> PARTITION BY</span> sensor_id, date_trunc(</span>'hour'</span>, reading_time)</span></span> ORDER BY</span> reading_time </span>DESC</span></span> ) </span>AS</span> row_num</span></span> FROM</span> sensor_readings</span></span> ) </span>AS</span> ranked_readings</span></span> WHERE</span> row_num </span>=</span> 1</span>;</span></span></code></pre>RANK and DENSE_RANK</a> </h3> The RANK()</code> and DENSE_RANK()</code> functions are used to assign a rank to each row within a partition, based on the order of one or more values. These functions are useful when scoring the values and handling ties. The main difference between RANK</code> and DENSE_RANK</code> is how they deal with gaps.</p> Example use to find the highest reading_value per sensor:</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> RANK</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_value </span>DESC</span>) </span>AS</span> rank</span></span> FROM</span> sensor_readings;</span></span></code></pre> If you consider the sorting for the sensor_id</code> 1 in our sample seed the difference between RANK</code> and DENSE_RANK</code> is easy to demonstrate.</p> sensor_id \| reading_time \| reading_value \| rank</span></span> -----------+------------------------+---------------+------</span></span> 1 \| 2024-07-02 11:29:01+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-02 12:03:59+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-03 10:12:15+02 \| 33.0 \| 3</span></span> 1 \| 2024-07-01 11:24:34+02 \| 32.7 \| 4</span></span></code></pre> Giving a natural ranking with tie on the reading value 33.1 and leaving a gap on 2nd rank, whereas DENSE_RANK</code> wouldn't include the gap.</p> sensor_id \| reading_time \| reading_value \| rank</span></span> -----------+------------------------+---------------+------</span></span> 1 \| 2024-07-02 11:29:01+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-02 12:03:59+02 \| 33.1 \| 1</span></span> 1 \| 2024-07-03 10:12:15+02 \| 33.0 \| 2</span></span> 1 \| 2024-07-01 11:24:34+02 \| 32.7 \| 3</span></span></code></pre>LAG</code> and LEAD</code></a> </h3> To evaluate previous or subsequent rows without the need to self-join the dataset, you can utilise the power of the window functions LAG</code> and LEAD</code>. These functions are particularly useful for calculating differences between rows, comparing current and previous values, or fetching future values for comparison.</p> The LAG</code> function provides access to a value in a previous row within the partition, while LEAD</code> provides access to a subsequent row.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> LAG</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> previous_value,</span></span> LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> next_value</span></span> FROM</span> sensor_readings;</span></span></code></pre> In this query, we are using a value of 1, but you can choose any position necessary. By using LAG()</code> and LEAD()</code>, you can perform advanced analyses that require looking backward or forward within your dataset, making it easier to derive meaningful insights and trends.</p> FIRST_value</code> and LAST_VALUES</code></a> </h3> The functions FIRST_value</code> and LAST_VALUES</code> are similar, except they (as the name says) give the first/last value of the partition.</p> The FIRST_VALUE</code> is useful when comparing the partition values to the first value (for example opening price for a day), and LAST_VALUE</code> to identity the closing values.</p> Other Window functions</a> </h3> The complete list of the window functions is available in the documentation</a>.</p> Re-using the window definition</a> </h3> As you might have noticed, the query we used to demonstrate LAG</code> and LEAD</code> was rather verbose. This was due to the repeated definition of the window for both columns. Luckily, the syntax of the window functions allows you to define and re-use the window definition.</p> SELECT</span></span> sensor_id,</span></span> reading_time,</span></span> reading_value,</span></span> LAG</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> previous_value,</span></span> LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> next_value</span></span> FROM</span> sensor_readings</span></span> WINDOW</span> readings_window </span>AS</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time);</span></span></code></pre> This way you can ensure the consistent window definition and consistency in complex queries.</p>

Same rows, different SUM

2026-06-28T21:25:00+00:00

Everyone knows not to store money as a double precision</code>. One can hope. The rule is so well drilled that it has stopped being interesting, and it is also not where the trouble usually starts. The float is already in the schema before anyone weighs in on it: a measurement column someone later sums for a report, telemetry that drifts into a finance dashboard, a third-party feed ingested as double precision</code> because that is how it arrived.</p>

Here is the part the rule does not warn you about. Take a table of five million floating-point readings, sum the column, and run it three times in a row. Nothing else touches the table. Same connection, same data, same statement.</p>

SELECT</span> sum</span>(reading) </span>FROM</span> measurements;</span></span></code></pre>        sum</span></span>
--------------------</span></span>
 2500519211.7874823</span></span>
</span>
       sum</span></span>
-------------------</span></span>
 2500519211.787477</span></span>
</span>
        sum</span></span>
--------------------</span></span>
 2500519211.7874575</span></span></code></pre>
Three runs, three different totals. No UPDATE</code>, no concurrent writer, no random seed. The rows did not change between runs, and the query is the same character for character. Yet the answer is not.</p>
This is not a Postgres bug, and it is not specific to Postgres. It is what happens when floating-point arithmetic meets parallel aggregation, and it has been generating "my dashboard total changed and I have no idea why" tickets for as long as databases have parallelized. The non-determinism does not wait for you to opt into bad practice; it shows up the moment a parallel plan runs over whatever floats you happen to have.</p>
The schema</a>
</h2>
One column of double precision</code>, five million rows.</p>
CREATE TABLE</span> measurements</span> (</span></span>
    id      </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    reading </span>double precision NOT NULL</span></span>
);</span></span>
</span>
INSERT INTO</span> measurements (reading)</span></span>
SELECT</span> random()</span> *</span> 1000</span></span>
FROM</span> generate_series</span>(</span>1</span>, </span>5000000</span>);</span></span>
</span>
ANALYZE measurements;</span></span></code></pre>
Five million rows is enough that Postgres parallelizes the sum on its own. No settings forced, defaults all the way:</p>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT</span> sum</span>(reading) </span>FROM</span> measurements;</span></span></code></pre>                     QUERY PLAN</span></span>
-----------------------------------------------------</span></span>
 Finalize Aggregate</span></span>
   ->  Gather</span></span>
         Workers Planned: 2</span></span>
         ->  Partial Aggregate</span></span>
               ->  Parallel Seq Scan on measurements</span></span></code></pre>
Two workers each sum part of the table in parallel, and a final step folds the partial sums together. Everything that follows is in that split.</p>
Floating point does not associate</a>
</h2>
On paper, (a + b) + c</code> and a + (b + c)</code> are the same number. Postgres is not doing arithmetic on paper. Floating-point addition is not associative: change the grouping and you can change the answer, because every intermediate result is rounded to fit 64 bits. Move the parentheses and you move which roundings happen.</p>
The textbook demonstration takes three values and one subtraction:</p>
SELECT</span> 1</span>.0e20::float8 </span>+</span> 1</span>.</span>0</span>::float8 </span>-</span> 1</span>.0e20::float8;</span></span></code></pre> ?column?</span></span>
----------</span></span>
        0</span></span></code></pre>
The answer should be 1</code>. Read the expression left to right and watch where it goes wrong:</p>

1.0e20 + 1.0</code> should be 100000000000000000001</code>. But a double only carries about 16 significant digits, and that number needs 21. The trailing 1</code> falls off the end, below the smallest digit the number can represent. So the addition rounds back to exactly 1.0e20</code>.</li>
1.0e20 - 1.0e20</code> is then exactly 0</code>.</li>
</ul>
The 1</code> never made it past the first step. It was swallowed by the addition, before the subtraction even ran. This is absorption</em>: a value too small relative to its neighbour to leave a mark. It is not a database phenomenon. The same expression evaluates to 0</code> in C, Python, JavaScript, and every other language that uses IEEE 754 doubles.</p>
A double precision</code> number keeps 52 bits for the mantissa, the digits that carry precision. This is around 15 to 17 decimal digits, no more. When the running total becomes big and the next number is small, the small one does not fit into those bits, so the rounding throws its tail away. Sum five million numbers and the total is, for most of its life, much bigger than any single number you add to it.</div>
So the sum of a column depends on the order</em> in which the rows are added. Not on which rows, on the order. That is the seam parallel query pries open.</p>
Why parallelism surfaces it</a>
</h2>
The plan spells out exactly how. Parallel Seq Scan</code> does not split the table into fixed slices upfront. It hands every worker a shared pointer into the same block iterator, and each worker grabs the next block, processes it, and comes back for another. Each worker runs its own Partial Aggregate</code>, summing only the rows it happened to grab, and by default the leader process pitches in as a third one (parallel_leader_participation</code>). Gather</code> collects those partial sums in whatever order they arrive, and Finalize Aggregate</code> folds them into a single total.</p>
Two things vary from run to run, and neither is under your control:</p>

Which rows each process sees.</strong> Because the blocks are pulled from a shared iterator on demand, the split is a race. A worker that gets a few microseconds ahead, for any reason the OS scheduler decides, grabs the next block, so the same worker ends up with a different set of rows on every run. The division is never guaranteed to be the same twice.</li>
The order the partial sums come back.</strong> Finalize Aggregate</code> folds the partials into its total in arrival order. With only two values this would not matter, since a + b</code> equals b + a</code>. But there are three of them, the two workers and the leader, and combining three floats is once again an associativity question: which two get added first decides the rounding.</li>
</ol>
Both feed directly into the one thing float addition is sensitive to: grouping. Different row split, different partial sums; different arrival order, different final grouping of the additions. The total lands a few bits away from where it landed last time. Across five million readings that shows up in the sixth decimal place, exactly where the three runs above disagreed.</p>
sum(float8)</code> is marked parallel-safe, which is why the planner is free to do this. It is safe in the sense that it will not crash or corrupt anything. It is not safe in the sense people assume, that the answer is a function of the input.</p>

This is not unique to SUM</code>.</strong> AVG</code>, STDDEV</code>, VARIANCE</code>, and any other floating-point aggregate that accumulates across rows inherit the same order sensitivity. SUM</code> is just the easiest to demonstrate and the one most likely to land in a number a human reads.</p>
</div>
It was never really parallelism</a>
</h2>
It is tempting to file this under "parallel query is flaky" and turn it off. That misses the point. Parallelism does not cause the non-determinism, it only exposes</em> an order-dependence that was there the whole time.</p>
Turn parallelism off completely and the sum becomes stable:</p>
SET</span> max_parallel_workers_per_gather </span>=</span> 0</span>;</span></span>
</span>
SELECT</span> sum</span>(reading) </span>FROM</span> measurements;  </span>-- 2500519211.787516</span></span>
SELECT</span> sum</span>(reading) </span>FROM</span> measurements;  </span>-- 2500519211.787516</span></span>
SELECT</span> sum</span>(reading) </span>FROM</span> measurements;  </span>-- 2500519211.787516</span></span></code></pre>
Stable, but only because, with nothing else scanning the table, a serial sequential scan visits the heap in the same physical order every time, so the additions group the same way every time. The order is fixed, so the rounding is fixed.</p>
"Same order every time" holds because the scan starts at block zero and walks forward. Postgres has one wrinkle that can break even that: synchronize_seqscan</code> (on by default). When a scan of a table larger than shared_buffers / 4</code> is already in progress, a second scan starts at its</em> current block to share cached pages, then wraps around at the end. So a concurrent reader can hand a serial scan a different starting point, and the float sum moves with it, no writes and no parallelism required. It is the same order-dependence, entering through a different door.</div>
 Change the order and the serial sum moves too. We can force different orders without any parallelism at all, using an ordered-set aggregate:
SET</span> max_parallel_workers_per_gather </span>=</span> 0</span>;     </span>-- serial, so ONLY order varies</span></span>
</span>
SELECT</span> sum</span>(reading </span>ORDER BY</span> id)           </span>FROM</span> measurements;  </span>-- 2500519211.787516</span></span>
SELECT</span> sum</span>(reading </span>ORDER BY</span> reading)      </span>FROM</span> measurements;  </span>-- 2500519211.7876067</span></span>
SELECT</span> sum</span>(reading </span>ORDER BY</span> reading </span>DESC</span>) </span>FROM</span> measurements;  </span>-- 2500519211.7875423</span></span></code></pre>
Same rows, same single thread, three summation orders, three answers. Parallelism is one way to shuffle the order, but a plain ORDER BY</code> inside the aggregate does it just as well. The dependence is on order, full stop.</p>
Deterministic is not the same as correct</a>
</h2>
There is a subtler trap hiding in the "turn it off and it is stable" result. A stable answer is not necessarily the right answer. To see the true total, sum the same values as numeric</code>, which adds in exact decimal with no rounding until the very end:</p>
SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements;</span></span></code></pre>              sum</span></span>
-------------------------------</span></span>
 2500519211.787503050078417764</span></span></code></pre>
That is near enough the exact sum of the stored values. Now line everything up:</p>
How the sum was computed</th> Result</th> Off by</th></tr></thead>

Exact (numeric</code>)</td> 2500519211.787503050…</code></td> reference</td></tr>
Serial float8</code>, physical order</td> 2500519211.787516</code></td> ~1.3e-5</td></tr>
Parallel float8</code>, run A</td> 2500519211.7874823</code></td> ~2.1e-5</td></tr>
Parallel float8</code>, run B</td> 2500519211.787477</code></td> ~2.6e-5</td></tr>
Ordered by value ascending</td> 2500519211.7876067</code></td> ~1.0e-4</td></tr>
</tbody></table>
Every floating-point total is wrong, including the "stable" serial one. They differ from the exact value in the fifth and sixth decimal places. Parallelism did not introduce the error - the serial sum was already wrong. What parallelism removed was the serial sum's one comforting property: getting the same wrong answer every time</em>, which people often mistake for correctness.</p>
Summing smallest-to-largest is the classic textbook advice for reducing floating-point error, and it usually does help, but here ascending order happens to land further from the exact value than physical order does. Order changes the error; it does not reliably shrink it. If you need a smaller error there are compensated-summation algorithms (Kahan and friends), but inside a SQL aggregate you do not get to choose them.</div>
Where it actually bites</a>
</h2>
In the sixth decimal of a five-million-row random sum, nobody cares. The problem is the places where someone does.</p>
The obvious one is reconciliation. A finance team sums a revenue column stored as double precision</code>, exports it, and the number disagrees with the same query run an hour later, or with the same logic in a different tool running a different parallel degree. Nothing in the data changed, so the hunt goes to caching, to replicas, to "a dirty read somewhere" long before anyone suspects the aggregate itself.</p>
The same thing wrecks regression tests, just less visibly. Assert that a float total equals an exact literal, or compare two pipelines for byte-identical sums, and the test will flap. It passes on a small fixture that runs serially, then fails in CI on a larger one that crosses the parallel threshold, with no code change in between. The test is asserting a property the type does not have, and it will keep flapping until someone notices that.</p>
And anywhere a float sum feeds an equality check, a join, or a GROUP BY</code> key, you are building on sand. Two totals that "should" be equal can differ in their last bits, and now your join silently drops rows.</p>
Postgres already knows this is dangerous</a>
</h2>
The most convincing evidence that this is a real, known hazard and not a corner case is that Postgres deliberately gives up an optimization because of it.</p>
Window functions can compute a moving sum efficiently using an inverse transition function</em>: as the frame slides forward, subtract the value leaving the window instead of re-summing the whole frame. It works for numeric</code> and integers. For float4</code> and float8</code>, Postgres refuses to do it. From the documentation on user-defined aggregates</a>:</p>

An example of an aggregate for which adding an inverse transition function seems easy at first, yet where this requirement cannot be met is sum</code> over float4</code> or float8</code> inputs.</p>
</blockquote>
The docs then show why, with the same absorption we saw earlier. If a window holds 1e20</code> and 1</code>, subtracting 1e20</code> as it leaves the frame yields 0</code>, not the 1</code> that should remain. So the built-in float sum simply does not use the inverse optimization, and a moving sum(float8)</code> recomputes the frame to stay correct:</p>
SELECT</span> n, </span>sum</span>(x) </span>OVER</span> (</span>ORDER BY</span> n </span>ROWS BETWEEN</span> CURRENT </span>ROW AND</span> 1</span> FOLLOWING</span>)</span></span>
FROM</span> (</span>VALUES</span> (</span>1</span>, </span>1</span>.0e20::float8), (</span>2</span>, </span>1</span>.</span>0</span>::float8)) </span>AS</span> v(n, x);</span></span></code></pre> n | sum</span></span>
---+-------</span></span>
 1 | 1e+20</span></span>
 2 |     1</span></span></code></pre>
The second row is 1</code>, correct, because Postgres declined the shortcut that would have made it 0</code>. The engine is quietly working around the imprecision of float sums on your behalf. The non-determinism in the parallel case is the same hazard showing up somewhere the engine cannot paper over it.</p>
What to do about it</a>
</h2>
The fix is almost always the type, not the query.</p>
Use numeric</code> for anything anyone reconciles, compares, or reads as a figure.</strong> Money, totals on reports, anything that has to match another system or itself across runs. numeric</code> sums in exact decimal and is order-independent: same rows, same answer, parallel or not, in any order.</p>
SET</span> max_parallel_workers_per_gather </span>=</span> 4</span>;</span></span>
SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements;  </span>-- 2500519211.787503050078417764</span></span>
SELECT</span> sum</span>(reading::</span>numeric</span>) </span>FROM</span> measurements;  </span>-- 2500519211.787503050078417764</span></span></code></pre>
The cast above stabilizes a single query, but the durable fix is storing the column as numeric</code> in the first place, so the exact values are what live on disk.</p>
numeric</code> is not free. It is arbitrary-precision arithmetic done in software, so it is slower to compute and takes more space on disk than a hardware float the CPU adds in a single instruction. For anything that has to reconcile, that is the price of correctness, and it is almost always worth paying. For a column nobody ever sums into a figure a person reads, it usually is not.</p>
If the data is genuinely floating point</strong> (sensor readings, scientific measurements, anything where the input is already an approximation), then accept that aggregates over it are approximate and do not assert exact equality on them. Round for display, but compare against a tolerance rather than by rounding, since two totals a hair apart round to different neighbours as readily as to the same one, and never use a raw float sum as a join or grouping key.</p>
Pinning max_parallel_workers_per_gather = 0</code> makes a float sum repeatable, but does not make it correct,</strong> and it costs you parallelism on every other query in the session. The result is still wrong, just consistently wrong. Reach for the type instead.</p>
None of this is Postgres-specific. Any database that parallelizes aggregation, and they all do, has the same behavior over the same IEEE 754 floats. The trigger that makes it visible is simply crossing whatever threshold turns parallelism on, which is why it so often shows up when a table grows past a certain size and "nothing else changed."</p>
PS: Thanks to Peter J. Holzer for spotting that the cast to numeric</code> rounds on the way in, and that rounding is the wrong way to compare floats.</p>


The NULL in your NOT IN
2026-06-14T23:30:00+00:00
A NOT IN</code> query can return the wrong answer without telling you. It is valid SQL, it runs without an error, and it hands back a perfectly well-formed result set that happens to be empty when it should not be. No warning, no hint, nothing in the logs: just zero rows where you expected hundreds, and a database that considers it correct.</p>
Almost always the cause is a single NULL</code> sitting somewhere you forgot to look, combined with two keywords you have typed a thousand times: NOT IN</code>. None of it is a Postgres bug. This is exactly what the SQL standard mandates, implemented faithfully. That is precisely what makes it so easy to walk into, and why the planner could not safely optimize around it for the better part of Postgres's history. It comes down to one if</code> statement in the parser.</p>
Sample schema</a>
</h2>
Nothing elaborate. A table of products, one of which has no category assigned yet, and a table of archived categories that happens to contain a NULL</code>:</p>
CREATE TABLE</span> products</span> (id </span>int</span>, category_id </span>int</span>);</span></span>
INSERT INTO</span> products </span>VALUES</span> (</span>1</span>, </span>10</span>), (</span>2</span>, </span>20</span>), (</span>3</span>, </span>NULL</span>), (</span>4</span>, </span>10</span>);</span></span>
</span>
CREATE TABLE</span> archived</span> (category_id </span>int</span>);</span></span>
INSERT INTO</span> archived </span>VALUES</span> (</span>20</span>), (</span>NULL</span>);</span></span></code></pre>
The NULL</code> in archived</code> is not contrived. The moment a column is nullable (and most are, by default), a NULL</code> can find its way into any subquery you point a NOT IN</code> at. That is the whole point: this is not an exotic data condition, it is the ordinary one.</p>
The query that returns nothing</a>
</h2>
Here is the request you have written a hundred times: give me the products whose category is not</em> archived.</p>
SELECT</span> id, category_id </span>FROM</span> products</span></span>
WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived);</span></span></code></pre>
You expect products 1 and 4 (category 10, which is not in the archived set). What comes back is:</p>
 id | category_id</span></span>
----+-------------</span></span>
(0 rows)</span></span></code></pre>
Every row gone. Not a subset, not an off-by-one: all of them. Drop the NULL</code> from archived</code> and the same query behaves:</p>
SELECT</span> id, category_id </span>FROM</span> products</span></span>
WHERE</span> category_id </span>NOT IN</span> (</span>SELECT</span> category_id </span>FROM</span> archived</span></span>
                          WHERE</span> category_id </span>IS NOT NULL</span>);</span></span></code></pre> id | category_id</span></span>
----+-------------</span></span>
  1 |          10</span></span>
  4 |          10</span></span>
(2 rows)</span></span></code></pre>
To understand why a single NULL</code> empties the entire result, we have to stop thinking of NOT IN</code> as a single thing and watch the parser take it apart.</p>
IN is an OR, NOT IN is an AND</a>
</h2>
IN</code> is not a primitive operator. It is shorthand that the parser rewrites into a chain of equality comparisons joined by OR</code>:</p>
x </span>IN</span> (a, b, c)</span></span>
-- becomes</span></span>
x </span>=</span> a </span>OR</span> x </span>=</span> b </span>OR</span> x </span>=</span> c</span></span></code></pre>
NOT IN</code> is the logical negation of that, and by De Morgan's law negating an OR</code> of equalities gives you an AND</code> of inequalities:</p>
x </span>NOT IN</span> (a, b, c)</span></span>
-- becomes</span></span>
x </span><></span> a </span>AND</span> x </span><></span> b </span>AND</span> x </span><></span> c</span></span></code></pre>
This is not an analogy. It is literally the expression Postgres builds, and you can read it straight off an EXPLAIN</code>. The literal-list forms collapse into array operators whose names give the whole game away:</p>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> products </span>WHERE</span> category_id </span>IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span>
--  Filter: (category_id = ANY ('{1,2,3}'::integer[]))</span></span>
</span>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> products </span>WHERE</span> category_id </span>NOT IN</span> (</span>1</span>, </span>2</span>, </span>3</span>);</span></span>
--  Filter: (category_id <> ALL ('{1,2,3}'::integer[]))</span></span></code></pre>
IN</code> is = ANY</code>: equal to any</em> element, an OR</code>. NOT IN</code> is <> ALL</code>: different from all</em> elements, an AND</code>.</p>
The actual node types matter here, because they are what you end up staring at when you dump a parse tree or read a normalized pg_stat_statements</code></a> entry. A literal list compiles to a single ScalarArrayOpExpr</code>: the scalar on the left, the array on the right, and a useOr</code> flag that is the entire difference between = ANY</code> and <> ALL</code>. The subquery forms are a different node altogether, a SubLink</code>. Recognising those two names on sight tells you immediately which path the planner is on.</p>
If "IN</code> and = ANY</code> are the same operator" is news: they compile to the same parse node and the same plan, with the spellings diverging only in plan-cache churn and selectivity estimates. The NOT IN</code> case in front of you here is the one corner where the choice is not cosmetic but a matter of correctness.</div>
Three-valued logic does the rest</a>
</h2>
SQL does not have two truth values, it has three: true</strong>, false</strong>, and unknown</strong>. Any comparison against NULL</code> yields unknown</code>, because NULL</code> means "no value here" and you cannot ask whether an absent value is different from 20:</p>
-- not false. unknown (displayed as a blank)</span></span>
SELECT</span> 10</span> <> NULL</span>;</span></span></code></pre>
Now walk the NOT IN</code> expansion for product 1 (category 10) against the archived set of 20 and NULL</code>:</p>

 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a>
 20 AND 10 <> NULL, which evaluate to true and unknown; the NULL poisons its branch, and AND carries the unknown to the root, so the row is dropped"></a>
</div>
true AND unknown</code> is unknown</code>, not true</code>. A WHERE</code> clause keeps a row only when its predicate evaluates to true</strong>. Both false</code> and unknown</code> cause the row to be discarded. So product 1 is dropped. Run the same arithmetic for product 4 and you land on unknown</code> again.</p>

The mechanism in one sentence:</strong> the instant a single NULL</code> enters the right-hand side, the trailing AND unknown</code> term can never be true</code>, so the whole NOT IN</code> can never be true</code>, so every</strong> row is discarded, regardless of how many million rows you have or what they contain.</p>
</div>
NULLs on the left side too</a>
</h2>
Keeping NULL</code>s out of the subquery is not enough. The same unknown</code> arises from NULL</code>s on the left</em>: product 3 (whose category_id</code> is NULL</code>) evaluates to unknown AND unknown</code>, so it is dropped even against a spotless right-hand set. IN</code> and NOT IN</code> are not complements: a row can fail both tests simultaneously. There is a NULL</code>-shaped gap between them that belongs to neither.</p>
The seam, in the source</a>
</h2>
All of this reduces to one branch in one function. Open src/backend/parser/parse_expr.c</code> and find transformAExprIn</code>, the routine that turns both IN</code> and NOT IN</code> list expressions into something the planner can chew on. The very first thing it decides is whether it is building an OR</code> or an AND</code>:</p>
/*</span></span>
 * If the operator is <>, combine with AND not OR.</span></span>
 */</span></span>
if</span> (</span>strcmp</span>(</span>strVal</span>(</span>linitial</span>(a</span>-></span>name</span>)),</span> "<>"</span>)</span> ==</span> 0</span>)</span></span>
    useOr </span>=</span> false</span>;</span></span>
else</span></span>
    useOr </span>=</span> true</span>;</span></span></code></pre>
That is the entire fork. IN</code> arrives carrying the operator =</code> and gets useOr = true</code>; NOT IN</code> arrives carrying <></code> and gets useOr = false</code>. The flag rides all the way down to where the boolean tree is finally assembled, several hundred lines later:</p>
result </span>=</span> (Node </span>*</span>)</span> makeBoolExpr</span>(useOr </span>?</span> OR_EXPR </span>:</span> AND_EXPR,</span></span>
                               list_make2</span>(result, cmp),</span></span>
                               a</span>-></span>location</span>);</span></span></code></pre>
OR_EXPR</code> for IN</code>, AND_EXPR</code> for NOT IN</code>. There is no special-casing of NULL</code> anywhere in this function, and there does not need to be: the three-valued behavior is an emergent property of having chosen AND</code>. The parser does the obvious, correct thing, and the NULL</code> semantics fall straight out of standard boolean logic. The "bug", if you insist on the word, belongs to the SQL standard, which Postgres implements faithfully.</p>
A grammar asymmetry: list vs. subquery</a>
</h3>
A list and a subquery are built differently. The list form is the <> ALL</code> chain of inequalities you just saw. The subquery form is not a <> ALL</code> at all: it becomes NOT (foo = ANY (subquery))</code>. Different shapes, same truth table, a NULL</code> in the comparison makes the result unknown</code>, and unknown</code> loses. That NOT (... = ANY ...)</code> shape is the one the planner sees.</p>
Why the planner won't save you</a>
</h2>
The correctness problem has a plan-shape twin. Watch what the planner does with three sibling queries on a larger schema of 200,000 orders</code> and 1,000 vip</code> rows.</p>
First, the positive case, IN</code> against a subquery:</p>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span>
WHERE</span> customer_id </span>IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre> Hash Semi Join</span></span>
   Hash Cond: (orders.customer_id = vip.customer_id)</span></span>
   ->  Seq Scan on orders</span></span>
   ->  Hash</span></span>
         ->  Seq Scan on vip</span></span></code></pre>
A clean semi-join</strong>. The planner promotes the subquery to a first-class relation, picks a hash join, and is free to reorder it against the rest of the query. Now the natural mirror, orders whose customer is not</em> a VIP, written first with NOT EXISTS</code>:</p>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders o</span></span>
WHERE NOT EXISTS</span> (</span>SELECT</span> 1</span> FROM</span> vip v </span>WHERE</span> v</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span>);</span></span></code></pre>                  QUERY PLAN</span></span>
----------------------------------------------</span></span>
 Hash Anti Join</span></span>
   Hash Cond: (o.customer_id = v.customer_id)</span></span>
   ->  Seq Scan on orders o</span></span>
   ->  Hash</span></span>
         ->  Seq Scan on vip v</span></span></code></pre>
A hash anti-join</strong>, the efficient and symmetric counterpart to the semi-join. Now the same intent expressed with NOT IN</code>:</p>
EXPLAIN (COSTS </span>OFF</span>) </span>SELECT * FROM</span> orders</span></span>
WHERE</span> customer_id </span>NOT IN</span> (</span>SELECT</span> customer_id </span>FROM</span> vip);</span></span></code></pre>                          QUERY PLAN</span></span>
---------------------------------------------------------------</span></span>
 Seq Scan on orders</span></span>
   Filter: (NOT (ANY (customer_id = (hashed SubPlan 1).col1)))</span></span>
   SubPlan 1</span></span>
     ->  Seq Scan on vip</span></span>
(4 rows)</span></span></code></pre>
No join at all. The subquery collapses into an opaque SubPlan</code> filter bolted onto a sequential scan, the NOT (... = ANY ...)</code> shape from the grammar is right there in the filter.</p>
The wording of that filter line is version-specific. PostgreSQL 16 prints the terser Filter: (NOT (hashed SubPlan 1))</code>; a later version changed EXPLAIN</code> to expose the inner comparison, which is the NOT (ANY (... = (hashed SubPlan 1).col1))</code> form shown here on 18.4. The plan underneath is the same opaque subplan in every released version.</div>
The difference between those last two plans is structural, not a tuning detail. A Hash Anti Join</code> builds one hash table from the inner relation and streams the outer relation through it exactly once, and when the inner side outgrows work_mem</code> it partitions into batches and spills to disk without changing the result. A SubPlan</code> gives the planner none of that. Even the hashed</code> variant you see here is evaluated in place as a filter on the scan: it cannot be reordered during the global join search, it cannot have the outer query's join clauses pushed into it to drive an index scan, and it has no multi-batch spill logic to fall back on if the hashed set turns out larger than the estimate. The subquery stops being a relation the optimizer can plan and becomes an opaque function the optimizer has to call, once per outer row. This is the same optimization-fence effect as an uninlined CTE</a>: the planner simply cannot see inside. On small inputs nobody notices; on large ones it is the difference between milliseconds and minutes.</p>
The reason the planner is stuck with that shape is precisely the NULL</code> semantics from earlier. An anti-join keeps a row when it finds no match</em>. But NOT IN</code> must discard</em> a row when the comparison goes unknown</code>, and "unknown" is not the same as "no match". Those two behaviors diverge exactly when a NULL</code> is in play, so the planner historically could not prove the rewrite safe and never attempted it.</p>

Declaring the columns NOT NULL</code> does not rescue you on released Postgres.</strong> I constrained both the outer and inner columns NOT NULL</code> on 18.4 and re-ran the query, and the plan was still the opaque SubPlan</code> filter. The released planner does not even look. So NOT IN (subquery)</code> has been a correctness trap for as long as Postgres has existed, and a planner pessimization ever since anti-joins arrived in 8.4 (2009) and every other negated subquery learned to use them — neither of which the released optimizer rescues you from.</p>
</div>
The fix is landing in PostgreSQL 19</a>
</h2>
That last paragraph is finally going out of date. This is a performance fix, not a semantic one: it rescues the case that was already correct</em> but badly planned</em> (NOT NULL</code> columns stuck behind the opaque SubPlan</code> from the previous section) and leaves nullable columns exactly where the SQL standard puts them. Tracing this through a checkout of the development branch, I found a function that exists in no released Postgres: sublink_testexpr_is_not_nullable</code>, in src/backend/optimizer/plan/subselect.c</code>. It guards a brand-new branch inside convert_ANY_sublink_to_join</code>:</p>
/*</span></span>
 * Per SQL spec, NOT IN is not ordinarily equivalent to an anti-join, so</span></span>
 * that by default we have to fail when under_not.  However, if we can</span></span>
 * prove that neither the outer query's expressions nor the sub-select's</span></span>
 * output columns can be NULL, and further that the operator itself cannot</span></span>
 * return NULL for non-null inputs, then the logic is identical and it's</span></span>
 * safe to convert NOT IN to an anti-join.</span></span>
 */</span></span>
if</span> (under_not </span>&&</span></span>
    (</span>!</span>sublink_testexpr_is_not_nullable</span>(root, sublink)</span> ||</span></span>
     !</span>query_outputs_are_not_nullable</span>(subselect)))</span></span>
    return</span> NULL</span>;</span></span></code></pre>
git blame</code> dates it to commit 383eb21ebff</code>, "Convert NOT IN sublinks to anti-joins when safe"</em>, merged in March 2026 by Richard Guo. It lives on master</code>, bound for PostgreSQL 19</strong>. It appears in no REL_18</code> tag, which is exactly why my 18.4 box still produced the SubPlan</code> even with NOT NULL</code> columns. The commit message states the bargain plainly:</p>

if we can prove that neither side of the comparison can yield NULL values, and further that the operator itself cannot return NULL for non-null inputs, the behavior of NOT IN and anti-join becomes identical.</p>
</blockquote>
The proof has to establish three things:</p>

Both operands provably non-NULL</code>.</strong> Established from schema NOT NULL</code> constraints (via a NOT-NULL-attnums hash table), from outer-join nullability tracking (so a Var</code> from the nullable side of an outer join does not qualify), and from qual clauses that force a Var</code> non-null.</li>
The subquery's output columns provably non-NULL</code></strong>, the query_outputs_are_not_nullable</code> half of the guard.</li>
A NULL</code>-safe operator.</strong> The operator must belong to a B-tree or Hash operator family. That is a proxy for "behaves like a normal boolean comparison and won't return NULL</code> on non-null inputs", because an operator that did</em> return NULL</code> there would break the very index it claims to support.</li>
</ul>
When all three hold, the NULL</code>-handling mismatch evaporates and NOT IN</code> is finally allowed to become a JOIN_ANTI</code>:</p>
result</span>-></span>jointype </span>=</span> under_not </span>?</span> JOIN_ANTI </span>:</span> JOIN_SEMI;</span></span></code></pre>
It lifts a planner limitation that has stood since anti-joins themselves arrived in 8.4, but only when the planner can prove</em> no NULL</code> can reach the comparison. If your column is nullable, you are exactly where you have always been, in PostgreSQL 19 as in 9.x. The semantics never changed; the optimizer merely learned to recognize the cases where NULL</code> is provably absent.</p>
Decision matrix</a>
</h2>
You wrote</th> Internal shape</th> NULL on right →</th> NULL on left →</th> Planner (≤ PG 18)</th></tr></thead>

IN (1,2,3)</code></td> = ANY</code>, an OR</code></td> absorbed, no harm</td> row dropped (no match)</td> ScalarArrayOpExpr</code></td></tr>
NOT IN (1,2,3)</code></td> <> ALL</code>, an AND</code></td> all rows dropped</strong></td> row dropped</strong></td> ScalarArrayOpExpr</code></td></tr>
IN (subquery)</code></td> = ANY</code> sublink</td> absorbed</td> row dropped</td> Semi Join</strong></td></tr>
NOT IN (subquery)</code></td> NOT (= ANY)</code> sublink</td> all rows dropped</strong></td> row dropped</strong></td> opaque SubPlan</code> filter ¹</td></tr>
NOT EXISTS (...)</code></td> anti-join sublink</td> row kept</td> row kept</td> Anti Join</strong></td></tr>
</tbody></table>
¹ PostgreSQL 19 promotes this to an Anti Join when both the outer expression and the subquery output column are provably NOT NULL</code>.</p>
What to do instead</a>
</h2>
Don't wait for PostgreSQL 19, and don't lean on it once it ships: it only fires on provably NOT NULL</code> columns. Every portable fix comes down to one rule: keep NULL</code> away from NOT IN</code>. In practice that means changing the query, and there are three moves, in order of preference.</p>
Default to NOT EXISTS</code>.</strong> Make this the habit and you never hit the trap again. Same anti-join semantics and the same plan as the working NOT IN</code> would have wanted, on every version, nullable columns or not:</p>
SELECT</span> id, category_id </span>FROM</span> products p</span></span>
WHERE NOT EXISTS</span> (</span></span>
    SELECT</span> 1</span> FROM</span> archived a </span>WHERE</span> a</span>.</span>category_id</span> =</span> p</span>.</span>category_id</span></span>
);</span></span></code></pre>
It keeps product 3, the NULL</code> category, because NOT EXISTS</code> asks whether a matching archived row exists, not whether category_id = NULL</code> is true. That is usually the answer you wanted. If you do want the NULL</code>-category rows gone, add AND p.category_id IS NOT NULL</code> and you have decided it on purpose.</p>
Filter the NULL</code>s out of the subquery</strong> when you have to keep the NOT IN</code> (a generated query you can only edit inside the parentheses, say):</p>
SELECT</span> id </span>FROM</span> products</span></span>
WHERE</span> category_id </span>NOT IN</span> (</span></span>
    SELECT</span> category_id </span>FROM</span> archived </span>WHERE</span> category_id </span>IS NOT NULL</span></span>
);</span></span></code></pre>
This covers the right-hand NULL</code> and nothing else: product 3 still disappears, because the left-hand NULL</code> is untouched. Use it only when you cannot reach for NOT EXISTS</code>.</p>
Use EXCEPT</code> for whole-set comparisons.</strong> It matches rows with IS NOT DISTINCT FROM</code>, so two NULL</code>s count as equal and the three-valued trap never fires. But that same rule means a NULL</code> in archived</code> removes the NULL</code> rows from products</code>:</p>
SELECT</span> category_id </span>FROM</span> products</span></span>
EXCEPT</span></span>
SELECT</span> category_id </span>FROM</span> archived;</span></span>
-- returns: {10}</span></span>
-- product 3 (NULL category) is dropped because archived also has a NULL</span></span></code></pre>
So NOT EXISTS</code> keeps product 3 and EXCEPT</code> drops it. Pick the one whose answer matches what you're after.</p>
EDIT (2026-06-15): I originally said this stood for about 25 years, which mixed up two things. The correctness trap is as old as NOT IN</code>; the planner pessimization only goes back to 8.4 (2009), when anti-joins arrived. Reworded to keep the two separate.</em></p>


Welcome to ORDER BY jungle
2026-05-15T11:43:00+02:00
SQL is fun and not at all boring. The latest article by Markus Winand on Order by Has Come a Long Way</a> sent me on quite a journey.</p>
First, set up a table called nums</code> with one integer column and four rows:</p>
CREATE TABLE</span> nums</span> (a </span>int</span>);</span></span>
INSERT INTO</span> nums </span>VALUES</span> (</span>0</span>), (</span>1</span>), (</span>2</span>), (</span>3</span>);</span></span></code></pre>
Try to guess what these two queries return.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre>
Most of us would guess the same rows in a different order. The actual answer is that they produce exactly the same rows in exactly the same order</strong>. By the same logic you might expect</p>
SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>c;</span></span></code></pre>
to do exactly the same. Except it does not. It errors with column "c" does not exist</code> despite the alias being right there in the statement. Welcome to ORDER BY jungle.</p>
Names and expressions are not the same</a>
</h2>
If you ask most developers how ORDER BY</code> works, they will say "you put a column name there and it sorts the rows". In 99% of queries that is exactly what happens. People sort by created_at</code> or id</code> and move on.</p>
Strictly speaking, three, if you count ORDER BY 1</code>. Positional references are their own can of worms and out of scope for this post.</div>But `ORDER BY` accepts two different kinds of things:
SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY</span> created_at;</span></span>
SELECT</span> created_at, user_id </span>FROM</span> events </span>ORDER BY date</span>(created_at);</span></span></code></pre>
Both feel natural. And the thing nobody tells you is that they go down completely different code paths in the parser. Different scope rules, different lookups, different error messages. The first looks at your SELECT</code> list. The second looks at your FROM</code> clause. They never look at the same place.</p>
Same answer, two different sorts</a>
</h2>
Look at the first query again.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span></code></pre>
You wrote ORDER BY a</code>. A bare identifier, no decoration. Postgres goes down the name</em> path. It scans the SELECT</code> list for something called a</code>, finds the aliased column -a AS a</code>, and sorts by its output values. The negated values are -3, -2, -1, 0</code>, ascending is -3, -2, -1, 0</code>. That is what comes out.</p>
Now the twin.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre>
You wrote ORDER BY -a</code>. This is no longer an identifier. It's an expression: unary minus around a column reference. The parser does not even try the same logic.</p>
Instead it switches to the expression path</strong>, where the only a</code> it knows is the column in nums</code>, and sorts the input values negated. And by arithmetic luck, the two queries land on the same row order. Same output, completely different logic. If you don't believe it is just luck, drop the negation from the SELECT</code> list and keep it in ORDER BY</code>:</p>
SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY -</span>a;</span></span></code></pre> c</span></span>
---</span></span>
 3</span></span>
 2</span></span>
 1</span></span>
 0</span></span>
(4 rows)</span></span></code></pre>
ORDER BY -a</code> is an expression, so it sorts by -input_a</code> ascending, which is input_a</code> descending. The alias c</code> was never consulted. The result has nothing to do with whatever c</code> happens to be.</p>
And ORDER BY -c</code> is now obvious. -c</code> is an expression, so the parser looks for column c</code> in FROM</code>, doesn't find it, and errors. The alias exists, but in a scope this code path cannot see.</p>
Above the identifier, or around it</a>
</h2>
Once the rule is clear (bare identifier hits the SELECT</code> list, anything else hits the table) the rest of the surprises fall out.</p>
SELECT</span> 'hello'</span> AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x::</span>text</span>;</span></span>
-- ERROR: column "x" does not exist</span></span></code></pre>
It is probably not surprising that casts</strong> count as expressions and push the lookup to the table.</p>
The surprise might come with</p>
SELECT</span> a </span>AS</span> c </span>FROM</span> nums </span>ORDER BY</span> c </span>DESC NULLS FIRST</span>;</span></span></code></pre>
Which will work as expected. Both DESC</code> and NULLS FIRST</code> are part of the sort clause itself, not of the sort expression. They sit above the identifier in the parse tree, so they never touch it. The parser still sees a bare c</code>, takes the fast path, finds the alias, sorts by it, and then applies "descending, nulls first" on top of the resolved key.</p>
The same cannot be said about collation</strong>.</p>
SELECT</span> 'A'</span>::</span>text AS</span> x </span>FROM</span> nums </span>ORDER BY</span> x </span>COLLATE</span> "C"</span>;</span></span>
-- ERROR: column "x" does not exist</span></span></code></pre>
This is a really bad one. COLLATE</code> might look the same as a sort modifier, but it is not. It wraps the expression in the parse tree.</p>
Parentheses are a special case.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> (a);</span></span>
-- works, sorts by alias</span></span></code></pre>
Postgres collapses redundant parens before the bare-identifier check, so (a)</code> is still bare a</code>. The seam is asymmetric in the way that maximises confusion: COLLATE</code> is "still a name to a human, an expression to the parser", and (a)</code> is "an expression to a human, still a name to the parser". You get both flavours of wrong intuition mixed here.</p>
Unary plus.</strong> +a</code> and a</code> evaluate to the same value, but they do not parse to the same node.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY +</span>a;</span></span></code></pre>
A plus sign you would not even think about changes which rows come out in which order.</p>
The parser stores a column reference as a list of name parts: one part when it is unqualified, two or more once you add a table or schema. The fast path only fires on lists of length one.</div>
Finally, schema- and table-qualified references</strong>. ORDER BY nums.a</code> looks like an identifier, but it is not.</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a;</span></span>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> nums</span>.</span>a</span>; </span></span></code></pre> a</span></span>
----</span></span>
 -3</span></span>
 -2</span></span>
 -1</span></span>
  0</span></span>
(4 rows)</span></span>
</span>
 a</span></span>
----</span></span>
  0</span></span>
 -1</span></span>
 -2</span></span>
 -3</span></span>
(4 rows)</span></span></code></pre>Aliases that aren't the names you think</a>
</h2>
Here is one that cost me an afternoon once. Easy to come across once an ORM or a generated view declared the alias for you. SQLAlchemy, Hibernate, jOOQ, and most code generators quote anything that isn't pure lowercase. Two queries, identical except that the alias is quoted in one. Two different result sets.</p>
SELECT -</span>a </span>AS</span>  A  </span>FROM</span> nums </span>ORDER BY</span> a;    </span>-- sorts by alias (-3,-2,-1,0)</span></span>
SELECT -</span>a </span>AS</span> "A"</span> FROM</span> nums </span>ORDER BY</span> a;    </span>-- sorts by input (0,-1,-2,-3)</span></span></code></pre>
The bare-identifier check compares names with strcmp</code>. Unquoted A</code> folds to lowercase a</code> and matches. Quoted "A"</code> preserves case, stays A</code>, and does not match the lowercase a</code> in the ORDER BY</code>. The lookup fails, the parser falls through to the expression path, the expression path finds the column a</code> in nums</code>, and the query runs successfully while doing something different from what you meant.</p>
GROUP BY checks the opposite scope first</a>
</h2>
Both GROUP BY</code> and ORDER BY</code> accept a bare identifier, and both can resolve it either way: to a table column or to a SELECT</code>-list alias. The difference is the order they check:</p>

ORDER BY a</code> looks at the SELECT</code> list first, then the table.</li>
GROUP BY a</code> looks at the table first, then the SELECT</code> list.</li>
</ul>
For most queries this never matters. The two clauses end up picking the same thing because nothing is shadowed. The surprise happens when an alias has the same name as a base column but a different value:</p>
SELECT</span> a</span>/</span>2</span> AS</span> a, </span>count</span>(</span>*</span>)</span></span>
FROM</span> nums</span></span>
GROUP BY</span> a</span></span>
ORDER BY</span> a;</span></span></code></pre>
Now the two clauses disagree about what a</code> means. GROUP BY a</code> picks the input column (four distinct values, four groups, one row each). ORDER BY a</code> picks the alias, which is a/2</code>. The result has four rows because the grouping was on a finer-grained key than the projection:</p>
 a | count</span></span>
---+-------</span></span>
 0 |     1</span></span>
 0 |     1</span></span>
 1 |     1</span></span>
 1 |     1</span></span></code></pre>
Two rows where a/2 = 0</code> (from input 0</code> and 1</code>), two where a/2 = 1</code> (from input 2</code> and 3</code>). The duplicates are real. The same identifier means two different columns in two adjacent clauses of one query.</p>
Window ORDER BY does not even pretend</a>
</h2>
This one trips people up because it does not look like a different clause.</p>
SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY</span> neg) </span>FROM</span> nums;</span></span>
-- ERROR: column "neg" does not exist</span></span></code></pre>
OVER (ORDER BY ...)</code> is a different parse path entirely. It does not check the targetlist at all, only the FROM scope. The bare-name fast path simply does not exist here.</p>
SELECT</span> a, </span>-</span>a </span>AS</span> neg, </span>row_number</span>() </span>OVER</span> (</span>ORDER BY -</span>a) </span>FROM</span> nums;</span></span>
-- this works</span></span></code></pre>
Two ORDER BY</code> clauses in the same query, two different scoping rules.</p>
UNION ORDER BY is name-only</a>
</h2>
When ORDER BY</code> follows a UNION</code>, neither path is fully open.</p>
-- ok</span></span>
(</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a;</span></span>
-- ERROR</span></span>
(</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY -</span>a;</span></span>
-- ERROR</span></span>
(</span>SELECT</span> a </span>FROM</span> nums) </span>UNION ALL</span> (</span>SELECT</span> 9</span>) </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre>
The error message is unusually helpful:</p>

Only result column names can be used, not expressions or functions. HINT: Add the expression/function to every SELECT, or move the UNION into a FROM clause.</p>
</blockquote>
Set operations do not have a single FROM scope to fall back to, so the expression path is closed entirely. Bare names or nothing.</p>
The seam, in the source</a>
</h2>
Full disclosure: I got this section wrong three times before Claude Code helped me trace the actual parse tree. Lack of sleep from a whole night of geeking out over ORDER BY</code> is the other plausible explanation.</div>Open `src/backend/parser/parse_clause.c` and find `findTargetlistEntrySQL92`. It is forty lines of comment, two `if` blocks, and a final `return`. SQL92's two resolution rules are tried first; SQL99 is the fallback.
Block one: the bare-name path.</strong> The gate is a ColumnRef</code> node with exactly one name part, and that part must be a string identifier (not *</code>, which is also a ColumnRef</code> but with an A_Star</code> field). If the node passes, the function walks the target list looking for a non-resjunk</code> entry whose resname</code> equals the identifier. The loop keeps going past the first match to detect ambiguity: identical expressions are fine (this is why SELECT a, a FROM nums ORDER BY a</code> works), different expressions error out. On a unique match, return.</p>
If the loop finds nothing, the block does not</strong> return. Control falls through. This is the case behind the quoted-alias surprise earlier in the post: AS "A"</code> stores resname = "A"</code>, ORDER BY a</code> looks up resname = "a"</code>, the strcmp</code> fails, and the function moves on as if no SQL92 fast path applied.</p>
GROUP BY</code> is the small exception inside this block. The name is first tested against the FROM</code> scope, and a hit there causes the targetlist loop to be skipped. That is how GROUP BY</code> ends up preferring the input column.</p>
Block two: the positional path.</strong> The gate is IsA(node, A_Const)</code>. A non-integer constant errors immediately ("non-integer constant in ..."), which catches ORDER BY NULL</code>, ORDER BY 'a'</code>, ORDER BY TRUE</code>. An integer is used as a 1-based position into the non-resjunk</code> target list; anything outside the range errors as "position %d is not in select list". Block two never falls through.</p>
Both 1</code> and -1</code> arrive here as integer A_Const</code>s. doNegate</code> in the grammar folds '-' Iconst</code> into a single integer constant before the function ever runs, so ORDER BY 1</code> and ORDER BY -1</code> go through the same code, with only the integer value (and the result of the position lookup) differing.</p>
The fallthrough.</strong> Anything not caught above reaches the last line:</p>
/*</span></span>
 * Otherwise, we have an expression, so process it per SQL99 rules.</span></span>
 */</span></span>
return</span> findTargetlistEntrySQL99</span>(pstate, node, tlist, exprKind);</span></span></code></pre>
That is the seam. SQL92 succeeds in two narrow shapes: a bare identifier with a matching alias, or an in-range positive integer. Everything else, including a bare identifier whose alias lookup found nothing, becomes a SQL99 expression resolved against FROM</code>.</p>
A useful workaround</a>
</h2>
If you want the alias inside an expression in ORDER BY</code>, the portable trick is to wrap the query in a subselect:</p>
SELECT *</span></span>
FROM</span> (</span>SELECT -</span>a </span>AS</span> x </span>FROM</span> nums) s</span></span>
ORDER BY</span> x </span>+</span> 0</span>;</span></span></code></pre>
Now x</code> is a real column in the FROM scope of the outer query. The expression path finds it. The seam has been moved out of the way.</p>
This is, conceptually, what you would want the engine to do for you when you write ORDER BY x + 0</code> directly. The SQL-99 standard does not actually require that, though, and Postgres (along with SQL Server) documents explicitly that an alias inside an ORDER BY</code> expression is not supported. So you do it by hand.</p>
The boring takeaway</a>
</h2>
Most of the time none of this matters. You sort by a column you just selected, the alias and the input column have the same name and the same value, and either parser path gives the same answer. The seam is invisible.</p>
The minute the alias and the input column disagree in expression, value, case, or anything wrapped around the identifier, the parser picks one or the other silently, by a rule older than most working programmers.</p>
There are two parsers. The bare-name path is SQL-92, the expression path is SQL-99, and they were stitched together in the late 1990s. They still disagree about which scope your identifiers live in, and knowing which one you triggered tells you which scope.</p>
If after reading this post you still have to stop and think for a minute before predicting what</p>
SELECT -</span>a </span>AS</span> a </span>FROM</span> nums </span>ORDER BY</span> a </span>COLLATE</span> "C"</span>;</span></span></code></pre>
does, that is the right reaction. It means you have the mental model.</p>

The opening puzzle queries are from Jamie Brandon's comment on the Lobsters thread</a> discussing Markus Winand's history of ORDER BY</code></a> on modern-sql.com. Everything that follows here is the explanation that comment did not give. Both pieces are worth reading on their own.</em></p>


Strong views on PostgreSQL VIEWs
2026-05-10T00:00:00+00:00
VIEWs should be the cleanest abstraction SQL, and therefore Postgres, has on offer. I love the concept. The promise of decoupling logical intent from physical storage is perfect on paper. In practice, few things in the database world trigger such a heated debate or carry as much historical baggage. VIEWs mix big promises with false hopes, and the promises rarely survive contact with production.</p>
The appeal is straightforward. Abstract</strong> "active customer" once and reuse it everywhere. Every query, report and dashboard uses the same definition. The "active customer" then becomes the foundation of a "customer orders" view, which in turn powers an operational "customer summary" view.</p>
-- layer 1: who counts as an active customer?</span></span>
CREATE VIEW</span> active_customers</span> AS</span></span>
SELECT</span> c.</span>*</span></span>
FROM</span> customers c</span></span>
WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span>
  AND</span> c</span>.</span>status</span> =</span> 'active'</span></span>
  AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span>
</span>
-- layer 2: active customers with their recent orders</span></span>
CREATE VIEW</span> customer_orders</span> AS</span></span>
SELECT</span></span>
    ac.</span>*</span>,</span></span>
    o</span>.</span>id</span>         AS</span> order_id,</span></span>
    o</span>.</span>total_cents</span>,</span></span>
    o</span>.</span>created_at</span> AS</span> ordered_at,</span></span>
    o</span>.</span>status</span>     AS</span> order_status</span></span>
FROM</span> active_customers ac</span></span>
LEFT JOIN</span> orders o </span>ON</span> o</span>.</span>customer_id</span> =</span> ac</span>.</span>id</span></span>
WHERE</span> o</span>.</span>created_at</span> > now</span>()</span> -</span> interval </span>'12 months'</span></span>
   OR</span> o</span>.</span>created_at</span> IS NULL</span>;</span></span>
</span>
-- layer 3: one row per customer, ready for the dashboard</span></span>
CREATE VIEW</span> customer_summary</span> AS</span></span>
SELECT</span></span>
    co</span>.</span>id</span>,</span></span>
    co</span>.</span>email</span>,</span></span>
    co</span>.</span>name</span>,</span></span>
    COUNT</span>(</span>co</span>.</span>order_id</span>)                                   </span>AS</span> orders_12mo,</span></span>
    COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>)                     </span>AS</span> revenue_12mo_cents,</span></span>
    MAX</span>(</span>co</span>.</span>ordered_at</span>)                                   </span>AS</span> last_order_at,</span></span>
    COUNT</span>(</span>*</span>) </span>FILTER</span> (</span>WHERE</span> co</span>.</span>order_status</span> =</span> 'refunded'</span>) </span>AS</span> refunds_12mo</span></span>
FROM</span> customer_orders co</span></span>
GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre>
Each layer has one job. "Active customer" is defined exactly once - if marketing changes the ninety-day rule tomorrow, it is one line in one place, and the dashboard query collapses to SELECT * FROM customer_summary WHERE id = $1</code>.</p>
VIEWs also have the potential to be a real security boundary</strong>. Three hand-written queries means three places where a predicate can be forgotten, and three subtly different result sets a quarter later. With one definition, you have one result set.</p>
For simple views, there is no performance penalty either. PostgreSQL inlines them directly into the calling query, so the planner sees through the view and plans as if you had written the underlying SQL by hand. That mechanism, and the cases where it stops working, has a deep-dive of its own</a>. This article assumes it as background and focuses on the other side: what views are made of, and what happens when the schema underneath them moves.</p>
VIEWs are just like tables</a>
</h2>
The trouble starts with the syntax.</p>
SELECT * FROM customer_summary</code> reads exactly like SELECT * FROM customers</code>. The catalog reinforces it: pg_class.relkind = 'v'</code> instead of 'r'</code>, but everything else (column lists, grants, comments, even \d</code> output) looks the same. So developers reach for them the same way: joining them, nesting them, wrapping them in other views without thinking about it.</p>
A table is a heap with indexes. A view is a stored parse tree that gets expanded into your query at planning time. Most of the time the planner is clever enough that the difference does not matter: simple views are inlined, predicates are pushed down, the resulting plan is what you would have written by hand.</p>
The nested view spiral</a>
</h3>
In practice, what you get is the dependency tree from the introduction (customer_summary</code> on top of customer_orders</code> on top of active_customers</code>) and a new engineer who has no way to know that SELECT * FROM customer_summary WHERE id = $1</code> expands into a three-layer rewrite, a LEFT JOIN</code> against twelve months of orders, and a GROUP BY</code> that the planner cannot push the id</code> predicate through.</p>
Nothing in the query, the schema browser, or the ORM signals this. You only learn the shape of the tree by reading every definition top to bottom, which is exactly the work the abstraction was meant to save you.</p>
You cannot reason about the performance of a query against a view without reading the view, the views it depends on, and the tables underneath.</p>
Views are rewrite rules</a>
</h3>
A view is a macro. When you reference one, Postgres pastes its body into your query before the planner runs. There is no stored result, no view "object" the executor consults. The pg_class</code> row you see when you query the catalog is an empty shell: it holds the name, the column list, and the grants, but no definition.</p>
The definition itself, the SELECT</code> you wrote, is stored separately in pg_rewrite</code>:</p>
SELECT</span> ev_class::regclass, ev_type, is_instead</span></span>
FROM</span> pg_rewrite</span></span>
WHERE</span> ev_class </span>=</span> 'active_customers'</span>::regclass;</span></span></code></pre>     ev_class     | ev_type | is_instead</span></span>
------------------+---------+------------</span></span>
 active_customers | 1       | t</span></span></code></pre>
That row is the view. ev_type = 1</code> is SELECT</code>, is_instead = t</code> means "replace, do not augment" (the pg_class</code> shell has no rows to read, so substitution is the only option that makes sense). Between the parser and the planner Postgres runs a stage called the rewriter</strong>, and the rewriter's job is to walk the parse tree, find references to relations that have rules, and substitute the rule body in place. This way a query like SELECT * FROM active_customers WHERE id = $1</code> arrives at the planner already rewritten:</p>
-- what the planner actually sees</span></span>
SELECT *</span></span>
FROM</span> (</span></span>
    SELECT</span> c</span>.</span>id</span>, </span>c</span>.</span>email</span>, </span>c</span>.</span>name</span>, </span>c</span>.</span>status</span>,</span></span>
           c</span>.</span>last_login_at</span>, </span>c</span>.</span>deleted_at</span></span>
    FROM</span> customers c</span></span>
    WHERE</span> c</span>.</span>deleted_at</span> IS NULL</span></span>
      AND</span> c</span>.</span>status</span> =</span> 'active'</span></span>
      AND</span> c</span>.</span>last_login_at</span> > now</span>()</span> -</span> interval </span>'90 days'</span></span>
) active_customers</span></span>
WHERE</span> id </span>=</span> $</span>1</span>;</span></span></code></pre>
Notice that c.*</code> is already expanded into an explicit column list. That happened at CREATE VIEW</code> time and was frozen into the stored parse tree; the rewriter does not re-expand it. From here the planner can usually flatten the subquery into the outer query through a pass called subquery pull-up</strong>, leaving id = $1</code> as a predicate next to the others so an index on customers.id</code> is reachable.</p>
That flattening is what makes simple views free. When it bails out (because the view body has LIMIT</code>, DISTINCT</code>, an aggregate, a set operation, or a few other shapes) the subquery stays put and outer predicates cannot move past it. The full list of planner barriers and how to spot them in EXPLAIN</code></a> lives in the inlining post.</p>
Historically Postgres exposed this same machinery as user-facing CREATE RULE</code>, on the theory that arbitrary query rewriting was a general-purpose feature. It mostly was not. Rules are effectively deprecated outside of views; triggers do the same jobs without the surprises. The rule infrastructure survives because views need it.</span></p>
Once you hold that model, every awkward thing about views falls out of it. Two in particular drive the rest of this article.</p>
Columns are referenced by attribute number, not name.</strong> The stored parse tree does not remember that it reads customers.email</code>; it remembers that it reads attribute 2 of relation 16385. Rename the column and the view keeps working, because the OID and the position are unchanged. Drop a column in the middle and Postgres refuses upfront, because the stored attribute numbers would no longer line up. Every dependency error later in this article traces back to this.</p>
The body is expanded once per reference.</strong> Every mention of the view in a query produces an independent copy of its parse tree in the rewritten plan. For a stable expression that is invisible. For a volatile one like random()</code> or clock_timestamp()</code>, each copy evaluates separately:</p>
CREATE VIEW</span> v_rand</span> AS</span></span>
SELECT</span> id, random()</span> AS</span> r </span>FROM</span> customers;</span></span>
</span>
SELECT</span> a</span>.</span>id</span>, </span>a</span>.</span>r</span> AS</span> r_a, </span>b</span>.</span>r</span> AS</span> r_b</span></span>
FROM</span> v_rand a</span></span>
JOIN</span> v_rand b </span>ON</span> a</span>.</span>id</span> =</span> b</span>.</span>id</span></span>
LIMIT</span> 3</span>;</span></span></code></pre> id |         r_a          |         r_b</span></span>
----+----------------------+---------------------</span></span>
  1 |     0.84043639656488 |  0.0833458769902089</span></span>
  2 | 0.009846241116064247 |  0.6574000469586228</span></span>
  3 |  0.14667469313524628 | 0.21862693208148087</span></span></code></pre>
Same view, same row, two different r</code> values. The rewriter expanded v_rand</code> twice, so the rewritten query contains two independent random()</code> calls, and the planner has no reason to share them. You might hit this expecting a.r = b.r</code> and end up rewriting the view to push the volatile call outside.</p>
This is also why CREATE OR REPLACE VIEW</code> can only append columns to the end of the column list. Existing attribute numbers must stay stable for every dependent, so anything that would shift them is forbidden.</span></p>
A useful side effect of the same mechanism: views are immune to the search-path attacks that bite SECURITY DEFINER</code> functions. Identifiers are resolved to OIDs at CREATE VIEW</code> time, so a later SET search_path</code> cannot redirect a view to a different table. Functions resolve identifiers per call against the caller's search_path</code> unless you pin it; views never have that exposure.</p>
Writable views, the half-kept promise</a>
</h3>
Funnily enough, I was not aware writable views existed until earlier this year. It came up during a hallway conversation with Ervin Weber</a> and Jesper St John</a> at PgDay Nordic 2026 in Helsinki. Twelve years writing PostgreSQL and the auto-updatable rules had simply never crossed my desk.</span></p>
The original promise was symmetry: if a view looks like a table for reads, it should look like one for writes. Postgres delivers this for the easy case (a view over a single base table, no joins, no aggregates, no DISTINCT</code>) and calls them auto-updatable</strong>. You can INSERT</code>, UPDATE</code>, and DELETE</code> through them, and the rewrite rule translates the operation against the underlying table.</p>
INSERT INTO</span> active_customers (email, </span>name</span>)</span></span>
VALUES</span> (</span>'alice@example.com'</span>, </span>'Alice'</span>);</span></span>
-- becomes: INSERT INTO customers (email, name, status, last_login_at, deleted_at)</span></span>
--          VALUES ('alice@example.com', 'Alice', DEFAULT, DEFAULT, DEFAULT);</span></span></code></pre>
Add a join, an aggregate, a GROUP BY</code>, or anything else outside the auto-updatable rules and the write path goes silent. SELECT</code> still works, but INSERT</code> errors and you have to wire up INSTEAD OF</code> triggers by hand. So customer_orders</code> and customer_summary</code> are read-only by accident, not by design, and the rules for what counts are not visible in the view definition itself. You find out at write time, in production.</p>
WITH CHECK OPTION</code> rejects any write that would produce a row the view cannot see, with LOCAL</code> checking only this view's predicate and CASCADED</code> (the default) checking every underlying view up the chain. It is how you hand an application a writable, scoped slice of a table without trusting it to enforce the scope itself.</p>
Views and row-level security</a>
</h3>
RLS policies live on tables, not views. For a regular view that matters less than it sounds: the policies still fire when the view's query hits the base table. What determines whether they fire correctly is whose context the view runs under.</p>
By default, a view runs as its owner. Superusers bypass RLS unconditionally; table owners bypass it unless FORCE ROW LEVEL SECURITY</code> is set. A view owned by a privileged role is therefore a hole in any RLS scheme, silently, because nothing in the view definition or the catalog signals it.</p>
Since PostgreSQL 15 the fix is a single option:</p>
CREATE VIEW</span> my_view</span> WITH</span> (security_invoker </span>=</span> true) </span>AS</span></span>
  SELECT</span> ... </span>FROM</span> base_table;</span></span></code></pre>
With security_invoker</code> set, the view runs as the calling user and RLS evaluates against the actual caller. Before 15, the workaround was a dedicated low-privilege view owner plus FORCE ROW LEVEL SECURITY</code> on the base table.</p>
Materialized views are a different problem. REFRESH MATERIALIZED VIEW</code> runs as the owner, so the snapshot contains whatever the owner can see. Every user then reads that same materialized data. There is no per-row filtering because the rows are physically stored; security_invoker</code> has nothing to attach to.</p>
The architect's stigma</a>
</h3>
When I first got into databases, views were the thing senior people warned you off. "Don't use views in production", "views are slow", "views hide the real query". You learned to write the SQL inline, even when the same predicate appeared in twenty places, because the alternative was disapproval at the next code review.</p>
In Oracle shops the warning had specific roots. The cost-based optimiser of that era struggled to merge predicates through views with non-trivial projections, joins, or set operations, so a clean three-layer view could expand into a plan that scanned everything and filtered late. The shop-wide answer was almost always materialised views or PL/SQL packages that encapsulated every access path; a simple view sat awkwardly between the two, not cached like a matview, not encapsulated like a package, and with optimiser surprises on top.</span></p>
Postgres carried the same baggage by association. Nested-view spirals were real, rule-system surprises were real, and the optimiser of fifteen years ago was less capable than today's. The rule of thumb hardened into doctrine and outlived its conditions.</p>
The improvements are real; the reputation has not caught up. Reach for a view in a code review today and someone will still object on principle. Worth knowing before your PR hits review.</p>
What happens when you change a table</a>
</h2>
Everything covered so far stays invisible while the schema underneath holds still. View bodies live in pg_rewrite</code>, columns are pinned to attribute numbers, types are frozen at CREATE VIEW</code> time, and the planner just expands and runs.</p>
The moment someone tries to change a column the view tree references, every one of those mechanics surfaces at once. Suppose someone needs to drop the name</code> column from customers</code>:</p>
ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span></code></pre>ERROR:  cannot drop column name of table customers because other objects depend on it</span></span>
DETAIL:  view active_customers depends on column name of table customers</span></span>
HINT:  Use DROP ... CASCADE to drop the dependent objects too.</span></span></code></pre>
active_customers</code> references name</code> (the c.*</code> was expanded at view-creation time), customer_orders</code> inherits the dependency through ac.*</code>, and customer_summary</code> is another layer down. The whole tree is frozen on one column.</p>
It is not just dropping. Try widening a type:</p>
ALTER TABLE</span> customers </span>ALTER</span> COLUMN email </span>TYPE TEXT</span>;</span></span></code></pre>ERROR:  cannot alter type of a column used by a view or rule</span></span>
DETAIL:  rule _RETURN on view active_customers depends on column "email"</span></span></code></pre>
The same wall blocks any structural change to a column a view references: drop, rename, type change. Even widening VARCHAR(255)</code> to TEXT</code>, a change that loses no data and adds no constraint, is forbidden because the view's stored definition references the old type OID.</p>
The hint suggests CASCADE</code>:</p>
ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span> CASCADE;</span></span></code></pre>NOTICE:  drop cascades to 3 other objects</span></span>
DETAIL:  drop cascades to view active_customers</span></span>
drop cascades to view customer_orders</span></span>
drop cascades to view customer_summary</span></span></code></pre>
All three views are dropped, not modified, along with their grants and any other dependent objects. On a schema with dozens of interconnected views, CASCADE</code> is nightmare.</p>

Never use CASCADE on production views without a full recreation script ready.</strong>
CASCADE doesn't modify views, it drops them. All GRANTs, row-level security policies, and downstream dependencies disappear with them. There is no undo.
</div>
So the manual path is: save every view definition, drop them in reverse dependency order, alter the table, recreate them in forward dependency order, reapply all grants. For three views this is tedious. For thirty views across multiple schemas, it is a full migration project.</p>
-- 1. save definitions </span></span>
SELECT</span> pg_get_viewdef(</span>'customer_summary'</span>, true);</span></span>
SELECT</span> pg_get_viewdef(</span>'customer_orders'</span>, true);</span></span>
SELECT</span> pg_get_viewdef(</span>'active_customers'</span>, true);</span></span>
</span>
-- 2. drop in leaf-first order</span></span>
DROP VIEW</span> customer_summary;</span></span>
DROP VIEW</span> customer_orders;</span></span>
DROP VIEW</span> active_customers;</span></span>
</span>
-- 3. now you can alter the table</span></span>
ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span>
</span>
-- 4. recreate in correct order</span></span>
CREATE VIEW</span> active_customers</span> AS</span></span>
SELECT</span> id, email, </span>status</span>, last_login_at, deleted_at</span></span>
FROM</span> customers</span></span>
WHERE</span> deleted_at </span>IS NULL</span></span>
  AND status =</span> 'active'</span></span>
  AND</span> last_login_at </span>> now</span>()</span> -</span> interval </span>'90 days'</span>;</span></span>
</span>
CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span>
CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span>
</span>
-- 5. reapply grants and so on</span></span>
-- ...</span></span></code></pre>The SELECT * trap</a>
</h2>
You might think SELECT *</code> saves you from column-level dependencies. It does not. It makes things worse.</p>
CREATE VIEW</span> all_customers</span> AS</span></span>
SELECT * FROM</span> customers;</span></span></code></pre>
This looks flexible. PostgreSQL expands SELECT *</code> at view creation time</strong> and freezes the result. Check what the database actually stored:</p>
SELECT</span> pg_get_viewdef(</span>'all_customers'</span>::regclass, true);</span></span></code></pre> SELECT id,</span></span>
    email,</span></span>
    name,</span></span>
    status,</span></span>
    last_login_at,</span></span>
    deleted_at</span></span>
   FROM customers;</span></span></code></pre>
The *</code> was expanded to the columns that existed when the view was created. Now add a column to the base table:</p>
ALTER TABLE</span> customers </span>ADD</span> COLUMN phone </span>TEXT</span>;</span></span>
SELECT * FROM</span> all_customers;</span></span></code></pre>
The phone</code> column is not there</strong>. The view still returns the original columns. To pick up new columns, you have to CREATE OR REPLACE VIEW all_customers AS SELECT * FROM customers</code>, which re-expands the *</code> against the current table definition.</p>
Drop a column that was in the original expansion and the migration is blocked the same way it would be with an explicit column list, except now the dependency is hidden in the catalog instead of visible in the view body. SELECT *</code> gives the illusion of flexibility while creating the same rigid coupling.</p>
This is actually documented behaviour and follows the SQL standard. But it surprises nearly everyone the first time they encounter it, and it is the source of many "my view is missing columns" bug reports.</span></p>
Always use explicit column lists in views.</strong> At least then the dependency is visible and the breakage is predictable.</p>
The dependency wall, the type-OID coupling, the frozen SELECT *</code> expansion: all of it is the same trade-off, applied consistently, trading flexibility for catching breakage early.</p>
Where this isn't a PostgreSQL bug</a>
</h2>
The rigidity is a design choice, not a defect. Other databases made different choices, and none of them are free.</p>
Oracle</strong> marks dependent views as INVALID</code> rather than blocking the DDL. The views are recompiled automatically on next access. If they still work, great; if not, you get an error at query time. This sounds better until you realize what it means in practice: you can deploy a migration, get a clean exit code, and not discover that a critical reporting view is broken until a user hits it on Monday morning. The breakage moves from migration time, where you are paying attention, to runtime, where you may not be. For teams that value deployment confidence, this is arguably worse.</p>
SQL Server</strong> has sp_refreshview</code>, which recompiles a view's metadata against the current table definitions. You can alter a table and then refresh the dependent views to pick up the changes. But sp_refreshview</code> works on one view at a time. There is no built-in way to refresh an entire dependency chain in the right order. And if a view references a dropped column, sp_refreshview</code> fails; it does not remove the reference for you. You still have to manually edit and recreate the view. It is a convenience, not a solution.</p>
PostgreSQL chose compile-time safety: no surprise INVALID</code> views in production, no silently wrong results from a lazy recompilation papering over a structural change. The cost is manual dependency management on every schema change, and it is high enough to push experienced teams away from views entirely.</p>
The saving grace: transactional DDL</a>
</h2>
Before the workarounds, there is one PostgreSQL feature that materially changes the risk calculus: DDL is transactional</strong>. The entire drop-alter-recreate-regrant sequence can live inside a single BEGIN/COMMIT</code>:</p>
BEGIN</span>;</span></span>
DROP VIEW</span> customer_summary;</span></span>
DROP VIEW</span> customer_orders;</span></span>
DROP VIEW</span> active_customers;</span></span>
</span>
ALTER TABLE</span> customers </span>DROP</span> COLUMN </span>name</span>;</span></span>
</span>
CREATE VIEW</span> active_customers</span> AS</span> ...;</span></span>
CREATE VIEW</span> customer_orders</span> AS</span> ...;</span></span>
CREATE VIEW</span> customer_summary</span> AS</span> ...;</span></span>
</span>
GRANT SELECT ON</span> customer_summary </span>TO</span> reporting;</span></span>
COMMIT</span>;</span></span></code></pre>
If any statement raises (a typo in a recreated view, a missing grant, an unexpected dependency) the whole migration rolls back and the database is exactly as it was. Oracle and SQL Server cannot do this for most DDL; their CREATE VIEW</code> auto-commits, so a partial failure leaves you with a half-migrated schema and no ROLLBACK</code> to lean on. PostgreSQL's "you have to drop and recreate everything" pain is real, but the recovery story is much better than Oracle's or SQL Server's.</p>
This does not solve the locking problem. DROP VIEW</code> and CREATE VIEW</code> take AccessExclusiveLock</code> on the view; ALTER TABLE</code> takes one on the table. For the duration of the transaction, anything that touches those objects waits behind it. A long view-rebuild transaction on a busy system is a stop-the-world window. Keep the transaction tight: prepare the new SQL outside the transaction, hold no other locks, and do not run it during peak traffic. Transactional DDL gives you safety, not concurrency.</p>
The workarounds</a>
</h2>
None of these solve the fundamental problem; they make it manageable.</p>
Avoid views entirely.</strong> Replace views with application-level query builders or ORM scopes. You lose the shared abstraction: when three services need the same definition of "active customer," each one implements it independently. Definitions drift. For teams with full control over a small codebase, this is a reasonable trade. For shared data platforms, it is not.</p>
Script the recreation.</strong> Use pg_get_viewdef()</code> to extract definitions, then write migration scripts that drop and recreate views around schema changes. The problems: no impact analysis (you have to figure out the dependency order yourself), no grant preservation (view definitions do not include permissions), and it does not scale past a handful of views. Your migration framework will not help here either. They run your SQL files in order, but figuring out what</em> SQL to write is entirely on you.</p>
Version your views.</strong> Instead of modifying views in place, create active_customers_v2</code> alongside active_customers_v1</code>. Migrate consumers one at a time. Drop the old version when nothing uses it. This works well when views are consumed by multiple independent teams, since you cannot force everyone to migrate in the same deployment window. The trade-off is naming discipline and version proliferation: if customer_orders_v1</code> depends from active_customers_v1</code>, you now need customer_orders_v2</code> too.</p>
Schema-based versioning (CREATE SCHEMA api_v2; CREATE VIEW api_v2.active_customers AS ...</code>) is cleaner than suffixes for public-facing APIs. It gives you a natural namespace and lets you SET search_path</code> to switch versions.</span></p>
Query pg_depend yourself.</strong> PostgreSQL tracks all object dependencies in the pg_depend</code> system catalog. The catch: views do not depend on tables directly. The rewrite rule that implements the view does, so every traversal goes pg_depend -> pg_rewrite -> pg_class</code>. Direct view dependents of customers</code>:</p>
SELECT</span></span>
    depns</span>.</span>nspname</span> AS</span> dependent_schema,</span></span>
    depc</span>.</span>relname</span>  AS</span> dependent_view,</span></span>
    COALESCE</span>(</span></span>
        (</span>SELECT</span> a</span>.</span>attname</span> FROM</span> pg_attribute a</span></span>
         WHERE</span> a</span>.</span>attrelid</span> =</span> d</span>.</span>refobjid</span> AND</span> a</span>.</span>attnum</span> =</span> d</span>.</span>refobjsubid</span>),</span></span>
        '*'</span></span>
    ) </span>AS</span> source_column</span></span>
FROM</span> pg_depend d</span></span>
JOIN</span> pg_rewrite r       </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span>
JOIN</span> pg_class depc      </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span>
JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span>
WHERE</span> d</span>.</span>refobjid</span>    =</span> 'customers'</span>::regclass</span></span>
  AND</span> d</span>.</span>classid</span>     =</span> 'pg_rewrite'</span>::regclass</span></span>
  AND</span> d</span>.</span>refclassid</span>  =</span> 'pg_class'</span>::regclass</span></span>
  AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span>
  AND</span> depc</span>.</span>oid</span>     <></span> 'customers'</span>::regclass</span></span>
ORDER BY</span> depns</span>.</span>nspname</span>, </span>depc</span>.</span>relname</span>;</span></span></code></pre>
Useful, but only one hop. customer_summary</code> depends on customers</code> transitively through two intermediate views, which this query misses. A recursive CTE walks the chain:</p>
WITH RECURSIVE</span> view_deps </span>AS</span> (</span></span>
    -- direct dependents of the target table</span></span>
    SELECT DISTINCT</span></span>
        depc</span>.</span>oid</span>                              AS</span> view_oid,</span></span>
        depns</span>.</span>nspname</span> ||</span> '.'</span> ||</span> depc</span>.</span>relname</span>  AS</span> view_name,</span></span>
        1</span>                                     AS</span> depth</span></span>
    FROM</span> pg_depend d</span></span>
    JOIN</span> pg_rewrite r       </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span>
    JOIN</span> pg_class depc      </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span>
    JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span>
    WHERE</span> d</span>.</span>refobjid</span> =</span> 'customers'</span>::regclass</span></span>
      AND</span> d</span>.</span>classid</span>  =</span> 'pg_rewrite'</span>::regclass</span></span>
      AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span>
</span>
    UNION</span></span>
</span>
    -- views depending on views we already found</span></span>
    SELECT</span> depc</span>.</span>oid</span>,</span></span>
           depns</span>.</span>nspname</span> ||</span> '.'</span> ||</span> depc</span>.</span>relname</span>,</span></span>
           vd</span>.</span>depth</span> +</span> 1</span></span>
    FROM</span> view_deps vd</span></span>
    JOIN</span> pg_depend d        </span>ON</span> d</span>.</span>refobjid</span> =</span> vd</span>.</span>view_oid</span></span>
    JOIN</span> pg_rewrite r       </span>ON</span> r</span>.</span>oid</span> =</span> d</span>.</span>objid</span></span>
    JOIN</span> pg_class depc      </span>ON</span> depc</span>.</span>oid</span> =</span> r</span>.</span>ev_class</span></span>
    JOIN</span> pg_namespace depns </span>ON</span> depns</span>.</span>oid</span> =</span> depc</span>.</span>relnamespace</span></span>
    WHERE</span> d</span>.</span>classid</span>  =</span> 'pg_rewrite'</span>::regclass</span></span>
      AND</span> depc</span>.</span>relkind</span> IN</span> (</span>'v'</span>, </span>'m'</span>)</span></span>
      AND</span> depc</span>.</span>oid</span> <></span> vd</span>.</span>view_oid</span>  -- skip the view's own _RETURN rule</span></span>
)</span></span>
SELECT</span> view_name, </span>MIN</span>(depth) </span>AS</span> depth</span></span>
FROM</span> view_deps</span></span>
GROUP BY</span> view_name</span></span>
ORDER BY</span> depth, view_name;</span></span></code></pre>
The depc.oid <> vd.view_oid</code> filter matters: each view's _RETURN</code> rule has a self-dependency entry that without the filter sends the walk back to the view it just visited.</p>
This works, but at this point you are writing a dependency-analysis tool. Add topological sort, grant capture, matview handling, RLS-policy preservation, and you have small product.</p>

Materialized views have it worse.</strong> Everything above applies, plus you lose the cached result set on DROP</code>. After recreating, REFRESH</code> rebuilds from scratch: minutes to hours on large datasets, with no CONCURRENTLY</code> option until a unique index exists. For matviews that back dashboards, this means downtime. RLS is also off the table: REFRESH</code> runs as the matview owner, so the snapshot reflects the owner's access, not the querying user's.
</div>
Prior art hiding in pg_dump</a>
</h2>
The mechanism the manual workaround keeps reinventing, rewriting a view's body while preserving its identity, grants, and dependents, already exists inside PostgreSQL. It happens to live in pg_dump</code>, used only when needed.</p>
Most of the time pg_dump</code> does the obvious thing: builds a dependency graph, topologically sorts it, and emits each view in base-first order with a normal CREATE VIEW</code>. Dump our four-view chain and that is exactly what you get.</p>
The interesting case is when the sort fails. Cycles are uncommon but real: a view body that calls a function whose own body references the view back, a trigger on a base table that reads through a view reading the same table, cross-referencing matviews with RLS policies pointing back at views. When pg_dump</code> hits one, it falls back to the placeholder view trick</strong>: emit one of the views early as a stub with the right column list and types but a dummy body, then come back with CREATE OR REPLACE VIEW</code> to install the real definition once the rest of the cycle exists.</p>
-- emitted early, as a placeholder</span></span>
CREATE VIEW</span> customer_summary</span> AS</span></span>
SELECT</span></span>
    NULL</span>::</span>integer AS</span> id,</span></span>
    NULL</span>::</span>text    AS</span> email,</span></span>
    NULL</span>::</span>text    AS name</span>,</span></span>
    NULL</span>::</span>bigint  AS</span> orders_12mo,</span></span>
    NULL</span>::</span>bigint  AS</span> revenue_12mo_cents;</span></span>
</span>
-- emitted later, once dependencies exist</span></span>
CREATE OR REPLACE VIEW</span> customer_summary</span> AS</span></span>
SELECT</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>,</span></span>
       COUNT</span>(</span>co</span>.</span>order_id</span>)               </span>AS</span> orders_12mo,</span></span>
       COALESCE</span>(</span>SUM</span>(</span>co</span>.</span>total_cents</span>), </span>0</span>) </span>AS</span> revenue_12mo_cents</span></span>
FROM</span> customer_orders co</span></span>
GROUP BY</span> co</span>.</span>id</span>, </span>co</span>.</span>email</span>, </span>co</span>.</span>name</span>;</span></span></code></pre>
Grants, comments, and policies attach to the stub and survive the rewrite because the OID never changes. CREATE OR REPLACE VIEW</code> mutates the same pg_class</code> row in place. PostgreSQL has been doing this on every pg_dump --schema-only</code> for years. The mechanism is there; it is simply not exposed as user-facing DDL.</p>
Before you reach for a VIEW</a>
</h2>
Views are worth using. The abstraction is real: shared definitions, clean layering, column-level security, and no runtime cost when the planner can inline them. The trouble is that none of the rigidity is visible from the outside. A view looks like a table in the catalog, in \d</code>, in the ORM, and in every query that touches it. The teardown cost only appears the first time someone tries to drop a column, widen a type, or CASCADE</code> their way out of a migration on a Friday afternoon.</p>
So before you reach for one, hold the trade-off in mind:</p>

SELECT *</code> in a view body is a trap.</strong> It freezes the column list at creation time, hides the dependency in the catalog, and still blocks the same DDL an explicit list would. Always write the columns out.</li>
Every layer multiplies the teardown cost.</strong> A three-deep view chain means three drops, three recreates, three sets of grants and policies to reapply, in the right order, for any structural change to a column at the bottom. Keep dependency trees shallow, and be honest about whether the abstraction is paying for itself.</li>
CASCADE</code> is not a fix.</strong> It drops dependent views along with their grants, RLS policies, and downstream dependencies, with no undo. Never run it in production without a recreation script ready.</li>
security_invoker</code> is not the default.</strong> A view runs as its owner, which silently bypasses RLS on base tables when the owner is privileged. On PostgreSQL 15+, WITH (security_invoker = true)</code> is what makes the security boundary real. For materialized views there is no equivalent. Enforce access at the layer that reads the matview, never below it.</li>
</ul>
When the migration finally has to happen, two things make the cycle survivable. Wrap the whole drop-alter-recreate-regrant sequence in a transaction so a typo or missed grant rolls back cleanly, and keep that transaction tight — AccessExclusiveLock</code> blocks everything else for its duration. Then map the dependency graph before you cut: pg_depend</code> joined through pg_rewrite</code>, walked with a recursive CTE, tells you what actually breaks. Run that query before the migration, not after the incident.</p>
The pain traces back to one missing primitive. ALTER VIEW</code> today handles renames, owner and schema changes, column defaults, and options like security_barrier</code>, but nothing structural. CREATE OR REPLACE VIEW</code> can append columns at the end and nothing else. A real ALTER VIEW DROP COLUMN</code>, ADD COLUMN</code>, ALTER COLUMN TYPE</code> is what would make views safe to evolve. The catalog and the in-place-rewrite mechanism are already there, as pg_dump</code>'s placeholder trick demonstrates; the user-facing DDL is what is missing.</p>
Even without it, views are still worth using. Just don't pretend they're tables.</p>
EDIT (2026-05-17)</em>: The original article understated how RLS interacts with views. Added a section on security_invoker</code> and why the ownership model matters, plus a note on why materialized views are structurally incompatible with RLS.</p>


Good CTE, bad CTE
2026-03-29T14:47:00+02:00
The Common Table Expression</a>, or CTE, is often the first feature developers reach for beyond basic SQL, and often the only one. You write a subquery after WITH</code>, give it a name, and use it in the rest of your query. It only exists for the duration of that query.</p>
But the popularity of CTEs usually has less to do with modernizing code and more to do with the promise of imperative logic. For many, CTE acts as an easy to understand remedy for 'scary queries' and way how to force execution order on the database. The way how many write queries is as if they tell optimizer "first do this, then do that".</p>
This creates a problem. CTEs handle query decomposition, recursion and multi statement DDLs. Planner treats them differently depending how you write and use them though. For long time (prior PostgreSQL 12) CTEs acted as optimization fence. The planner couldn't push predicates into them, couldn't use indexes on the underlying tables. Couldn't do anything that materialize them and scan through the result.</p>
PostgreSQL 12 changed this. CTEs now get inlined, materialized, or something in between, depending on how you write them.</p>
Sample schema</a>
</h2>
We will use the same schema as in the article PostgreSQL Statistics: Why queries run slow</a>.</p>
CREATE TABLE</span> customers</span> (</span></span>
    id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    name text NOT NULL</span></span>
);</span></span>
</span>
CREATE TABLE</span> orders</span> (</span></span>
    id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    customer_id </span>integer NOT NULL REFERENCES</span> customers(id),</span></span>
    amount </span>numeric</span>(</span>10</span>,</span>2</span>)</span> NOT NULL</span>,</span></span>
    status text NOT NULL DEFAULT</span> 'pending'</span>,</span></span>
    note </span>text</span>,</span></span>
    created_at </span>date NOT NULL DEFAULT</span> CURRENT_DATE</span></span>
);</span></span>
</span>
CREATE TABLE</span> orders_archive</span> (</span>LIKE</span> orders INCLUDING ALL EXCLUDING </span>IDENTITY</span>);</span></span>
</span>
INSERT INTO</span> customers (</span>name</span>)</span></span>
SELECT</span> 'Customer '</span> ||</span> i</span></span>
FROM</span> generate_series</span>(</span>1</span>, </span>2000</span>) </span>AS</span> i;</span></span>
</span>
INSERT INTO</span> orders (customer_id, amount, </span>status</span>, note, created_at)</span></span>
SELECT</span></span>
    (random()</span> *</span> 1999</span> +</span> 1</span>)::</span>int</span>,</span></span>
    (random()</span> *</span> 500</span> +</span> 5</span>)::</span>numeric</span>(</span>10</span>,</span>2</span>),</span></span>
    (</span>ARRAY</span>['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int],</span></span>
    CASE WHEN</span> random()</span> <</span> 0</span>.</span>3</span> THEN</span> 'Some note text here for padding'</span> ELSE NULL END</span>,</span></span>
    '2022-01-01'</span>::</span>date +</span> (random()</span> *</span> 1095</span>)::</span>int</span></span>
FROM</span> generate_series</span>(</span>1</span>, </span>100000</span>);</span></span>
</span>
ANALYZE customers;</span></span>
ANALYZE orders;</span></span></code></pre>
For recursive examples later on we'll also need an employees</code> table with a self-referencing hierarchy:</p>
CREATE TABLE</span> employees</span> (</span></span>
    id </span>integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    name text NOT NULL</span>,</span></span>
    manager_id </span>integer REFERENCES</span> employees(id),</span></span>
    department </span>text NOT NULL</span></span>
);</span></span>
</span>
INSERT INTO</span> employees (</span>name</span>, manager_id, department) </span>VALUES</span></span>
    (</span>'Alice'</span>,   </span>NULL</span>, </span>'Engineering'</span>),</span></span>
    (</span>'Bob'</span>,     </span>1</span>,    </span>'Engineering'</span>),</span></span>
    (</span>'Charlie'</span>, </span>1</span>,    </span>'Engineering'</span>),</span></span>
    (</span>'Diana'</span>,   </span>2</span>,    </span>'Engineering'</span>),</span></span>
    (</span>'Eve'</span>,     </span>2</span>,    </span>'Engineering'</span>),</span></span>
    (</span>'Frank'</span>,   </span>3</span>,    </span>'Sales'</span>),</span></span>
    (</span>'Grace'</span>,   </span>3</span>,    </span>'Sales'</span>),</span></span>
    (</span>'Hank'</span>,    </span>6</span>,    </span>'Sales'</span>),</span></span>
    (</span>'Ivy'</span>,     </span>6</span>,    </span>'Sales'</span>);</span></span>
</span>
ANALYZE employees;</span></span></code></pre>The optimization fence era (pre-PG 12)</a>
</h2>
As we already covered before PostgreSQL 12, every CTE was materialized. No exceptions. The planner would compute the CTE result set in full, store it in a temporary tuplestore, and then scan that tuplestore whenever the main query referenced the CTE. This made CTEs an optimization fence</strong> because the planner could not look through them.</p>
Consider this simple query:</p>
EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span>
)</span></span>
SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre>
On PostgreSQL 11 and earlier, the EXPLAIN</code> output would look something like this:</p>
                            QUERY PLAN</span></span>
-------------------------------------------------------------------</span></span>
 CTE Scan on filtered  (cost=1840.00..2290.00 rows=2 width=58)</span></span>
   Filter: (status = 'pending')</span></span>
   CTE filtered</span></span>
     ->  Seq Scan on orders  (cost=0.00..1840.00 rows=10000 width=58)</span></span>
           Filter: (created_at > '2025-01-01'::date)</span></span></code></pre>
Notice what happens here. The CTE runs a sequential scan on orders</code> with the date filter. It materializes all matching rows. Then the outer query applies the status = 'pending'</code> filter after</em> materialization. Even if a composite index on (created_at, status)</code> existed, the planner couldn't use it as it can't see through the CTE boundary to combine the predicates.</p>
Why was it designed this way? Two reasons. First, reason was snapshot isolation. Materializing the CTE guaranteed that the result set was computed once, from a single snapshot, regardless of how many times it was referenced. Second, as protection for side-effect edge cases. If a CTE contained a data-modifying statement (INSERT</code>, UPDATE</code>, DELETE</code>), materialising ensured it executed exactly once.</p>
The workaround was well-known in the community: rewrite CTEs as subqueries. Subqueries had always been subject to the planner's normal optimisation rules, including predicate pushdown and inlining. The same query written as SELECT * FROM (SELECT * FROM orders WHERE created_at > '2025-01-01') sub WHERE status = 'pending'</code> would produce a much better plan.</div>
This led to an entire workaround culture. Developers would write queries with CTEs for readability during development, then rewrite them as nested subqueries for production. The community had a saying: CTEs are optimization fences</em>. It was repeated so often. Many developers still believe it today. But it hasn't been true since PostgreSQL 12.</p>
PostgreSQL 12: CTE inlining</a>
</h2>
PostgreSQL 12 introduced automatic CTE inlining. Non-recursive, side-effect-free, singly-referenced CTEs are now inlined by default. The planner started treating them as subqueries and applies all its normal optimisations. Predicate pushdown, index usage, join reordering apply exactly as if the CTE syntax never existed.</p>
The same query from the previous section now produces a completely different plan:</p>
EXPLAIN </span>WITH</span> filtered </span>AS</span> (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span>
)</span></span>
SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre>                                    QUERY PLAN</span></span>
---------------------------------------------------------------------------</span></span>
 Seq Scan on orders  (cost=0.00..2355.00 rows=2 width=58)</span></span>
   Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span></code></pre>
The CTE is gone from the plan entirely. Both predicates are merged into a single scan on orders</code>. If there's a suitable index, the planner can use it. The CTE syntax doesn't change the execution plan.</p>
PostgreSQL 12 also introduced two new keywords that let you override the planner's decision:</p>

MATERIALIZED</code> - forces the CTE to materialize, even when the planner would inline it</li>
NOT MATERIALIZED</code> - forces inlining, even when the planner would materialize it</li>
</ul>
-- force materialization</span></span>
EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span>
)</span></span>
SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span>
</span>
-- Force inlining</span></span>
EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span>
)</span></span>
SELECT * FROM</span> filtered </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre>
This follows the same principle as VIEW inlining</a>.</p>
When does a CTE get materialized?</a>
</h2>
Case 1: single reference, no side effects (INLINED)</a>
</h3>
The simplest and most common case. If you reference the CTE exactly once and it contains no side effects, the planner inlines it.</p>
EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2025-01-01'</span></span>
)</span></span>
SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre>                                  QUERY PLAN</span></span>
------------------------------------------------------------------------------</span></span>
 Seq Scan on orders  (cost=0.00..2355.00 rows=2 width=59)</span></span>
   Filter: ((created_at > '2025-01-01'::date) AND (status = 'pending'::text))</span></span>
(2 rows)</span></span></code></pre>
Both predicates are merged. The planner considers all access paths on orders</code> directly.</p>
Case 2: multiple references (MATERIALIZED)</a>
</h3>
When a CTE is referenced more than once, the planner materializes it. This is actually a feature, CTE is computed once and reused. Therefore avoiding redundant work.</p>
EXPLAIN </span>WITH</span> summary </span>AS</span> (</span></span>
    SELECT status</span>, </span>count</span>(</span>*</span>) </span>AS</span> cnt </span>FROM</span> orders </span>GROUP BY status</span></span>
)</span></span>
SELECT</span> a</span>.</span>status</span>, </span>b</span>.</span>status</span></span>
FROM</span> summary a, summary b</span></span>
WHERE</span> a</span>.</span>cnt</span> ></span> b</span>.</span>cnt</span>;</span></span></code></pre>                                 QUERY PLAN</span></span>
----------------------------------------------------------------------------</span></span>
 Nested Loop  (cost=2355.04..2355.52 rows=5 width=64)</span></span>
   Join Filter: (a.cnt > b.cnt)</span></span>
   CTE summary</span></span>
     ->  HashAggregate  (cost=2355.00..2355.04 rows=4 width=17)</span></span>
           Group Key: orders.status</span></span>
           ->  Seq Scan on orders  (cost=0.00..1855.00 rows=100000 width=9)</span></span>
   ->  CTE Scan on summary a  (cost=0.00..0.08 rows=4 width=40)</span></span>
   ->  CTE Scan on summary b  (cost=0.00..0.08 rows=4 width=40)</span></span>
(8 rows)</span></span></code></pre>
The CTE Scan</code> nodes appear twice, but the HashAggregate</code> runs only once. For expensive computations referenced multiple times, this is exactly what you want.</p>
Case 3: recursive CTE (ALWAYS MATERIALIZED)</a>
</h3>
Recursive CTEs must maintain a working table between iterations. There's no way to inline them. We'll cover recursion in detail later in the article.</p>
EXPLAIN </span>WITH RECURSIVE</span> subordinates </span>AS</span> (</span></span>
    SELECT</span> id, </span>name</span>, manager_id </span>FROM</span> employees </span>WHERE</span> id </span>=</span> 1</span></span>
    UNION ALL</span></span>
    SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span></span>
    FROM</span> employees e</span></span>
    JOIN</span> subordinates s </span>ON</span> e</span>.</span>manager_id</span> =</span> s</span>.</span>id</span></span>
)</span></span>
SELECT * FROM</span> subordinates;</span></span></code></pre>                                          QUERY PLAN</span></span>
-------------------------------------------------------------------------------</span></span>
 CTE Scan on subordinates  (cost=17.21..18.83 rows=81 width=40)</span></span>
   CTE subordinates</span></span>
     ->  Recursive Union  (cost=0.00..17.21 rows=81 width=13)</span></span>
           ->  Seq Scan on employees  (cost=0.00..1.11 rows=1 width=13)</span></span>
                 Filter: (id = 1)</span></span>
           ->  Hash Join  (cost=0.33..1.53 rows=8 width=13)</span></span>
                 Hash Cond: (e.manager_id = s.id)</span></span>
                 ->  Seq Scan on employees e  (cost=0.00..1.09 rows=9 width=13)</span></span>
                 ->  Hash  (cost=0.20..0.20 rows=10 width=4)</span></span>
                       ->  WorkTable Scan on subordinates s  (cost=0.00..0.20 rows=10 width=4)</span></span>
(10 rows)</span></span></code></pre>Case 4: data-modifying CTE (ALWAYS MATERIALIZED)</a>
</h3>
CTEs that contain INSERT</code>, UPDATE</code>, or DELETE</code> are always materialized. The side effects must execute exactly once, in a predictable order.</p>
EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span>
    DELETE FROM</span> orders </span>WHERE status =</span> 'cancelled'</span> RETURNING </span>*</span></span>
)</span></span>
SELECT</span> count</span>(</span>*</span>) </span>FROM</span> deleted;</span></span></code></pre>                                QUERY PLAN</span></span>
---------------------------------------------------------------------------</span></span>
 Aggregate  (cost=2670.13..2670.14 rows=1 width=8)</span></span>
   CTE deleted</span></span>
     ->  Delete on orders  (cost=0.00..2105.00 rows=25117 width=6)</span></span>
           ->  Seq Scan on orders  (cost=0.00..2105.00 rows=25117 width=6)</span></span>
                 Filter: (status = 'cancelled'::text)</span></span>
   ->  CTE Scan on deleted  (cost=0.00..502.34 rows=25117 width=0)</span></span>
(6 rows)</span></span></code></pre>
The CTE Scan</code> is present because the DELETE</code> must be fully executed before the count(*)</code> can run.</p>
Case 5: VOLATILE function (MATERIALIZED)</a>
</h3>
If a CTE contains a VOLATILE</code> function, the planner materializes it to prevent the function from being evaluated multiple times with potentially different results.</p>
EXPLAIN </span>WITH</span> rand </span>AS</span> (</span></span>
    SELECT</span> id, random()</span> AS</span> r </span>FROM</span> orders</span></span>
)</span></span>
SELECT * FROM</span> rand </span>WHERE</span> r </span><</span> 0</span>.</span>01</span>;</span></span></code></pre>                              QUERY PLAN</span></span>
-----------------------------------------------------------------------</span></span>
 CTE Scan on rand  (cost=2105.00..4355.00 rows=33333 width=12)</span></span>
   Filter: (r < '0.01'::double precision)</span></span>
   CTE rand</span></span>
     ->  Seq Scan on orders  (cost=0.00..2105.00 rows=100000 width=12)</span></span>
(4 rows)</span></span></code></pre>
Even though rand</code> is referenced only once, the CTE Scan</code> is there. random()</code> is VOLATILE</code>, which forces materialization.</p>
Case 6: STABLE functions (INLINED)</a>
</h3>
STABLE</code> functions like now()</code> do not</strong> prevent inlining. The reason is that the time is frozen at transaction start.</p>
EXPLAIN </span>WITH</span> recent </span>AS</span> (</span></span>
    SELECT * FROM</span> orders</span></span>
    WHERE</span> created_at </span>> now</span>()</span> -</span> interval </span>'7 days'</span></span>
)</span></span>
SELECT * FROM</span> recent </span>WHERE status =</span> 'pending'</span>;</span></span></code></pre>                                       QUERY PLAN</span></span>
-------------------------------------------------------------------------------</span></span>
 Seq Scan on orders  (cost=0.00..2855.00 rows=2 width=59)</span></span>
   Filter: ((status = 'pending'::text) AND (created_at > (now() - '7 days'::interval)))</span></span>
(2 rows)</span></span></code></pre>
No CTE Scan</code>. The planner inlines the CTE and merges both predicates, just like Case 1. now()</code> is STABLE</code> as it returns the same value within a transaction - and the planner's inlining check only looks for VOLATILE</code> functions (via contain_volatile_functions()</code>). STABLE</code> passes that check.</p>

Why do people think STABLE</code> blocks inlining? Because before PostgreSQL 12, all</strong> CTEs were materialized regardless of volatility. When PG 12 introduced inlining, the only function-level barrier was VOLATILE</code>. But the old "CTEs are optimization fences" mental model was so deeply ingrained that many developers assumed STABLE</code> was also a problem. It isn't.</p>
The function that actually blocks inlining: clock_timestamp()</code>. Unlike now()</code>, it's VOLATILE</code> and returns a different value on every call. A CTE with clock_timestamp()</code> will be materialized. Similarly, random()</code> and nextval()</code> are VOLATILE</code> and force materialization (as shown in Case 5).</p>
</div>
If you see a CTE with now()</code> being materialized, the cause is something else. Either multiple references, a data-modifying statement, or an explicit MATERIALIZED</code> hint. Don't blame STABLE</code>.</p>
Case 7: forcing behaviour with hints</a>
</h3>
You can always override the planner's decision.</p>
-- force materialization on something that would normally be inlined</span></span>
EXPLAIN </span>WITH</span> filtered </span>AS</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span>
)</span></span>
SELECT * FROM</span> filtered </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre>                              QUERY PLAN</span></span>
----------------------------------------------------------------------</span></span>
 CTE Scan on filtered  (cost=2105.00..2670.58 rows=5290 width=92)</span></span>
   Filter: (amount > '400'::numeric)</span></span>
   CTE filtered</span></span>
     ->  Seq Scan on orders  (cost=0.00..2105.00 rows=25137 width=59)</span></span>
           Filter: (status = 'pending'::text)</span></span>
(5 rows)</span></span></code></pre>-- force inlining on something that would normally be materialized</span></span>
EXPLAIN </span>WITH</span> filtered </span>AS NOT</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span></span>
)</span></span>
SELECT * FROM</span> filtered a</span></span>
JOIN</span> filtered b </span>ON</span> a</span>.</span>customer_id</span> =</span> b</span>.</span>customer_id</span>;</span></span></code></pre>                                    QUERY PLAN</span></span>
--------------------------------------------------------------------------------</span></span>
 Hash Join  (cost=2419.21..10686.65 rows=317742 width=118)</span></span>
   Hash Cond: (orders.customer_id = orders_1.customer_id)</span></span>
   ->  Seq Scan on orders  (cost=0.00..2105.00 rows=25137 width=59)</span></span>
         Filter: (status = 'pending'::text)</span></span>
   ->  Hash  (cost=2105.00..2105.00 rows=25137 width=59)</span></span>
         ->  Seq Scan on orders orders_1  (cost=0.00..2105.00 rows=25137 width=59)</span></span>
               Filter: (status = 'pending'::text)</span></span>
(7 rows)    </span></span></code></pre>
If the CTE were materialized, you would see a single scan of the orders</code> table, followed by two CTE Scans on the result. Instead, the planner has treated your query as if you had written a standard join between two subqueries.</p>

Be careful with NOT MATERIALIZED</code> on multiply-referenced CTEs.</strong> When you force inlining, the subquery runs once for each reference. In the example above, the orders</code> table is scanned twice. Once for a</code> and once for b</code>. For small result sets this might be fine. For large ones, you're doing double the work. Measure before using.</p>
</div>
Case 8: FOR UPDATE / FOR SHARE (MATERIALIZED)</a>
</h3>
Row-locking clauses force materialization even on a singly-referenced, side-effect-free CTE. Internally, the planner's contain_dml()</code> check treats FOR UPDATE</code> and FOR SHARE</code> the same as data-modifying statements.</p>
EXPLAIN </span>WITH</span> locked </span>AS</span> (</span></span>
    SELECT * FROM</span> orders </span>WHERE status =</span> 'pending'</span> FOR UPDATE</span></span>
)</span></span>
SELECT * FROM</span> locked </span>WHERE</span> amount </span>></span> 400</span>;</span></span></code></pre>                                 QUERY PLAN</span></span>
----------------------------------------------------------------------------</span></span>
 CTE Scan on locked  (cost=2356.37..2921.95 rows=5290 width=92)</span></span>
   Filter: (amount > '400'::numeric)</span></span>
   CTE locked</span></span>
     ->  LockRows  (cost=0.00..2356.37 rows=25137 width=65)</span></span>
           ->  Seq Scan on orders  (cost=0.00..2105.00 rows=25137 width=65)</span></span>
                 Filter: (status = 'pending'::text)</span></span>
(6 rows)</span></span></code></pre>
Without FOR UPDATE</code>, this CTE would be inlined. The LockRows</code> node and CTE Scan</code> confirm materialization.</p>
Decision matrix</a>
</h3>
Here's the full picture across PostgreSQL versions:</p>
Condition</th> PG ≤ 11</th> PG 12–16</th> PG 17–18</th></tr></thead>

Single ref, pure SELECT</td> Materialized</td> Inlined</td> Inlined</td></tr>
Multiple refs, pure SELECT</td> Materialized</td> Materialized</td> Materialized (better stats)</td></tr>
VOLATILE function</td> Materialized</td> Materialized</td> Materialized</td></tr>
STABLE function</td> Materialized</td> Inlined</td> Inlined</td></tr>
Data-modifying (DML)</td> Materialized</td> Materialized</td> Materialized</td></tr>
FOR UPDATE / FOR SHARE</td> Materialized</td> Materialized</td> Materialized</td></tr>
Recursive</td> Materialized</td> Materialized</td> Materialized</td></tr>
Explicit MATERIALIZED</code></td> -</td> Materialized</td> Materialized</td></tr>
Explicit NOT MATERIALIZED</code></td> -</td> Inlined</td> Inlined</td></tr>
</tbody></table>
The statistics black hole</a>
</h2>
As mentioned in PostgreSQL Statistics: Why queries run slow</a>, materialized CTEs are one of the places "where no statistics go." This is arguably the biggest practical problem with CTE materialization.</p>
When the planner materializes a CTE, the result set is stored in a temporary tuplestore. This tuplestore has no pg_statistic</code> entries - no histograms, no MCVs, no correlation data. The planner has to estimate row counts and value distributions using hardcoded defaults.</p>
Let's see this in action. Here's a CTE over 10,000 rows:</p>
EXPLAIN </span>WITH</span> all_orders </span>AS</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders</span></span>
)</span></span>
SELECT * FROM</span> all_orders </span>WHERE status =</span> 'pending'</span> AND</span> amount </span>></span> 400</span>;</span></span></code></pre>                              QUERY PLAN</span></span>
-----------------------------------------------------------------------</span></span>
 CTE Scan on all_orders  (cost=1855.00..4355.00 rows=5290 width=92)</span></span>
   Filter: ((amount > '400'::numeric) AND (status = 'pending'::text))</span></span>
   CTE all_orders</span></span>
     ->  Seq Scan on orders  (cost=0.00..1855.00 rows=100000 width=59)</span></span>
(4 rows)</span></span></code></pre>
The planner estimated 5,290 rows. Where does that number come from? The planner has no MCV list for status</code> inside the CTE, no histogram for amount</code>. It falls back to default selectivities of 0.3333</code> for the range comparison on amount</code> and a rough guess for the equality on status</code>, and multiplies them against the 100,000 input rows.</p>
If this CTE were inlined, the planner would read the actual statistics from pg_statistic</code> for the orders</code> table and produce estimates based on real data distribution, not defaults.</div>
In a simple query this might not matter much. But when a materialized CTE feeds into a join, default-based estimates can cascade. The planner might choose a nested loop where a hash join would be better, or vice versa. It might underestimate memory needs and spill to disk unexpectedly.</p>
PG 17: Statistics propagation</a>
</h3>
PostgreSQL 17 brought two significant improvements to materialized CTEs:</p>
Column statistics propagation.</strong> When the planner creates a CTE Scan</code> node, it now propagates column statistics from the underlying query into the scan node. This means n_distinct</code>, MCV lists, and histograms from the source table can inform estimates on the CTE scan.</p>
Path key propagation.</strong> Materialized CTEs now preserve sort order information. If the CTE's subquery produces sorted output, the planner knows about it and can skip redundant sorts downstream.</p>
These improvements significantly reduce the estimation gap, but they don't eliminate it. Inlined CTEs are still strictly better for planning accuracy, because the planner works directly with the base table statistics rather than propagated copies. If your CTE doesn't need to be materialized, don't force it.</p>
When materialization helps</a>
</h2>
Materialization isn't always bad.</p>
Multiple references.</strong> If the CTE result is used in multiple places, materialization computes it once. Without it, the subquery runs once per reference.</p>
EXPLAIN </span>WITH</span> monthly_totals </span>AS</span> (</span></span>
    SELECT</span> date_trunc(</span>'month'</span>, created_at) </span>AS month</span>,</span></span>
           status</span>,</span></span>
           sum</span>(amount) </span>AS</span> total</span></span>
    FROM</span> orders</span></span>
    GROUP BY</span> 1</span>, </span>2</span></span>
)</span></span>
SELECT</span> cur</span>.</span>month</span>, </span>cur</span>.</span>status</span>, </span>cur</span>.</span>total</span>,</span></span>
       prev</span>.</span>total</span> AS</span> prev_month_total,</span></span>
       cur</span>.</span>total</span> -</span> prev</span>.</span>total</span> AS</span> delta</span></span>
FROM</span> monthly_totals cur</span></span>
LEFT JOIN</span> monthly_totals prev</span></span>
    ON</span> cur</span>.</span>month</span> =</span> prev</span>.</span>month</span> +</span> interval </span>'1 month'</span></span>
    AND</span> cur</span>.</span>status</span> =</span> prev</span>.</span>status</span>; </span></span></code></pre>                                                  QUERY PLAN</span></span>
-------------------------------------------------------------------------------</span></span>
 Merge Left Join  (cost=3887.46..3990.90 rows=4384 width=136)</span></span>
   Merge Cond: ((cur.month = ((prev.month + '1 mon'::interval))) AND (cur.status = prev.status))</span></span>
   CTE monthly_totals</span></span>
     ->  HashAggregate  (cost=3105.00..3181.72 rows=4384 width=49)</span></span>
           Group Key: date_trunc('month'::text, (orders.created_at)::timestamp with time zone), orders.status</span></span>
           ->  Seq Scan on orders  (cost=0.00..2355.00 rows=100000 width=23)</span></span>
   ->  Sort  (cost=352.87..363.83 rows=4384 width=72)</span></span>
         Sort Key: cur.month, cur.status</span></span>
         ->  CTE Scan on monthly_totals cur  (cost=0.00..87.68 rows=4384 width=72)</span></span>
   ->  Sort  (cost=352.87..363.83 rows=4384 width=72)</span></span>
         Sort Key: ((prev.month + '1 mon'::interval)), prev.status</span></span>
         ->  CTE Scan on monthly_totals prev  (cost=0.00..87.68 rows=4384 width=72)</span></span>
(12 rows)</span></span></code></pre>
The aggregation runs once. Both cur</code> and prev</code> read from the materialized result. Without materialization, the entire aggregation would run twice.</p>
Expensive VOLATILE expressions.</strong> If a CTE contains calls to volatile functions or expensive computations, materialization ensures they execute exactly once.</p>
Data-modifying operations.</strong> The whole point of writable CTEs is that the side effects happen once and the RETURNING</code> data is available downstream. Materialization is not optional here.</p>
When inlining isn't enough</a>
</h2>
The imperative mindset from the introduction, "first do this, then do that", doesn't go away just because the planner inlines your CTEs. And this one is my personal favourite, and source that never stop delivering query refactoring.</p>
Developers still structure queries as sequential pipelines, and that structure itself can create performance problems that have nothing to do with materialization.</p>
A common pattern is building queries as an assembly line: one CTE filters rows, the next LEFT JOINs related tables and aggregates metadata with GROUP BY</code>, the next filters on the aggregated results. It reads like a clean pipeline, but the GROUP BY</code> in the middle creates a wall the planner can't optimize past.</p>
WITH</span> recent_orders </span>AS</span> (</span></span>
    SELECT * FROM</span> orders </span>WHERE</span> created_at </span>></span> '2024-01-01'</span></span>
),</span></span>
order_metadata </span>AS</span> (</span></span>
    SELECT</span></span>
        o</span>.</span>id</span>,</span></span>
        bool_or(</span>oa</span>.</span>id</span> IS NOT NULL</span>) </span>AS</span> was_archived,</span></span>
        count</span>(</span>o2</span>.</span>id</span>) </span>AS</span> related_count</span></span>
    FROM</span> recent_orders o</span></span>
    LEFT JOIN</span> orders_archive oa </span>ON</span> o</span>.</span>id</span> =</span> oa</span>.</span>id</span></span>
    LEFT JOIN</span> orders o2 </span>ON</span> o</span>.</span>customer_id</span> =</span> o2</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span>
    GROUP BY</span> o</span>.</span>id</span></span>
)</span></span>
SELECT</span> o.</span>*</span>, </span>m</span>.</span>was_archived</span>, </span>m</span>.</span>related_count</span></span>
FROM</span> recent_orders o</span></span>
JOIN</span> order_metadata m </span>ON</span> o</span>.</span>id</span> =</span> m</span>.</span>id</span></span>
WHERE</span> m</span>.</span>was_archived</span> =</span> false</span></span>
  AND</span> m</span>.</span>related_count</span> ></span> 0</span>;</span></span></code></pre>
Each CTE here is referenced once, so they all get inlined. No materialization, no optimization fence. The planner sees the full query. So what's the problem?</p>
The GROUP BY</code> in order_metadata</code>. Even after inlining, the planner cannot push the was_archived = false</code> predicate past the aggregation. It must first LEFT JOIN every filtered order against orders_archive</code> and self-join orders</code>, compute the aggregates for all of them, and only then discard the rows that don't match. If recent_orders</code> returns 50,000 rows but only 200 were ever archived, you're joining and aggregating 49,800 rows for nothing.</p>
The fix is to replace the aggregation-then-filter pattern with correlated EXISTS</code> subqueries:</p>
SELECT</span> o.</span>*</span></span>
FROM</span> orders o</span></span>
WHERE</span> o</span>.</span>created_at</span> ></span> '2024-01-01'</span></span>
  AND NOT EXISTS</span> (</span></span>
    SELECT</span> 1</span> FROM</span> orders_archive oa </span>WHERE</span> oa</span>.</span>id</span> =</span> o</span>.</span>id</span></span>
  )</span></span>
  AND EXISTS</span> (</span></span>
    SELECT</span> 1</span> FROM</span> orders o2</span></span>
    WHERE</span> o2</span>.</span>customer_id</span> =</span> o</span>.</span>customer_id</span> AND</span> o2</span>.</span>id</span> !=</span> o</span>.</span>id</span></span>
  );</span></span></code></pre>
EXISTS</code> short-circuits after finding the first matching row. The planner can push created_at > '2024-01-01'</code> all the way down to an index scan on orders</code>, then probe each related table per result. No aggregation, no wasted work.</p>

The rule of thumb:</strong> if your CTE contains a GROUP BY</code> or a LEFT JOIN</code> just to compute a boolean ("does this row have related data?"), you've built a wall the planner can't see past. A correlated EXISTS</code> lets the planner push filters down and stop scanning early. This applies whether the CTE is materialized or inlined.</p>
</div>
Writable CTEs (the power and the traps)</a>
</h2>
Data-modifying CTEs let you INSERT</code>, UPDATE</code>, or DELETE</code> inside a WITH</code> clause and use the RETURNING</code> data in subsequent CTEs or the main query.</p>
EXPLAIN </span>WITH</span> deleted </span>AS</span> (</span></span>
    DELETE FROM</span> orders</span></span>
    WHERE status =</span> 'cancelled'</span></span>
      AND</span> created_at </span><</span> '2023-01-01'</span></span>
    RETURNING </span>*</span></span>
),</span></span>
archived </span>AS</span> (</span></span>
    INSERT INTO</span> orders_archive</span></span>
    SELECT * FROM</span> deleted</span></span>
    RETURNING id</span></span>
)</span></span>
SELECT</span> count</span>(</span>*</span>) </span>FROM</span> archived;</span></span></code></pre>                                          QUERY PLAN</span></span>
----------------------------------------------------------------------------------------------</span></span>
 Aggregate  (cost=2709.19..2709.20 rows=1 width=8)</span></span>
   CTE deleted</span></span>
     ->  Delete on orders  (cost=0.00..2355.00 rows=8334 width=6)</span></span>
           ->  Seq Scan on orders  (cost=0.00..2355.00 rows=8334 width=6)</span></span>
                 Filter: ((created_at < '2023-01-01'::date) AND (status = 'cancelled'::text))</span></span>
   CTE archived</span></span>
     ->  Insert on orders_archive  (cost=0.00..166.68 rows=8334 width=92)</span></span>
           ->  CTE Scan on deleted  (cost=0.00..166.68 rows=8334 width=92)</span></span>
   ->  CTE Scan on archived  (cost=0.00..166.68 rows=8334 width=0)</span></span>
(9 rows)  </span></span></code></pre>
This deletes old cancelled orders, moves them to an archive table, and counts how many were archived - all in a single atomic statement, no application-level coordination needed.</p>
But there are sharp edges.</p>
You cannot read what you just wrote</a>
</h3>
All sub-statements in a data-modifying CTE see the same snapshot. This means the effects of one CTE are not visible</strong> to other CTEs or the main query when they read the target table</em>. Only the RETURNING</code> clause communicates data between CTE steps.</p>
SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span>
-------</span></span>
    31</span></span>
(1 row)</span></span></code></pre>WITH</span> ins </span>AS</span> (</span></span>
    INSERT INTO</span> orders (customer_id, amount, </span>status</span>, created_at)</span></span>
    VALUES</span> (</span>1</span>, </span>100</span>.</span>00</span>, </span>'pending'</span>, CURRENT_DATE)</span></span>
    RETURNING id</span></span>
)</span></span>
-- this does NOT see the row we just inserted</span></span>
SELECT</span> count</span>(</span>1</span>) </span>FROM</span> orders </span>WHERE</span> customer_id </span>=</span> 1</span>;</span></span></code></pre> count</span></span>
-------</span></span>
    31</span></span>
(1 row)</span></span></code></pre>
The count(1)</code> query sees the pre-insert snapshot. If you need the inserted data, you must use the RETURNING</code> clause from the ins</code> CTE, not re-read the table.</p>
Tuple shuffling</a>
</h3>
A common pattern is using writable CTEs to move rows between tables atomically:</p>
WITH</span> moved </span>AS</span> (</span></span>
    DELETE FROM</span> orders_staging</span></span>
    RETURNING </span>*</span></span>
)</span></span>
INSERT INTO</span> orders</span></span>
SELECT * FROM</span> moved;</span></span></code></pre>
This deletes all rows from the staging table and inserts them into the production table in a single atomic operation. No window where data exists in both or neither table.</p>
Data-modifying CTEs disable parallel query for the entire statement. If you have a complex query that mixes reads and writes, the write CTE prevents parallelism even for the read-only parts.</div>
Recursive CTEs are always materialized</a>
</h2>
Recursive CTEs use an iterative working-table mechanism. Despite the name, they aren't truly recursive. PostgreSQL doesn't "call itself" by creating a nested stack of unfinished queries. Instead, it operates in loops.</p>

Execute the non-recursive term (the "seed"). Put results in the working table.</li>
Execute the recursive term using the working table as input. New rows become the next</em> working table.</li>
Repeat until the recursive term returns no new rows.</li>
Return the union of all iterations.</li>
</ol>
WITH RECURSIVE</span> org_chart </span>AS</span> (</span></span>
    -- Seed: start from the CEO</span></span>
    SELECT</span> id, </span>name</span>, manager_id, </span>1</span> AS</span> depth</span></span>
    FROM</span> employees</span></span>
    WHERE</span> manager_id </span>IS NULL</span></span>
</span>
    UNION ALL</span></span>
</span>
    -- Recursive term: find direct reports</span></span>
    SELECT</span> e</span>.</span>id</span>, </span>e</span>.</span>name</span>, </span>e</span>.</span>manager_id</span>, </span>oc</span>.</span>depth</span> +</span> 1</span></span>
    FROM</span> employees e</span></span>
    JOIN</span> org_chart oc </span>ON</span> e</span>.</span>manager_id</span> =</span> oc</span>.</span>id</span></span>
)</span></span>
SELECT * FROM</span> org_chart </span>ORDER BY</span> depth, </span>name</span>;</span></span></code></pre> id |  name   | manager_id | depth</span></span>
----+---------+------------+-------</span></span>
  1 | Alice   |            |     1</span></span>
  2 | Bob     |          1 |     2</span></span>
  3 | Charlie |          1 |     2</span></span>
  4 | Diana   |          2 |     3</span></span>
  5 | Eve     |          2 |     3</span></span>
  6 | Frank   |          3 |     3</span></span>
  7 | Grace   |          3 |     3</span></span>
  8 | Hank    |          6 |     4</span></span>
  9 | Ivy     |          6 |     4</span></span>
(9 rows)</span></span></code></pre>UNION vs UNION ALL</a>
</h3>
The choice between UNION</code> and UNION ALL</code> in recursive CTEs matters more than in regular queries.</p>
UNION ALL</code> keeps all rows, including duplicates. This is faster but dangerous: if your graph has cycles, the recursion never terminates. PostgreSQL will run until you cancel the query or memory is exhausted.</p>
UNION</code> deduplicates at each iteration. This prevents infinite loops in cyclic graphs but adds the cost of hashing and comparing rows at every step.</p>
PostgreSQL 14 added SQL-standard SEARCH</code> and CYCLE</code> clauses that replace the manual patterns for controlling traversal order (breadth-first vs. depth-first) and detecting cycles. SEARCH BREADTH FIRST BY</code> / SEARCH DEPTH FIRST BY</code> control the order, while CYCLE</code> automatically detects and marks cycles, which is far cleaner than the old pattern of accumulating an array of visited IDs.</p>

For pure hierarchical data (trees without cycles), consider the ltree</code> extension as an alternative to recursive CTEs. It stores the full path as a label tree and supports efficient ancestor/descendant queries with GiST indexes. The trade-off is denormalized storage vs. on-the-fly recursion.</p>
</div>
The exotic edge cases</a>
</h2>
Partition pruning lost</a>
</h3>
When you materialize a CTE over a partitioned table, partition pruning cannot happen on the CTE scan side. The materialized result is a flat tuplestore disconnected from the partition metadata.</p>
-- assume orders is range-partitioned by created_at</span></span>
WITH</span> recent </span>AS</span> MATERIALIZED (</span></span>
    SELECT * FROM</span> orders</span></span>
)</span></span>
SELECT * FROM</span> recent </span>WHERE</span> created_at </span>></span> '2025-06-01'</span>;</span></span></code></pre>
The created_at > '2025-06-01'</code> predicate is applied after</em> materialization. All partitions are scanned to build the CTE, even though only one or two would be needed. Use NOT MATERIALIZED</code> (or simply let the planner inline it) to preserve partition pruning.</p>
Prepared statements and plan caching</a>
</h3>
PostgreSQL generates custom plans for the first 5 executions of a prepared statement. After that, it may switch to a generic plan. CTE inlining decisions can differ between custom and generic plans, because generic plans don't know the actual parameter values.</p>
This means a CTE that gets inlined during your first 5 calls might start materializing on the 6th or vice versa. If you see sudden plan changes with prepared statements, check whether the CTE inlining behaviour has shifted.</p>
work_mem spilling</a>
</h3>
Materialized CTEs store their results in memory, bounded by work_mem</code>. When the result set exceeds this limit, it silently spills to disk as a temporary file. This is not an error - it just gets slower.</p>
Monitor with log_temp_files = 0</code> (logs all temp files) or check EXPLAIN (ANALYZE, BUFFERS)</code> for temp read/write counts.</p>

PostgreSQL 18: EXPLAIN now shows memory/disk usage</strong>

Starting with PostgreSQL 18, EXPLAIN ANALYZE</code> reports memory and disk usage for Material nodes, including CTE materialization. You can see exactly how much memory a materialized CTE consumed and whether it spilled to disk.</p>
</div>
CTE and security barrier views</a>
</h3>
A `security_barrier` view is a "black box" that forces the database to fully resolve the view's internal logic before any outer filters are applied.</div>
Security-barrier views already prevent subquery flattening as a security measure (to stop user-defined functions from seeing rows they shouldn't). When you combine a security-barrier view with a CTE, you compound the optimisation barriers. The planner can neither inline the view nor the CTE. If performance matters in this scenario, consider materializing the security-sensitive filtering into a temporary table first.
CTE vs. subquery vs. temporary table</a>
</h2>
CTE, referenced once</strong> the planner inlines it on PG 12+, so it doesn't affect the execution plan. This is the default choice for breaking up complex queries.</p>
CTE, referenced multiple times (small result)</strong> represents acceptable cost. Materialization means the subquery runs once. For aggregations or filtered subsets that produce a few hundred rows, the overhead is minimal.</p>
As Henrietta Dombrovskaya</a> emphasizes, "The best temporary table is the one you didn't create". Always exhaust your indexing and query-rewriting options before reaching for a temp table, as the DDL overhead often outweighs the execution gains.</div>
**CTE, referenced multiple times (large result)** ss case when you might consider a temporary table instead. A materialized CTE has no indexes and no statistics. A temporary table can have both, and as covered in [Introduction to Buffers](/posts/introduction-to-buffers/), temporary tables use local buffers with simpler locking and no WAL overhead. If you're joining against 100k+ rows from a CTE, create a temp table, add an index, and `ANALYZE` it.
Data-modifying operations</strong> with writable CTE. No alternative gives you the atomic, single-statement behavior.</p>
Recursive CTEs</strong>. There's no alternative in pure SQL.</p>
Large intermediate results needing indexes/statistics</strong>. If you really need them use temporary table. As discussed in Reading Buffer statistics</a>, temporary tables offer full planner support - indexes, statistics, and buffer management - that materialized CTEs simply don't have.</p>
Scenario</th> Recommendation</th></tr></thead>

Readability, single reference</td> CTE (inlined, free)</td></tr>
Compute once, use many times (small)</td> CTE (materialized)</td></tr>
Compute once, use many times (large)</td> Temporary table</td></tr>
Atomic data modification</td> Writable CTE</td></tr>
Hierarchy / graph traversal</td> Recursive CTE</td></tr>
Need indexes on intermediate data</td> Temporary table</td></tr>
</tbody></table>
The PG 18 state of affairs</a>
</h2>
PostgreSQL 18 continues to refine CTE handling without any revolutionary changes:</p>

EXPLAIN shows memory/disk usage</strong> for CTE materialization nodes. You can finally see whether your CTE fit in work_mem</code> or spilled to disk.</li>
Better query plans for CTEs on the same table.</strong> The planner is smarter about eliminating redundant scans when multiple CTEs reference the same underlying table.</li>
Data-modifying CTE fix for updatable views with rules.</strong> An edge case where writable CTEs interacted incorrectly with views defined using rules has been resolved.</li>
CTE inlining is mature.</strong> The core inlining logic hasn't changed since PG 12. What has improved is everything around it - better statistics propagation (PG 17), better materialisation diagnostics (PG 18), better cost estimation.</li>
</ul>
CTEs are a good tool. Just know when you're holding the sharp end.</p>


The hidden cost of PostgreSQL arrays
2026-01-12T20:50:00+00:00
Starting with arrays in PostgreSQL is as simple as declaring a column as integer[]</code>, inserting some values, and you are done.</p>
Or building the array on the fly.</p>
SELECT</span> '{1,2,3}'</span>::</span>int</span>[];</span></span>
SELECT array</span>[1,2,3];</span></span></code></pre>  int4</span></span>
---------</span></span>
 {1,2,3}</span></span>
(1 row)</span></span>
</span>
  array</span></span>
---------</span></span>
 {1,2,3}</span></span>
(1 row)</span></span></code></pre>
The official documentation</a> provides a good introduction. But beneath this straightforward interface lies a set of more complex properties than most of us realise. Arrays in PostgreSQL are not just "lists" in a field. They have their own memory management strategy, their own index logic, and a lot of edge-case scenarios.</p>
As it goes with boringSQL</strong> deep-dives, this article will explore the corners of array functionality that might break your production.</p>
The document model temptation</a>
</h2>
Wait? Are we going to talk about JSONB arrays? Not at all. The whole concept of arrays in RDBMSs is actually document storage in disguise</strong>.</p>
In database design, locality ensures faster retrieval times by keeping related data close on physical storage.</span>Whether you use a distinct integer[]</code> type or a JSON list [1, 2, 3]</code>, you are making the exact same architectural decision: you are prioritising locality over normalisation</strong>.</p>
When you store tag_ids</code> in an array, you are embedding related data directly into a row - just like a NoSQL database might embed subdocuments. This is not inherently wrong. Document databases exist for good reasons: they eliminate joins, simplify reads, and map naturally to application objects.</p>
But PostgreSQL is a relational database. It was designed around the relational model, where:</p>

foreign keys</strong> enforce referential integrity</a></li>
joins</strong> connect normalised tables</li>
updates</strong> modify individual rows, not entire lists</li>
</ul>
Arrays give you document-model convenience, but you lose relational promises. There are no foreign keys and no ON DELETE referential_action</code> (like CASCADE) for array elements. If you delete a tags</code> entry, the orphaned ID will remain in your array forever.</p>
The rule of thumb</strong> is that if you find yourself in need of referential integrity - you most likely want a link table, not an array. Arrays are for data that shares the same lifecycle as the parent row. Not for relationships spanning across different tables.</p>
A practical example is the author of a blog post (one author might write multiple other posts), whereas a whitelist of IP addresses for a service account is only applicable to the given entity.</p>
With JSONB being so flexible, you might wonder why we still bother with quirkier native types. The answer lies in the 'boring' part of the database: predictability and efficiency. An integer[]</code> column guarantees that every element is an integer — similar to how enums</a> enforce a fixed set of allowed values.</p>
Arrays are also more storage-efficient for primitives because they don't carry the metadata overhead of JSON objects.</p>
The syntax gotchas</a>
</h2>
This post assumes basic knowledge of arrays. We won't cover the basics.</p>
Arrays don't have to start at 1</a>
</h3>
By default, SQL arrays start at 1. And seemingly, there is nothing wrong with iterating through them in a fashion similar to:</p>
FOR</span> i </span>IN</span> 1</span> .. array_length(fruits, </span>1</span>) </span>LOOP</span></span>
    RAISE NOTICE </span>'Index % contains: %'</span>, i, fruits[i];</span></span>
END LOOP</span>;</span></span></code></pre>
that is until you find an array with the arbitrary bounds. Which PostgreSQL allows.</p>
SELECT</span> '[-5:-3]={10,20,30}'</span>::</span>int</span>[];</span></span></code></pre>
To make sure you iterate correctly through any given array, always use array_lower()</code> and array_upper()</code> in PL/pgSQL</p>
SELECT</span> array_lower(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> array_lower</span></span>
-------------</span></span>
          -5</span></span>
(1 row)</span></span></code></pre>
or generate_subscripts()</code> in SQL.</p>
SELECT</span> generate_subscripts(</span>'[-5:-3]={10,20,30}'</span>::</span>int</span>[], </span>1</span>);</span></span></code></pre> generate_subscripts</span></span>
---------------------</span></span>
                  -5</span></span>
                  -4</span></span>
                  -3</span></span>
(3 rows)</span></span></code></pre>Missing dimensions</a>
</h3>
When creating a table, you might expect strict typing. That's true for everything — except the array dimensions. You might think integer[][]</code> enforces a 2D matrix. Except it does not. The []</code> syntax is effectively syntactic sugar. PostgreSQL does not enforce the number of dimensions of sub-arrays at the schema level at all by default.</p>
CREATE TABLE</span> dimension_test</span> (</span></span>
    matrix </span>integer</span>[][] </span></span>
);</span></span>
</span>
INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3,4}}'</span>);</span></span>
</span>
-- this is not going to fail</span></span>
INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{1,2,3}'</span>);</span></span>
</span>
-- 3D matrix works too</span></span>
INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{{1,2},{3,4}}, {{5,6},{7,8}}}'</span>);</span></span></code></pre>
If you want to enforce a specific array dimension, you cannot rely on the type definition. Instead, you must use a CHECK</code> constraint.</p>
CREATE TABLE</span> strict_matrix</span> (</span></span>
    -- dims: ensure it's 2-Dimensional</span></span>
    -- array_length: make sure it's exactly 3x3</span></span>
    board </span>integer</span>[]</span> CHECK</span> (array_ndims(board) </span>=</span> 2</span> AND</span> array_length(board, </span>1</span>) </span>=</span> 3</span>)</span></span>
);</span></span></code></pre>INSERT INTO strict_matrix VALUES (</span></span>
    ARRAY[</span></span>
        [0, 1, 0],</span></span>
        [1, 1, 0],</span></span>
        [0, 0, 1]</span></span>
    ]</span></span>
);</span></span></code></pre>
The only exception is that PostgreSQL enforces uniformity of arrays on every nesting level. This means it rejects sub-arrays with different sizes.</p>
INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span>
ERROR:  malformed </span>array</span> literal: </span>"{{1,2}, {3}}"</span></span>
LINE</span> 1</span>: </span>INSERT INTO</span> dimension_test </span>VALUES</span> (</span>'{{1,2}, {3}}'</span>);</span></span>
                                           ^</span></span>
DETAIL:  Multidimensional arrays must have sub</span>-</span>arrays </span>with</span> matching dimensions.</span></span></code></pre>Slicing arrays</a>
</h3>
When accessing array values, it's important to consider that the syntax [1]</code> and [1:1]</code> are different. While the first one is an accessor, the second one acts like a constructor.</p>
select</span> matrix[1][1]</span> from</span> dimension_test ;</span></span>
 matrix</span></span>
--------</span></span>
      1</span></span>
(</span>1</span> row</span>)</span></span></code></pre>
When slicing an array, even if the slice is a single-element, it will be returned as a single-element array, not a scalar value.</p>
select</span> matrix[1:1][1:1], matrix[1][1:1], matrix[1:1][1]</span> from</span> dimension_test ;</span></span>
 matrix | matrix | matrix</span></span>
--------+--------+--------</span></span>
 {{</span>1</span>}}  | {{</span>1</span>}}  | {{</span>1</span>}}</span></span>
(</span>1</span> row</span>)</span></span></code></pre>The ugly</a>
</h3>
Accessing array values has a forgiving behaviour, making it more difficult to find underlying bugs:</p>
-- out-of-bound access returns NULL</span></span>
SELECT</span> (</span>ARRAY</span>[1,2,3])[10];</span></span>
 array</span></span>
--------</span></span>
 (</span>null</span>)</span></span>
(</span>1</span> row</span>)</span></span></code></pre>-- out of bounds slicing returns an empty array</span></span>
SELECT</span> (</span>ARRAY</span>[1,2,3])[5:10];</span></span>
 array</span></span>
-------</span></span>
 {}</span></span>
(</span>1</span> row</span>)</span></span></code></pre>
But the single most confusing aspect that might trip you up coming from other programming languages is the fact that PostgreSQL treats multi-dimensional arrays as a single matrix, not an array of arrays.</p>
-- wrong dimensionality </span></span>
SELECT</span> (</span>ARRAY</span>[[1,2],[3,4]])[1];</span></span>
 array</span></span>
--------</span></span>
 (</span>null</span>)</span></span>
(</span>1</span> row</span>)  </span></span></code></pre>
While in other languages you expect {1,2}</code> as a result, PostgreSQL can't give it to you. It tries to return the first cell and fails (because the index is not complete).</p>
And to paraphrase Fletcher's "Double Bubble" analogy (which also went very wrong — extra points for getting the reference), you can't fix this by using slice notation.</p>
select</span> (</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1];</span></span>
  int4</span></span>
---------</span></span>
 {{</span>1</span>,</span>2</span>}}</span></span>
(</span>1</span> row</span>)</span></span></code></pre>
The only way to solve this puzzle is to unnest</code> the slice and re-aggregate the results.</p>
SELECT</span> array_agg(val) </span>FROM</span> unnest((</span>'{{1,2},{3,4}}'</span>::</span>int</span>[])[1:1]) val;</span></span>
 array_agg</span></span>
-----------</span></span>
 {</span>1</span>,</span>2</span>}</span></span>
(</span>1</span> row</span>)</span></span></code></pre>
Just be aware that `array_agg` does not guarantee the order of aggregated elements unless you use an `ORDER BY` clause. While it usually works with simple `unnest` queries, relying on implicit ordering can be risky.
</div>
The other alternative is to cast it to JSONB and back. Not pretty no matter how you look at it. If you need to work with complex multi-dimensional structures where each sub-array has independent meaning, just use JSONB. It will do exactly what you expect.</p>
Indexing arrays</a>
</h2>
While you can use a B-tree index on an array column, it won't help you unless you are looking for whole-array equality and sorting an array by dictionary rules — and even on regular columns, PostgreSQL can ignore your indexes</a> when it decides a sequential scan is cheaper.</p>
-- which is bigger? B is correct</span></span>
-- A: {1, 1000, 1000}</span></span>
-- B: {2, 0}</span></span></code></pre>
Rendering B-tree indexes useless for any real-world index operations.</p>
When working with arrays, you actually need GIN</strong> (Generalized Inverted Index). If a B-tree index is a phone book, GIN is an index at the back of the book. To query one or more elements, you need to find all possible locations and then intersect them to find the locations that match.</p>
CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags);</span></span></code></pre>
GIN indexes are designed for set operations, making presence the key feature, while ignoring order. Here are operators that GIN provides for arrays:</p>
Containment</strong> @></code> - matches rows that include ALL of the selected items.</p>
-- match ALL tags</span></span>
tags @</span>></span> '{urgent, bug}'</span></span></code></pre>
Overlap</strong> &&</code> - does the row include ANY of the selected items.</p>
-- match bug or feature</span></span>
tags && </span>'{bug, feature}'</span></span></code></pre>
There's also <@</code> and =</code> (equality), but they should be easy to understand.</p>
Two sides of ANY</code></a>
</h3>
This is where it gets interesting. While you might often hear (myself included) that you shouldn't use dynamic SQL for IN</code> lists and should use ANY</code> instead, there are some dangers you need to be aware of.</p>
The ANY</code> operator behaves very differently depending on which side of the comparison the array sits.</p>
The advice to use ANY</code> holds true when you are passing lists into the database</strong>. Instead of generating a query with 100 distinct parameters (WHERE id IN ($1, $2, ... $100)</code>), which bloats your query cache and forces hard-parses, you should pass a single array parameter.</p>
-- good: one parameter, one query plan</span></span>
SELECT * FROM</span> users </span>WHERE</span> id </span>=</span> ANY($</span>1</span>::</span>int</span>[]);</span></span></code></pre>
The trap is assuming this syntax is equally efficient when querying array columns. It is not. If you use ANY</code> to check if a value exists inside a table column, you are effectively asking the database to loop.</p>
-- bad: GIN does not support ANY; turning this into a seq scan</span></span>
SELECT * FROM</span> tickets </span>WHERE</span> 'feature'</span> =</span> ANY(tags);</span></span></code></pre>
When you write WHERE 'feature' = ANY(tags)</code>, you are not actually using an array operator. What you wrote is a scalar integer equality operator =</code> applied inside a loop construct. Since the scalar operator =</code> is not part of array_ops</code>, the planner assumes the index cannot help and falls back to a sequential scan.</p>
The correct way to rewrite the query is:</p>
-- good again</span></span>
SELECT * FROM</span> tickets </span>WHERE</span> tags @</span>> ARRAY</span>['feature'];</span></span></code></pre>Fast updates and the trade-off</a>
</h3>
Because a GIN index is built to work with sets, it is expensive to maintain. With a B-tree index, one row equals one index entry. In a GIN index, one row equals N index entries, where N is the number of elements in your array.</p>
This leads to write multiplication. To prevent this, PostgreSQL defaults to using a "fast update" mechanism. This is a strategy where new entries are added to a pending list (an unsorted temporary buffer) and only merged into the main index structure later (during VACUUM).</p>
Given the unsorted nature of a GIN index, it is an exception to VACUUM is a Lie</a>, as VACUUM actually does perform structural maintenance here.</span>While this makes INSERT operations manageable, it can slow down SELECTs. Every time you query the index, PostgreSQL must scan the organised main index plus the entire messy pending list. If that list grows large, your query performance might degrade.</p>
If you operate a read-heavy workflow with infrequent writes, you should disable this to guarantee consistent reading performance.</p>
CREATE INDEX</span> posts_by_tags</span> ON</span> posts </span>USING</span> GIN (tags) </span>WITH</span> (fastupdate </span>= off</span>);</span></span></code></pre>Storage and modification</a>
</h2>
Now it's time to return to the document model</a>. In PostgreSQL, rows are immutable (MVCC); there is no such thing as an "in-place update", and arrays are stored as atomic values. The consequence: to modify a single element of an array, PostgreSQL must copy and rewrite the entire row</strong>.</p>
UPDATE</span> user_activity</span></span>
SET</span> event_ids </span>=</span> event_ids </span>||</span> 10001</span></span>
WHERE</span> user_id </span>=</span> 50</span>;</span></span></code></pre>
Every single append rewrites the entire array, which in effect results in a rewrite of the entire row.</p>
TOASTed</a>
</h3>
When any array grows large enough (> 2 KB — see below), PostgreSQL moves it automatically to a separate storage area using TOAST</a>. While this keeps the data row lean, it turns array updates into a severe performance bottleneck.</p>
The difference this introduces is subtle but comes with a big impact. While a standard MVCC update simply copies the row version on the main heap, updating a TOASTed array forces PostgreSQL to fetch all external chunks, decompress the entire object into memory, apply the change, and then recompress and write the new full-size blob back to the TOAST table. This turns a simple modification into a CPU and I/O-intensive operation that rewrites the entire dataset rather than just the delta.</p>
The threshold is derived from TOAST_TUPLES_PER_PAGE</code> (default: 4), ensuring 4 tuples fit on a page.</br></br>Threshold: ~2 KB</span>Where does the 2 KB value come from? The TOAST threshold is calculated to ensure at least four tuples can fit on a single 8 KB heap page</a>. PostgreSQL uses this to balance efficiency against the overhead of TOAST indirection.</p>
The compressions</a>
</h3>
Before version 14, PostgreSQL relied on pglz</code> — an algorithm prioritising compression ratio over speed. This made the "decompress-modify-compress" cycle of TOAST painful.</p>
PostgreSQL 14 introduced LZ4 as an alternative:</p>
ALTER TABLE</span> articles </span>ALTER</span> COLUMN tags </span>SET COMPRESSION</span> lz4;</span></span></code></pre>
LZ4 is significantly faster for both compression and decompression, with only slightly lower compression ratios. If you are working with large arrays, switching to LZ4 is one of the easiest ways to reduce the CPU penalty of TOAST.</p>
When a large array might make sense</a>
</h3>
You might have gained the impression that arrays are bad. When evaluating the use of arrays, the real question isn't how big the array is</strong> but rather how often do you modify it</strong>? An array of 10,000 elements that you write once and is read-only for the rest of its lifecycle is a completely valid use case. An array of 50 elements that you append to on every incoming request is the real villain here.</p>
If you combine this with compression, you might get an interesting mix.</p>
DROP TABLE IF EXISTS</span> compression_test;</span></span>
CREATE TABLE</span> compression_test</span> (</span></span>
    id </span>serial PRIMARY KEY</span>,</span></span>
    compressed_floats float4[], </span></span>
    raw_floats float4[]</span></span>
);</span></span>
</span>
-- do not compress raw_floats</span></span>
ALTER TABLE</span> compression_test </span>ALTER</span> COLUMN raw_floats </span>SET</span> STORAGE </span>EXTERNAL</span>;</span></span>
</span>
-- insert semi-random data with low cardinality</span></span>
INSERT INTO</span> compression_test (compressed_floats, raw_floats)</span></span>
SELECT</span> semi_random_arr, semi_random_arr</span></span>
FROM</span> (</span></span>
    SELECT ARRAY</span>(</span></span>
        SELECT</span> floor</span>(random()</span> *</span> 50</span>)::float4 </span></span>
        FROM</span> generate_series</span>(</span>1</span>, </span>10000</span>)</span></span>
    ) </span>as</span> semi_random_arr</span></span>
) </span>as</span> generator;</span></span>
</span>
SELECT</span> </span></span>
    'Compressed (EXTENDED)'</span> as</span> strategy,</span></span>
    pg_size_pretty(pg_column_size(compressed_floats)::</span>bigint</span>) </span>as</span> size_on_disk</span></span>
FROM</span> compression_test</span></span>
UNION ALL</span></span>
SELECT</span> </span></span>
    'Raw (EXTERNAL)'</span>,</span></span>
    pg_size_pretty(pg_column_size(raw_floats)::</span>bigint</span>)</span></span>
FROM</span> compression_test;</span></span></code></pre>       strategy        | size_on_disk</span></span>
-----------------------+--------------</span></span>
 Compressed (EXTENDED) | 15 kB</span></span>
 Raw (EXTERNAL)        | 39 kB</span></span>
(2 rows)  </span></span></code></pre>Bulk loading with arrays</a>
</h2>
Up until now, it might seem that arrays don't actually bring many benefits. While they can have rough edges around storage, they are incredibly useful for transport.</p>
For inserting 5,000 rows, unnest</code> beats both an application-side loop and a massive VALUES (...), (...)</code> string.</p>
INSERT INTO</span> measurements (sensor_id, </span>value</span>, captured_at)</span></span>
SELECT * FROM</span> unnest(</span></span>
    $</span>1</span>::</span>int</span>[],        </span>-- array of sensor IDs</span></span>
    $</span>2</span>::</span>float</span>[],      </span>-- array of values</span></span>
    $</span>3</span>::</span>timestamptz</span>[]</span> -- array of timestamps</span></span>
);</span></span></code></pre>
All that is needed is one network round trip, and one query to parse and plan. PostgreSQL handles the arrays row by row internally for you. This works both for UPSERTs and MERGE</a>.</p>
The cases for special arrays</a>
</h2>
Standard PostgreSQL arrays are polymorphic types (anyarray</code>). This provides a powerful feature that allows a single function definition to operate on many different data types. They have to handle integers, strings, timestamps, and custom types equally well. But if you have specific data types, you can unlock significant performance gains by using specialised extensions.</p>
The intarray</code> extension</a>
</h3>
If you are dealing exclusively with 4-byte integers (int4</code>/integer</code>), the built-in array operations are leaving performance on the table. The intarray</a> extension provides specialised functions and index operators that are significantly faster than the generic implementation.</p>
To use it, you must explicitly enable it:</p>
CREATE</span> EXTENSION </span>IF NOT EXISTS</span> intarray;</span></span></code></pre>
The difference in developer ergonomics is immediate. To sort an array in standard SQL, you are forced to unnest, order, and re-aggregate. With intarray, you get native functions like sort() and uniq().</p>
-- standard arrays</span></span>
SELECT</span> array_agg(val </span>ORDER BY</span> val)</span></span>
FROM</span> unnest(</span>'{3, 1, 2}'</span>::</span>int</span>[]) val;</span></span>
</span>
-- intarray</span></span>
SELECT</span> sort(</span>'{3, 1, 2}'</span>::</span>int</span>[]);</span></span></code></pre>
Beyond raw management functions, it introduces a specialised query syntax that simplifies complex boolean logic. Instead of chaining multiple overlap (&&</code>) and containment (@></code>) checks, you can express your requirements in a single "query string" using the @@</code> operator.</p>
-- standard arrays</span></span>
SELECT * FROM</span> staff</span></span>
WHERE</span> available_days @</span>></span> '{1}'</span>          -- must include Mon</span></span>
  AND</span> available_days && </span>'{6, 7}'</span>       -- must include Sat OR Sun</span></span>
  AND NOT</span> (available_days @</span>></span> '{2}'</span>);   </span>-- must NOT include Tue</span></span>
</span>
-- intarray</span></span>
SELECT * FROM</span> staff</span></span>
WHERE</span> available_days @@ </span>'1 & (6 | 7) & !2'</span>;</span></span></code></pre>
The only catch is the type restriction. intarray</code> is strictly limited to signed 32-bit integers. If your values exceed 2 billion, you are back where you started.</p>
AI with pgvector</code></a>
</h3>
You cannot talk about arrays in 2026 without mentioning pgvector</a>. While it markets itself as a "vector store", internally it is simply an array of floats with a different mathematical focus.</p>
Standard arrays are binary</strong>: they care about Exact Matches (Overlap &&</code>, Containment @></code>). Vectors are all about fuzzy distance</strong> (Cosine <=></code>, Euclidean <-></code>).</p>
If you are building search or recommendation features, pgvector allows you to treat your array column not as a list of "facts" (tag A, tag B), but as coordinates in semantic space.</p>
Nevertheless, the architectural decision is exactly the same as using a standard array: you are trading strict structure for convenience. Since there is no way to "join" two rows based on how similar they are, you store the vector directly on the row. You accept a larger table size in exchange for the ability to ask, "What is close to this?".</p>
PS: Thanks to Matthias Feist for inspiring this article.</p>


RegreSQL: Regression Testing for PostgreSQL Queries
2025-11-13T22:47:00+00:00
TL;DR</strong> - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production.</em></p>
As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification.</p>
For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL</a>, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL</em></a> book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries.</p>
I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level.</p>
Introduction</a>
</h2>
The RegreSQL</a></strong> promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc</a> (for Go), PG'OCaml</a> or Rust's SQLx</a> toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later).</p>
But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere?</p>
This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations.</p>
The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management.</p>
Basic regression testing</a>
</h2>
Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like</p>
$ regresql text</span></span>
Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓</span></span>
</span>
Running regression tests...</span></span>
</span>
✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span>
✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span>
</span>
✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span>
✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span>
</span>
✓ artist_top-artists-by-album.1.json (0.00s)</span></span>
</span>
✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span>
✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span>
</span>
✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span>
</span>
Results: 8 passed, 0 failed, 8 skipped (0.00s)</span></span></code></pre>
In this example based on Chinook database</a> (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project</code>) for *.sql</code> files and attempts to run all queries against the configured PostgreSQL connection.</p>
The individual files can contain either single or multiple sql queries. Like following example</p>
-- name: top-artists-by-album</span></span>
-- Get the list of the N artists with the most albums</span></span>
SELECT</span></span>
    artist</span>.</span>name</span>,</span></span>
    count</span>(</span>*</span>) </span>AS</span> albums</span></span>
FROM</span></span>
    artist</span></span>
    LEFT JOIN</span> album </span>USING</span> (artist_id)</span></span>
GROUP BY</span></span>
    artist</span>.</span>name</span></span>
ORDER BY</span></span>
    albums </span>DESC</span></span>
LIMIT</span> :n;</span></span></code></pre>
The syntax for the queries supports both positional arguments (like $1</code> known from libpq library) or (preferred) psql</code> style variable (:varname</code>). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now).</p>
This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test</code> and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines.</p>
Performance regression testing</a>
</h2>
Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored</a> entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows</a>. This is where it hurts.</p>
RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated</p>
$ regresql baseline</span></span>
Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span>
Creating baselines directory: regresql/baselines</span></span>
Creating directory 'regresql/baselines'</span></span>
</span>
Creating baselines for queries:</span></span>
</span>
  ./</span></span>
  Created baseline: album-by-artist_list-albums-by-artist.1.json</span></span>
  Created baseline: album-by-artist_list-albums-by-artist.2.json</span></span>
  Created baseline: album-tracks_list-tracks-by-albumid.1.json</span></span>
  Created baseline: album-tracks_list-tracks-by-albumid.2.json</span></span>
  Created baseline: artist_top-artists-by-album.1.json</span></span>
  Created baseline: genre-topn_genre-top-n.top-1.json</span></span>
  Created baseline: genre-topn_genre-top-n.top-3.json</span></span>
  Created baseline: genre-tracks_tracks-by-genre.json</span></span>
</span>
Baselines have been created successfully!</span></span>
Baseline files are stored in: regresql/baselines</span></span></code></pre>
the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans</a>. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place.</p>
To demonstrate this, let's review the test output with the baselines.</p>
Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓</span></span>
</span>
Running regression tests...</span></span>
</span>
✓ album-by-artist_list-albums-by-artist.1.json (0.00s)</span></span>
✓ album-by-artist_list-albums-by-artist.2.json (0.00s)</span></span>
✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span>
  ⚠️  Sequential scan detected on table 'artist'</span></span>
    Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span>
  ⚠️  Nested loop join with sequential scan detected</span></span>
    Suggestion: Add index on join column to avoid repeated sequential scans</span></span>
✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s)</span></span>
  ⚠️  Sequential scan detected on table 'artist'</span></span>
    Suggestion: Consider adding an index if this table is large or this query is frequently executed</span></span>
  ⚠️  Nested loop join with sequential scan detected</span></span>
    Suggestion: Add index on join column to avoid repeated sequential scans</span></span>
</span>
✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)</span></span>
✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)</span></span>
✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span>
✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s)</span></span>
</span>
✓ artist_top-artists-by-album.1.json (0.00s)</span></span>
✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s)</span></span>
  ⚠️  Multiple sequential scans detected on tables: album, artist</span></span>
    Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span>
</span>
✓ genre-topn_genre-top-n.top-1.json (0.00s)</span></span>
✓ genre-topn_genre-top-n.top-3.json (0.00s)</span></span>
✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span>
  ⚠️  Multiple sequential scans detected on tables: genre, artist</span></span>
    Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span>
  ⚠️  Multiple sort operations detected (2 sorts)</span></span>
    Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span>
  ⚠️  Nested loop join with sequential scan detected</span></span>
    Suggestion: Add index on join column to avoid repeated sequential scans</span></span>
✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s)</span></span>
  ⚠️  Multiple sequential scans detected on tables: artist, genre</span></span>
    Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span>
  ⚠️  Multiple sort operations detected (2 sorts)</span></span>
    Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting</span></span>
  ⚠️  Nested loop join with sequential scan detected</span></span>
    Suggestion: Add index on join column to avoid repeated sequential scans</span></span>
</span>
✓ genre-tracks_tracks-by-genre.json (0.00s)</span></span>
✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s)</span></span>
  ⚠️  Multiple sequential scans detected on tables: genre, track</span></span>
    Suggestion: Review query and consider adding indexes on filtered/joined columns</span></span>
</span>
Results: 16 passed (0.00s)</span></span></code></pre>
As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready".</p>
In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below.</p>
-- name: query_name</span></span>
-- metadata: key1=value1, key2=value2</span></span>
SELECT</span> ...;</span></span></code></pre>
At this point RegreSQL recognizes</p>

notest</code> to skip the query testing altogether (not just cost tracking)</li>
nobaseline</code> to skip cost tracking</li>
noseqscanwarn</code> to keep cost tracking but disable sequential scan warnings</li>
and difffloattolerance</code> to cost failure threshold (default 10% at the moment).</li>
</ul>
-- name: query_name</span></span>
-- regresql: notest, nobaseline</span></span>
-- regresql: noseqscanwarn</span></span>
-- regresql: difffloattolerance:0.25</span></span>
-- query that can vary in cost by 20% without being considered a failure</span></span>
SELECT</span> ...;</span></span></code></pre>ORM enters the room</a>
</h2>
ORMs abstract away SQL, but they still generate it - much like view inlining</a> where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data:</p>
orders</span> =</span> (</span></span>
    session.query(Order)</span></span>
    .filter(Order.user_id</span> ==</span> user_id)</span></span>
    .options(</span></span>
        joinedload(Order.user),</span></span>
        joinedload(Order.shipping_address),</span></span>
        selectinload(Order.items)</span>  # NEW: Load order items</span></span>
    )</span></span>
    .all()</span></span>
)</span></span></code></pre>
That innocent selectinload(Order.items)</code> generates a separate query - and without an index on order_items.order_id</code>, it performs a sequential scan.</p>
RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system:</p>
@event.listens_for</span>(engine,</span> "before_cursor_execute"</span>)</span></span>
def</span> capture_sql</span>(conn, cursor, statement,</span> *</span>args):</span></span>
    captured_queries.append(statement)</span></span></code></pre>
Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development.</p>
Test Data Management</a>
</h2>
Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through.  Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance  baselines drift. Your tests become flaky.</p>
Traditional approach to create test data might involve</p>

Database dumps</strong> become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?</li>
SQL scripts</strong> might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.</li>
Factories in application code</strong> might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?</li>
Shared test database</strong> is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.</li>
</ul>
What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000).</p>
This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale.</p>
RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/</code>. Here's a simple example</p>
  fixture</span>:</span> basic_users</span></span>
  description</span>:</span> a handful of test users</span></span>
  cleanup</span>:</span> rollback</span></span>
</span>
  data</span>:</span></span>
    -</span> table</span>:</span> users</span></span>
      rows</span>:</span></span>
        -</span> id</span>:</span> 1</span></span>
          email</span>:</span> alice@example.com</span></span>
          name</span>:</span> Alice Anderson</span></span>
          created_at</span>:</span> 2024-01-15</span></span>
        -</span> id</span>:</span> 2</span></span>
          email</span>:</span> bob@example.com</span></span>
          name</span>:</span> Bob Builder</span></span>
          created_at</span>:</span> 2024-02-20</span></span></code></pre>
To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml</code>) you can just reference the fixture</p>
  fixtures</span>:</span></span>
    -</span> basic_users</span></span>
</span>
  "1"</span>:</span></span>
    email</span>:</span> alice@example.com</span></span>
</span>
  "2"</span>:</span></span>
    email</span>:</span> bob@example.com</span></span></code></pre>
And when you run regresql test</code>, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators</strong> including</p>

sequences, random integer, decimal, string, uuid, email and name generators</li>
date_between for generating random timestamps within a range</li>
foreign key references to be able to reuse data from other table's fixtures</li>
range to select value from predefined sources</li>
Go template support</li>
</ul>
 fixture</span>:</span> realistic_orders</span></span>
  generate</span>:</span></span>
    -</span> table</span>:</span> customers</span></span>
      count</span>:</span> 1000</span></span>
      columns</span>:</span></span>
        id</span>:</span></span>
          generator</span>:</span> sequence</span></span>
          start</span>:</span> 1</span></span>
        email</span>:</span></span>
          generator</span>:</span> email</span></span>
          domain</span>:</span> shop.example.com</span></span>
        name</span>:</span></span>
          generator</span>:</span> name</span></span>
          type</span>:</span> full</span></span>
        created_at</span>:</span></span>
          generator</span>:</span> date_between</span></span>
          start</span>:</span> "2023-01-01"</span></span>
          end</span>:</span> "2024-12-31"</span></span>
</span>
    -</span> table</span>:</span> orders</span></span>
      count</span>:</span> 5000</span></span>
      columns</span>:</span></span>
        id</span>:</span></span>
          generator</span>:</span> sequence</span></span>
          start</span>:</span> 1</span></span>
        customer_id</span>:</span></span>
          generator</span>:</span> int</span></span>
          min</span>:</span> 1</span></span>
          max</span>:</span> 1000</span></span>
        amount</span>:</span></span>
          generator</span>:</span> decimal</span></span>
          min</span>:</span> 10.00</span></span>
          max</span>:</span> 999.99</span></span>
          precision</span>:</span> 2</span></span>
        order_date</span>:</span></span>
          generator</span>:</span> date_between</span></span>
          start</span>:</span> "2023-01-01"</span></span>
          end</span>:</span> "2024-12-31"</span></span></code></pre>
This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like.</p>
The fixtures are also stackable</strong> and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse.</p>
  fixture</span>:</span> orders_with_shipping</span></span>
  depends_on</span>:</span></span>
    -</span> basic_users</span></span>
</span>
  data</span>:</span></span>
    -</span> table</span>:</span> orders</span></span>
      rows</span>:</span></span>
        -</span> id</span>:</span> 101</span></span>
          user_id</span>:</span> 1</span>  # References Alice from basic_users</span></span>
          total</span>:</span> 99.99</span></span>
          status</span>:</span> shipped</span></span></code></pre>
Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation.</p>
  fixture</span>:</span> mixed_setup</span></span>
  description</span>:</span> Combine SQL with YAML and generated data</span></span>
  cleanup</span>:</span> rollback</span></span>
</span>
  # SQL executes first (either as file or inline)</span></span>
  sql</span>:</span></span>
    -</span> file</span>:</span> sql/setup_schema.sql</span></span>
    -</span> inline</span>:</span> "INSERT INTO config (key, value) VALUES ('version', '1.0');"</span></span>
</span>
  # followed YAML data</span></span>
  data</span>:</span></span>
    -</span> table</span>:</span> users</span></span>
      rows</span>:</span></span>
        -</span> id</span>:</span> 1</span></span>
          email</span>:</span> admin@example.com</span></span>
</span>
  # and finally generated data</span></span>
  generate</span>:</span></span>
    -</span> table</span>:</span> orders</span></span>
      count</span>:</span> 100</span></span>
      columns</span>:</span></span>
        id</span>:</span></span>
          generator</span>:</span> sequence</span></span>
          start</span>:</span> 1</span></span>
        user_id</span>:</span></span>
          generator</span>:</span> int</span></span>
          min</span>:</span> 1</span></span>
          max</span>:</span> 1</span></span></code></pre>
RegreSQL provides commands to inspect and validate your fixtures</p>
  # List all available fixtures</span></span>
  regresql</span> fixtures list</span></span>
</span>
  # Show fixture details and dependencies</span></span>
  regresql</span> fixtures show realistic_orders</span></span>
</span>
  # Validate fixture definitions</span></span>
  regresql</span> fixtures validate</span></span>
</span>
  # Show dependency graph</span></span>
  regresql</span> fixtures deps</span></span>
</span>
  # Apply fixture manually (for debugging)</span></span>
  regresql</span> fixtures apply basic_users</span></span></code></pre>
The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves.</p>
(EDIT 2025-11-20)</strong> The dynamically generated fixtures on each tests will break the correctness testing of
RegreSQL. You have to use fixtures according to your use</p>

Fixed generated fixtures re-used between tests</li>
Use RegreSQL to test fixtures before/after migration</li>
</ul>
I'm working on next release of RegreSQL to address DX of fixtures usage.</p>
RegreSQL future</a>
</h2>
Introducing a new open source project is an ambitious goal, and RegreSQL</a> is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL</strong> brand, where it's vital component (together with pgTap) for building SQL Labs</a> which (as I sincerely hope) will provide a foundation for its further development.</p>
At the same time RegreSQL</strong> is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable.</p>
RegreSQL is available at boringsql.com/products/regresql</a> - feel free to open issue, or drop me email about the project at radim@boringsql.com</a> or connect on LinkedIn</a>.</p>


Beyond Start and End: PostgreSQL Range Types
2025-11-02T21:40:00+00:00
One of the most read articles at boringSQL is Time to Better Know The Time in PostgreSQL</a> where we dived into the complexities of storing and handling time operations in PostgreSQL. While the article introduced the range data types, there's so much more to them. And not only for handling time ranges. In this article we will cover why to consider range types and how to work with them.</p>
Bug Not Invented Here</a>
</h2>
But before we can talk about the range types, let's try to understand why we should look at them in the first place. Let's imagine a booking platform for large flash sales of the seats, that goes live at 10pm and will be taken by storm by thousands of people who want to get their tickets.</p>
CREATE TABLE</span> seat_holds</span> (</span></span>
    hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
</span>
    seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span>
    user_session_id UUID </span>NOT NULL</span>,</span></span>
    -- define the hold period explicitly</span></span>
    hold_started_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP,</span></span>
    hold_expires_at </span>TIMESTAMPTZ NOT NULL</span>,</span></span>
    created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span>
);</span></span>
</span>
CREATE INDEX</span> seat_holds_on_seat_id</span> ON</span> seat_holds(seat_id);</span></span>
CREATE INDEX</span> seat_holds_on_expiration</span> ON</span> seat_holds(hold_expires_at);</span></span></code></pre>
While the table design looks perfectly reasonable, it has one serious flaw</strong> - there's no database-level atomicity guarantee there to prevent two holds for the same seat_id</code> at the same time. The table design requires on application logic to check for the existing holds before inserting a new hold, and at the same time it does not provide any high-concurrency guarantee.</p>
If all you have in your toolbelt are those two columns you will end up increasing the complexity to make it work. You started with one problem and soon your application developers might want to ask you to add caching layer (most likely external K/V store) to place the holds there and very soon you have N-problems when you will resort to building a complex, custom application-side locking mechanism that is bug-prone and difficult to maintain.</p>
Other possibility is to bring all the operations on seat holds into more complex transaction management. Which is literally invitation for disaster in extreme contention</strong> situation like flash ticket sales. No matter which blocking strategy you use SERIALIZABLE</code> transaction isoliation or pessimistic locking using SELECT ... FOR UPDATE</code> will create a large overhead in application logic (retries, massive contention on database resources, etc.).</p>
And as we are going to talk about range data types, there's a first possible option to solve the problem directly on database level.</p>
-- needed to add GiST support to equality of integers (for seat_id)</span></span>
CREATE</span> EXTENSION </span>IF NOT EXISTS</span> btree_gist;</span></span>
</span>
ALTER TABLE</span> seat_holds </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span>
    EXCLUDE </span>USING</span> gist (</span></span>
        seat_id </span>WITH =</span>,</span></span>
        tsrange(hold_started_at, hold_expires_at) </span>WITH</span> &&</span></span>
    );</span></span></code></pre>
Which will add the constraint directly on the table level and enable atomic conflict detection for your seat holds with minimum locking overhead. This guarantees that the database will never allow two overlapping holds to exist for the same seat, irrespective of how many concurrent users are attempting to book. The real win here is data integrity - the database now rejects the overlap outright instead of merely making it rare. While you'll still need retry logic in your application when conflicts occur, you've moved the correctness guarantee from application code (where bugs hide) into the database schema (where it's enforced). The GiST (Generalized Search Tree) index is the crucial component which makes checking for overlapping time ranges effective even under extreme load.</p>
But really, if you look at the proposed fix, it's still a workaround - we're converting two separate TIMESTAMPTZ</code> columns into a range type on the fly, when range types already include native GiST support out of the box.</p>
Introducing the Data Range Types</a>
</h2>
You’ve seen the power of the EXCLUDE</code> constraint to solve the concurrency problem, but why to settle for workaround (unless it's temporary as part of the bigger refactoring) instead of going all the way in?</p>
This brings us to the core of the matter: PostgreSQL's native Range Types</strong>.</p>
PostgreSQL provides a set of built-in range types, all following the pattern of type and range:</p>

int4range</code> for integer</li>
int8range</code> for bigint</li>
numrange</code> for numeric</li>
tsrange</code> for timestamp without time zone</li>
tstzrange</code> for timestamp with time zone (which we briefly saw above)</li>
daterange</code> for date</li>
</ul>
And it does not stop there. You can easily define your own custom range types</strong> over any basic data type.</p>
First win: cleaner schema</a>
</h2>
When using start</code> and end</code> columns you are not explicitely telling the database that these two columns are single concept representing time span. The logic to work with those two columns resides only in your queries and application code.</p>
When you refactor our sample table to embrace the native range type, it becomes more expressive and inherently correct.  The application code no longer needs to manage two separate boundaries.</p>
CREATE TABLE</span> seat_holds_native</span> (</span></span>
    hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    seat_id </span>INTEGER NOT NULL REFERENCES</span> seats(id),</span></span>
    user_session_id UUID </span>NOT NULL</span>,</span></span>
    hold_period TSTZRANGE </span>NOT NULL</span>,</span></span>
    created_at </span>TIMESTAMPTZ NOT NULL DEFAULT</span> CURRENT_TIMESTAMP</span></span>
);</span></span></code></pre>
This is the power of a first-class data type</strong>. We've shifted the burden from the application logic to the database schema, making the table definition itself communicate its intent more clearly.</p>
Second win: Atomicity guaranteed</a>
</h2>
While the new schema is cleaner, the real database win comes from enforcing our concurrency guarantee - preventing two seats from being double-booked. To achieve this you can reuse the exclusion constraint</strong> as demonstrated previously.</p>
ALTER TABLE</span> seat_holds_native </span>ADD CONSTRAINT</span> seat_holds_no_overlap</span></span>
    EXCLUDE </span>USING</span> gist (</span></span>
        seat_id </span>WITH =</span>,</span></span>
        hold_period </span>WITH</span> &&</span></span>
    );</span></span></code></pre>
This time you can use hold_period</code> directly without need to explicitely convert it. This constraint enforces two rules at once:</p>

seat_id WITH =</code> ensures the constraint only applies to holds for the same seat.</li>
hold_period WITH &&</code> checking the overlap of hold periods with the operator &&</code></li>
</ol>
Finally EXCLUDE USING gist</code>  is the crucial technical detail, telling PostgreSQL to use GiST index to enforce the constraint.  This is not specific to range types, as EXCLUDE</code> constraint can't exist without an index to enforce it (common use cases might include arrays, geometric data, etc.).</p>
Range boundaries: A quick math refresher</a>
</h2>
Before we dive into the operators, let's take a moment to understand how PostgreSQL represents range boundaries. If you remember your high school math, range types use the same notation as intervals in mathematics.</p>
PostgreSQL ranges can have four different boundary types:</p>
Inclusive boundaries -</strong> [</code> and ]</code> An inclusive boundary includes the endpoint value in the range. Think of it as "less than or equal to" or "greater than or equal to".</p>
-- [10, 20] includes both 10 and 20</span></span>
-- represents 10 ≤ x ≤ 20</span></span>
SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>);</span></span></code></pre>
Exclusive boundaries -</strong> (</code> and )</code> An exclusive boundary excludes the endpoint value from the range. This is "less than" or "greater than" without the equality.</p>
-- (10, 20) excludes both 10 and 20</span></span>
-- represents 10 < x < 20</span></span>
SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'()'</span>);</span></span>
</span></code></pre>
Mixed boundaries -</strong> [)</code> or (]</code> You can mix and match. The most common and default pattern</strong> in PostgreSQL is [)</code> - inclusive lower bound, exclusive upper bound. This is particularly useful for timestamps and dates because it naturally represents "from the start of one period up to (but not including) the start of the next".</p>
As mentioned, the default boundary [)</code> eliminates the natural ambiguity when representing consecutive periods.</p>
-- these ranges are adjacent, not overlapping</span></span>
-- Week 1</span></span>
SELECT</span> '[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span>
-- Week 2</span></span>
SELECT</span> '[2025-11-08, 2025-11-15)'</span>::tstzrange;</span></span></code></pre>
With this notation, the end of one period is exactly the start of the next, with no gaps or overlaps. This makes it perfect for time-based ranges, inventory availability windows, or any scenario where you're dividing a continuum into distinct segments.</p>
Canonicalization and Range Set Operations</a>
</h2>
Boundaries are not the only aspect of the ranges that behave like mathematical sets, allowing arithmetic operations and canonicalization of the discrete ranges.</p>
For discrete</strong> range types (int4range, int8range, daterange), multiple representations can actually mean the exact same set of values. For example, for integers, the range [10, 20] (inclusive on both ends) is the same set as (9, 21) (exclusive on both ends) or the default PostgreSQL canonical form [10, 21) (inclusive lower, exclusive upper).</p>
PostgreSQL uses a canonicalization function</strong> to convert all equivalent discrete ranges into a single, uniform representation (default [)</code> boundary mentioned above), which is essential for accurate equality checks and indexing.</p>
-- [10, 20] includes integers 10, 11, ..., 20.</span></span>
SELECT</span> int4range(</span>10</span>, </span>20</span>, </span>'[]'</span>) </span>AS</span> original_range;</span></span>
-- Result: [10,21)</span></span>
</span>
-- (9, 21) includes integers 10, 11, ..., 20.</span></span>
SELECT</span> int4range(</span>9</span>, </span>21</span>, </span>'()'</span>) </span>AS</span> original_range;</span></span>
-- Result: [10,21)</span></span></code></pre>
Exception are continuous ranges</strong> (think floats and timestamps with fractional seconds) where PostgreSQL won't use canonicalization because a boundary change always means a change in the contained values as there's no easy to define "next value". I.e. there's no next value for 20.0 (i.e. not 20.0001, nor 20.000001, etc.) and changing boundary would change it's meaning.</p>
The operator toolkit</a>
</h2>
The type ranges and by their definition GiST (in this instance range_ops) and GIN (array_ops) indexes come with number of operator that makes your life easier.</p>
Overlap operator - &&</code></strong>
As mentioned already above the overlap operator is the most fundamental one. It simply checks whether two ranges share any common data points.</p>
-- find holds active at any point between 10:00 and 11:00</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period && </span>'[2025-12-25 10:00, 2025-12-25 11:00)'</span>::tstzrange;</span></span></code></pre>
Contains operator - @></code></strong>
For checking against specific moments in time, we might turn to the Contains operator. It verifies whatever the range on the left completely containts the element on the right (which might be both underlying data type or range type).</p>
-- find holds that are active at the specific momement</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '2025-11-05 15:00'</span>::</span>timestamptz</span>;</span></span>
</span>
-- find holds that are active at the specific time range</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period @</span>></span> '[2025-12-25 10:00, 2025-12-25 10:15)'</span>::tstzrange;</span></span></code></pre>
Contained By operator - <@</code></strong>
In contrast, the Contained By operator checks the reverse relationship - whatever the range on the left is entirely contained by the range on the right.</p>
-- find holds that are within November '25</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span><</span>@ </span>'[2025-11-01, 2025-12-01)'</span>::tstzrange;</span></span></code></pre>
Strictly Before/After operators - <<</code> and >></code></strong> operators allow you to query ranges that are completely separated from the reference range (i.e. don't even touch the boundaries).</p>
-- find holds that finished strictly before 10 November</span></span>
SELECT * FROM seat_holds_native WHERE hold_period << '[2025-11-10, 2025-11-15)'::tstzrange;</span></span></code></pre>
Boundary Extension operators - &<</code> and &></code></strong> let you reason about range boundaries independently, checking whether one range extends beyond another's endpoints (i.e. it can start/end anywhere within the given range).</p>
-- find holds that end before or at the same time as reference range ends</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span><</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span>
</span>
-- find holds that start at or after reference range starts</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period &</span>></span>'[2025-11-08 09:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre>
Finally the Adjecent operator - -|-</code></strong> checks if two ranges are perfectly contiguous - they MUST touch at exactly one boundary point, but do not overlap. This might be invaluable when checking if a customer can extend an existing hold without any gap or conflict.</p>
-- find holds that are immediately adjacent (touching) to given range</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>-</span>|</span>-</span> '[2025-11-08 17:00, 2025-11-08 18:00)'</span>::tstzrange;</span></span></code></pre>To Infinity and Beyond</a>
</h2>
Similar to base types ranges in PostgreSQL can handle NULL values, but it does not stop there. There are also special states specifically applicable to data type ranges: empty</code> and infinity</code>.</p>
Let's start with infinite bounds</strong>, the bound that shows the real power of the ranges. You can define range that extend infinitely in either direction (or both at the same time).</p>
-- range that never expires (upper bound is infinite)</span></span>
SELECT</span> '[2025-11-01 10:00, infinity)'</span>::tstzrange;</span></span>
</span>
-- range that has always been valid (lower bound is infinite)</span></span>
SELECT</span> '[-infinity, 2025-11-01 10:00)'</span>::tstzrange;</span></span>
</span>
-- range covering all time</span></span>
SELECT</span> '[-infinity, infinity)'</span>::tstzrange;</span></span></code></pre>
This gives you ability to describe the "from this points forward" use cases. As we will cover later we can easily define lifetime subscription.</p>
-- lifetime subscription that never expires</span></span>
INSERT INTO</span> subscriptions (user_id, plan, active_period)</span></span>
VALUES</span> (</span>42</span>, </span>'lifetime'</span>, </span>'[2025-11-01, infinity)'</span>);</span></span>
</span>
-- all active subscriptions right now</span></span>
SELECT * FROM</span> subscriptions</span></span>
WHERE</span> active_period @</span>> NOW</span>();</span></span></code></pre>
Using infinity</code> is far more elegant solution that using NULL values or "special" values like 2099-31-12</code> - it's explicit and clearly communicates the data intent.</p>
At any point you can validate whatever range has infinite bounds:</p>
SELECT</span></span>
  lower_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> lower_is_infinite,</span></span>
  upper_inf(</span>'[2025-11-01, infinity)'</span>::tstzrange) </span>AS</span> upper_is_infinite;</span></span></code></pre>Understanding NULL vs empty: Schrödinger's Range</a>
</h2>
Ranges can be NULL or empty, and these are completely different things. NULL is Schrödinger's range - you haven't looked in the box yet, so it could be anything or nothing. Empty is when you've opened the box and confirmed it's empty.</p>
Let's see this in practice:</p>
-- NULL range: we don't know what the period is</span></span>
INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span>
VALUES</span> (</span>42</span>, </span>'abc-123'</span>, </span>NULL</span>);</span></span>
</span>
-- empty range: we know the period is explicitly "nothing"</span></span>
INSERT INTO</span> seat_holds_native (seat_id, user_session_id, hold_period)</span></span>
VALUES</span> (</span>43</span>, </span>'def-456'</span>, </span>'empty'</span>);</span></span></code></pre>
The main difference between them is when it comes to handling.</p>
SELECT NULL</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span>
-- Result: NULL (not true, not false—we don't know)</span></span>
</span>
SELECT</span> 'empty'</span>::tstzrange && </span>'[2025-11-01, 2025-11-08)'</span>::tstzrange;</span></span>
-- Result: false (we know it doesn't overlap)</span></span></code></pre>
And you can check for them in your queries using built-in function isempty</code>.</p>
-- check for NULL (like any column)</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> hold_period </span>IS NULL</span>;</span></span>
</span>
-- check for empty (special function)</span></span>
SELECT * FROM</span> seat_holds_native </span>WHERE</span> isempty(hold_period);</span></span></code></pre>
In practice, you'll mostly use NOT NULL</code> constraints to prevent NULL ranges entirely. Empty ranges are useful but rare - usually for representing cancelled/void periods you need to keep for special purposes - like audit trail.</p>
Practical Integer ranges for Tiered pricing</a>
</h2>
While we introduced ranges we mostly paid attention to the date/time handling the usefulness of range types goes well beyond that. One of the practical applications where integer ranges provide real values can be demostrated on tiered pricing.</p>
CREATE TABLE</span> quantity_discounts</span> (</span></span>
    discount_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    product_id </span>INTEGER NOT NULL</span>,</span></span>
    quantity_range INT4RANGE </span>NOT NULL</span>,</span></span>
    discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span>
</span>
    -- no overlapping tiers</span></span>
    EXCLUDE </span>USING</span> GIST (</span></span>
        product_id </span>WITH =</span>,</span></span>
        quantity_range </span>WITH</span> &&</span></span>
    )</span></span>
);</span></span>
</span>
INSERT INTO</span> quantity_discounts (product_id, quantity_range, discount_percentage) </span>VALUES</span></span>
    -- 1-9 units: no discount</span></span>
    (</span>1</span>, </span>'[1,10)'</span>,     </span>0</span>.</span>00</span>),</span></span>
    -- 10-49 units: 5% off</span></span>
    (</span>1</span>, </span>'[10,50)'</span>,    </span>5</span>.</span>00</span>),</span></span>
    -- 50-99 units: 10% off</span></span>
    (</span>1</span>, </span>'[50,100)'</span>,   </span>10</span>.</span>00</span>),</span></span>
    -- 100+ units: 15% off</span></span>
    (</span>1</span>, </span>'[100,1000)'</span>, </span>15</span>.</span>00</span>);</span></span>
</span>
-- verify what discount we offer for ordering 75 units?</span></span>
SELECT</span> discount_percentage</span></span>
FROM</span> quantity_discounts</span></span>
WHERE</span> product_id </span>=</span> 1</span></span>
  AND</span> quantity_range @</span>></span> 75</span>;</span></span>
-- Result: 10.00</span></span></code></pre>Making Bad Data Impossible</a>
</h2>
If the introduction of the range types provided the case for cleaner schema you can go ahead and make hard limits structurally impossible. While this is not advocating for the transition of the full business logic into database schema, you can eliminate the edge cases that should never make it to the database.</p>
CREATE TABLE</span> promotional_campaigns</span> (</span></span>
    campaign_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    name TEXT NOT NULL</span>,</span></span>
    active_period TSTZRANGE </span>NOT NULL</span>,</span></span>
    budget_range NUMRANGE </span>NOT NULL</span>,</span></span>
    discount_percentage </span>NUMERIC</span>(</span>5</span>,</span>2</span>)</span> NOT NULL</span>,</span></span>
</span>
    -- campaigns must be at least 1 days long</span></span>
    CONSTRAINT</span> campaigns_minimum_duration</span></span>
        CHECK</span> (</span>upper</span>(active_period) </span>-</span> lower</span>(active_period) </span>>=</span> INTERVAL </span>'1 days'</span>),</span></span>
</span>
    -- budget must be between $1000 and $100000</span></span>
    CONSTRAINT</span> campaigns_valid_budget</span></span>
        CHECK</span> (budget_range </span><</span>@ numrange(</span>1000</span>, </span>100000</span>, </span>'[]'</span>)),</span></span>
</span>
    -- active period must not be empty</span></span>
    CONSTRAINT</span> campaigns_valid_period</span></span>
        CHECK</span> (</span>NOT</span> isempty(active_period))</span></span>
);</span></span></code></pre>
While this example demonstrates hypothetical example within the schema definition, please remember they shouldn't be used to implement business process. The goal of the constraints is to enforce data integrity, i.e. structure requirements (minimum duration, non-empty data), physical or domain boundaries. Any other logic should make it's way either to application logic or parts that are easier to modify (think functions).</p>
Multiranges: When one range is not enough</a>
</h2>
Up until now, we've been working with single continuous ranges. But what happens when you need to represent fragmented ranges? In past you needed a separate table with a foreign key relationship. With multiranges, you can store multiple non-contiguous ranges in a single column.</p>
PostgreSQL 14 introduced multirange types for all the built-in range types:</p>

int4multirange</code>, int8multirange</code>, nummultirange</code></li>
tsmultirange</code>, tstzmultirange</code>, datemultirange</code></li>
</ul>
The real power of multiranges lies in schema density</strong> and query efficiency</strong>. We can prove this by comparing the cost of storing and querying the exact same data using two different valid range schemas.</p>
Let's consider storing 20 fragmented and non-contiguous periods - a pattern common for historical data.</p>
CREATE TABLE</span> user_periods_single_range</span> (</span></span>
    period_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    user_id </span>INTEGER NOT NULL</span>,</span></span>
    active_period TSTZRANGE </span>NOT NULL</span></span>
);</span></span>
</span>
CREATE INDEX</span> user_periods_single_range_gist_idx</span></span>
    ON</span> user_periods_single_range</span></span>
    USING</span> gist (active_period);</span></span>
</span>
-- 20 fragmented periods for user_id 42 (20 rows total)</span></span>
INSERT INTO</span> user_periods_single_range (user_id, active_period)</span></span>
SELECT</span></span>
    42</span>,</span></span>
    tstzrange(</span></span>
        '2025-01-01'</span>::</span>timestamptz +</span> (i </span>*</span> 2</span> ||</span> ' days'</span>)::interval,</span></span>
        '2025-01-01'</span>::</span>timestamptz +</span> (i </span>*</span> 2</span> +</span> 1</span> ||</span> ' days'</span>)::interval</span></span>
    )</span></span>
FROM</span> generate_series</span>(</span>1</span>, </span>20</span>) </span>AS</span> s(i);</span></span></code></pre>
compared to 1 row for the same 20 periods aggregated using range_agg</code> function to consolidate data.</p>
CREATE TABLE</span> user_periods_multirange</span> (</span></span>
    user_id </span>INTEGER PRIMARY KEY</span>,</span></span>
    active_periods TSTZMULTIRANGE </span>NOT NULL</span></span>
);</span></span>
</span>
CREATE INDEX</span> user_periods_multirange_gist_idx</span></span>
    ON</span> user_periods_multirange</span></span>
    USING</span> gist (active_periods);</span></span>
</span>
-- consolidate the 20 TSTZRANGE rows into 1 TSTZMULTIRANGE row</span></span>
INSERT INTO</span> user_periods_multirange (user_id, active_periods)</span></span>
SELECT</span></span>
    user_id,</span></span>
    -- range_agg function automatically handles merging adjacent ranges</span></span>
    range_agg(active_period)::TSTZMULTIRANGE</span></span>
FROM</span> user_periods_single_range</span></span>
WHERE</span> user_id </span>=</span> 42</span></span>
GROUP BY</span> user_id;</span></span></code></pre>
consider now the difference between</p>
EXPLAIN</span></span>
SELECT</span> period_id, user_id</span></span>
FROM</span> user_periods_single_range</span></span>
WHERE</span></span>
    user_id </span>=</span> 42</span></span>
    AND</span> active_period @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre>                                           QUERY PLAN</span></span>
-------------------------------------------------------------------------------------------------</span></span>
 Bitmap Heap Scan on user_periods_single_range  (cost=4.19..13.67 rows=1 width=12)</span></span>
   Recheck Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span>
   Filter: (user_id = 42)</span></span>
   ->  Bitmap Index Scan on user_periods_single_range_gist_idx  (cost=0.00..4.19 rows=6 width=0)</span></span>
         Index Cond: (active_period @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre>
and same version of the consolidated data</p>
EXPLAIN ANALYZE</span></span>
SELECT</span> user_id, active_periods</span></span>
FROM</span> user_periods_multirange</span></span>
WHERE</span></span>
    user_id </span>=</span> 42</span></span>
    AND</span> active_periods @</span>></span> '2025-01-20 12:00:00+00'</span>::</span>timestamptz</span>;</span></span></code></pre>                                                 QUERY PLAN</span></span>
-------------------------------------------------------------------------------------------------------------</span></span>
 Index Scan using user_periods_multirange_pkey on user_periods_multirange  (cost=0.15..8.17 rows=1 width=36)</span></span>
   Index Cond: (user_id = 42)</span></span>
   Filter: (active_periods @> '2025-01-20 13:00:00+01'::timestamp with time zone)</span></span></code></pre>
Giving you significant reduction in the query cost. And while the example was simple enough for the demonstration purposes you can easily define the helper schema for better indexable data access and opportunities to reduce the storage requirements.</p>
One important note is that subtle change with the &&</code> operators. Whereas with single range &&</code> operator checks if two continues ranges overlap, for multiranges the operator checks if ANY range in multirange overlaps.</p>
Creating custom range types</a>
</h2>
While PostgreSQL provides built-in range types for common data types, you can create custom range types for any data type that has a meaningful ordering. Let's demostrate this with a type for IP address ranges.</p>
To create a custom range type, you need to provide a subtype difference function that tells PostgreSQL how to calculate the "distance" between two values:</p>
-- function to calculate difference between two IP addresses using bigint</span></span>
CREATE OR REPLACE FUNCTION</span> inet_diff</span>(x </span>INET</span>, y </span>INET</span>)</span></span>
RETURNS</span> FLOAT8 </span>AS</span> $$</span></span>
    SELECT</span> (</span></span>
        (host(x)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>) </span>-</span></span>
        (host(y)::</span>inet -</span> '0.0.0.0'</span>::</span>inet</span>)</span></span>
    )::float8;</span></span>
$$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span>
</span>
-- custom inetrange type</span></span>
CREATE TYPE</span> inetrange</span> AS RANGE</span> (</span></span>
    subtype </span>= inet</span>,</span></span>
    subtype_diff </span>=</span> inet_diff</span></span>
);</span></span>
</span>
-- convert CIDR ranges to inetranges</span></span>
CREATE OR REPLACE FUNCTION</span> cidr_to_inetrange</span>(</span>cidr CIDR</span>)</span></span>
RETURNS</span> inetrange </span>AS</span> $$</span></span>
    SELECT</span> inetrange(</span></span>
        host(network($</span>1</span>))::</span>inet</span>,</span></span>
        host(broadcast($</span>1</span>))::</span>inet</span>,</span></span>
        '[]'</span></span>
    );</span></span>
$$ </span>LANGUAGE SQL</span> IMMUTABLE STRICT;</span></span></code></pre>
Now you can use these custom types just like the built-in ones:</p>
CREATE TABLE</span> ip_blocklists</span> (</span></span>
    blocklist_name </span>TEXT PRIMARY KEY</span>,</span></span>
    blocked_range inetrange </span>NOT NULL</span></span>
);</span></span>
</span>
INSERT INTO</span> ip_blocklists (blocklist_name, blocked_range) </span>VALUES</span></span>
	(</span>'attack #434401'</span>, cidr_to_inetrange(</span>'192.168.1.0/24'</span>)),</span></span>
	(</span>'attack #434401 (1)'</span>, </span>'[203.0.113.50,203.0.113.99]'</span>),</span></span>
    (</span>'attack #434401 (2)'</span>, </span>'[203.0.113.143,203.0.113.159]'</span>);</span></span></code></pre>
and locate which attack has been assigned to particular malicious IP address.</p>
SELECT</span> blocklist_name, blocked_range</span></span>
FROM</span> ip_blocklists</span></span>
WHERE</span> blocked_range @</span>></span> '192.168.1.25'</span>::</span>INET</span>;</span></span></code></pre> blocklist_name |        blocked_range</span></span>
----------------+-----------------------------</span></span>
 attack #434401 | [192.168.1.0,192.168.1.255]</span></span></code></pre>
But wait a second, wasn't the fragmented nature of the ranges used in case for multiranges? Building real-life production and auto adaptive block list would most likely soon create extremely fragmented set of targets to block.</p>
Can we defined it for our custom range types? Well no, because PostgreSQL is amazing! Since PostgreSQL 14 every time you define custom range type, it will automatically create corresponding multirange! Making it easy to consolidate the fragmented data corresponding to individual attack to multiranges.</p>
SELECT</span> typname </span>FROM</span> pg_type </span>WHERE</span> typname </span>LIKE</span> 'inet%range'</span>;</span></span></code></pre>    typname</span></span>
----------------</span></span>
 inetmultirange</span></span>
 inetrange</span></span></code></pre>
Danger zone! When creating a custom range type the subtype_diff</strong> function is more than just simple helper. It plays important role in indexing and query performance</strong>. It really tells PostgreSQL planner how far apart the values in range are, which is crucial when building GiST indexes for ranges.</p>
In our example above, if inet_diff</code> returned 0</code> for every pair of IP addresses, PostgreSQL would think all ranges are "equally close". This would lead to un-balanced indexes, with large hotposts in the indexes. End result would be that range operators would effectively be almost as slow as sequantial scans.</p>
Performance deep dive: GiST vs GIN</a>
</h2>
Throughout this article, we've been using GiST indexes almost exclusively, particularly when enforcing exclusion constraints. But PostgreSQL also supports GIN indexes for range types, and understanding when to use each can make the difference between a query that completes in milliseconds versus one that grinds your database to a halt.</p>
Before we deep dive, let's recap what those two indexes do. GiST (Generalized Search Tree)</strong> is a balanced tree structure that organizes ranges by their bounding boxes, and grouping those that are "close together" in same tree nodes. While GIN (Generalized Inverted Index)</strong> would decompose ranges into their components and indexing those. Therefore GiST works for continous ranges (timestamps and numerical values), while GIN works for discrete ranges (as you can't generate unpredictable range of values of floats for example). Given this characteristics you can almost certainly say GIN indexes are almost always going to be bigger compared to the GiST ones, as they are always trying to index a continous space.</p>
The most important thing to know upfront? As we already used without explicitely mentioning it - you can't use GIN with</strong> EXCLUDE constraints.</strong> GiST is mandatory there.</p>
Therefore while GiST index is going to be preferred for cases like</p>
CREATE TABLE</span> seat_holds</span> (</span></span>
    hold_id </span>BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    seat_id </span>INTEGER NOT NULL</span>,</span></span>
    hold_period TSTZRANGE </span>NOT NULL</span></span>
);</span></span></code></pre>
the GIN index is actually preferred for</p>
CREATE TABLE</span> venue_blackouts</span> (</span></span>
    venue_id </span>INTEGER</span>,</span></span>
    blocked_dates DATERANGE </span>NOT NULL</span></span>
);</span></span></code></pre>
The reason is simple - dates are discrete type. There's only 365 (or 366) combinations in a given year. While timestamps have microsecond precision - millions of possible values per day.</p>
The complexity of the index types is far beyond this article scope, so let's just iterate over basic rules what index type to use. Most applications should just use GiST and move on. The performance difference rarely matters until you're dealing with millions of rows and very specific query patterns. Don't prematurely optimize - GiST is the safe, versatile default that works well for almost everything. You can always add a GIN index later if profiling shows it would help. PostgreSQL's query planner is smart enough to pick the better index when both are available.</p>
Conclusion</a>
</h2>
From my experience range types represent one of PostgreSQL's most underutilized features, yet they offer immediate benefits: cleaner schemas, built-in data integrity, and query patterns that feel natural once you embrace them. What started as a solution to prevent double-booking seats evolved into a comprehensive look at how treating ranges as first-class citizens transforms your database design.</p>
But we've really just scratched the surface. Timestamp ranges in particular open an entire world of possibilities we haven't touched - temporal tables</strong>. The ability to maintain complete historical records with automatic versioning, query data "as of" any point in time, and track changes without cluttering your schema with audit columns deserves its own deep dive. That's a topic for a future article.</p>
For now, the next time you reach for separate start</code> and end</code> columns, stop and ask yourself: "Should this be a range?" More often than not, the answer is yes, and it will save you a debugging session later.</p>


Time to Better Know The Time in PostgreSQL
2025-04-06T00:00:00+00:00
To honor the name of the site (boringSQL) let's deep dive into a topic which might sound obvious, but it might be never ending source of surprises and misunderstanding.</p>
Simple things</a>
</h2>
We can start with the simple statement like</p>
SELECT</span> '2025-03-30 00:30'</span> as</span> t;</span></span></code></pre>
Result:</p>
       t</span></span>
------------------</span></span>
2025-03-30 00:30</span></span></code></pre>
which gives you (and I do hope that's not a big surprise) a simple text literal, rather than anything to do with date and time. As soon as you use the string literal in queries similar to</p>
SELECT * FROM</span> events </span>WHERE</span> start_at </span>></span> '2025-03-30 00:30'</span>;</span></span></code></pre>
PostgreSQL will implicitly convert the string into timestamp without time zone. The reason why it can happen is the fact start_at</code> is most likely timestamp based field, allowing automatic cast to match the column's data type. Which is unlike the query</p>
SELECT</span> '2025-03-30 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre>
Result:</p>
ERROR:  invalid input syntax for type interval: "2025-03-30 01:00"</span></span></code></pre>
which will simply not work, as PostgreSQL can't perform automatic cast (which as we will cover later might be surprising behavior, but that's how things are). The correct way to get this example query to work is to use either one of two ways (both functionally equivalent but different notation).</p>
-- PostgreSQL cast notation</span></span>
SELECT</span> '2025-03-30 01:00'</span>::</span>timestamp -</span> interval </span>'15 minutes'</span>;</span></span>
</span>
-- SQL standard explicit type notation</span></span>
SELECT timestamp</span> '2025-03-03 01:00'</span> -</span> interval </span>'15 minutes'</span>;</span></span></code></pre>PostgreSQL timestamp vs timestamptz</a>
</h2>
The next possible source of confusion when working with time in PostgreSQL is presence of two distinct data types:</p>

timestamp</code> (or timestamp without time zone</code>)</li>
timestamptz</code> (or timestamp with time zone</code>)
Despite what the names suggest, the key difference isn't whether they store timezone information</strong>, but rather how they handle it during storage and retrieval.</li>
</ul>

IMPORTANT: before we cover the details, remember to always use timestamptz</code>. As official Don't Do This</a> page shows, using timestamp</code> is like storing picture of the clock, instead of point in time. And while this article might use timestamp</code> it's purely for demonstration purposes.</p>
</blockquote>
All you need to remember is the fact timestamp</code> stores the exact datetime value as entered</strong> without any timezone context or adjustments. It's essentially a snapshot of a calendar date and wall clock time, disconnected from any particular geographic location.</p>
-- this stores what you provided</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp</span>;</span></span></code></pre>
Result:</p>
      timestamp</span></span>
---------------------</span></span>
2025-03-30 01:30:00</span></span></code></pre>
On the other hand timestamptz</code> normalizes all input to UTC internally</strong> and then converts values to the session's timezone for display. This provides geographic context to your datetime values.</p>
-- this converts your input to UTC based on your session timezone</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre>
Result:</p>
      timestamptz</span></span>
-----------------------</span></span>
2025-03-30 01:30:00+01</span></span></code></pre>
And this is where potential confusion might start. Let's take this example</p>
-- With UTC timezone</span></span>
SET</span> timezone </span>=</span> 'UTC'</span>;</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre>
Result:</p>
      timestamptz</span></span>
-----------------------</span></span>
 2025-03-30 01:30:00+00</span></span></code></pre>-- With Tokyo timezone</span></span>
SET</span> timezone </span>=</span> 'Asia/Tokyo'</span>;</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz</span>;</span></span></code></pre>
Result:</p>
      timestamptz</span></span>
-----------------------</span></span>
 2025-03-30 01:30:00+09</span></span></code></pre>
which gives you correct representation based on the session timezone. This comes in handy when we add the actual storage.</p>
-- Create table and view with Berlin timezone</span></span>
SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span>
CREATE TABLE</span> time_flies</span>(moment </span>timestamp with time zone</span>);</span></span>
INSERT INTO</span> time_flies </span>VALUES</span> (</span>'2025-03-30 00:30'</span>);</span></span>
SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre>
Result:</p>
         moment</span></span>
------------------------</span></span>
 2025-03-30 00:30:00+01</span></span></code></pre>-- View the same data with New York timezone</span></span>
SET</span> timezone </span>=</span> 'America/New_York'</span>;</span></span>
SELECT</span> moment </span>FROM</span> time_flies;</span></span></code></pre>
Result:</p>
         moment</span></span>
------------------------</span></span>
 2025-03-29 19:30:00-04</span></span></code></pre>
This demonstrates the advantage of using automatic time zone conversion, allowing you to avoid not only time-zone bugs, but also DST related issues.
While most of us might be fast asleep during our local DST changes, it's no fun to account for the time difference in a wrong. Using timestamptz</code> is sure way how to avoid it. Let's take an example of recent (at the time of writing the article) DST change.</p>
-- Set Berlin timezone (during DST change)</span></span>
SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span>
</span>
-- Using timestamp (without timezone awareness)</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre>
Result:</p>
      end_time</span></span>
---------------------</span></span>
 2025-03-30 02:15:00</span></span></code></pre>-- Using timestamptz (with timezone awareness)</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamptz +</span> interval </span>'45 minutes'</span> as</span> end_time;</span></span></code></pre>
Result:</p>
        end_time</span></span>
------------------------</span></span>
 2025-03-30 03:15:00+02</span></span></code></pre>AT TIME ZONE</a>
</h2>
Another powerful but potentially confusing operator in PostgreSQL is AT TIME ZONE</code> which allows you to convert timestamps between different time zones, but its behavior differs depending on the input data type.</p>
When applied to timestamp (without time zone)</strong>
When you apply AT TIME ZONE to a timestamp, PostgreSQL interprets the input as being in the specified time zone</strong> and converts it to a timestamptz:</p>
-- First, set UTC timezone</span></span>
SET</span> timezone</span>=</span>'UTC'</span>;</span></span>
</span>
-- Interpret '2025-03-30 01:30' as if it were in the 'Europe/Paris' time zone</span></span>
SELECT</span> '2025-03-30 01:30'</span>::</span>timestamp AT TIME ZONE</span> 'Europe/Paris'</span>;</span></span></code></pre>
Result:</p>
        timezone</span></span>
------------------------</span></span>
 2025-03-30 00:30:00+00</span></span></code></pre>
What it does is effectively "Take this wall clock time, consider it as being in the specified time zone, and give me the corresponding moment in time (as a timestamptz)". The representation here is based on session time zone settings.</p>
When applied to timestamptz (with time zone)</strong>
Conversely, when you apply AT TIME ZONE to a timestamptz, PostgreSQL converts the timestamp to the specified time zone and returns a timestamp without time zone</strong>:</p>
-- Set timezone</span></span>
SET</span> timezone</span>=</span>'America/Port_of_Spain'</span>;</span></span>
</span>
-- Convert the timestamptz to how it would appear on wall clocks in Tokyo</span></span>
SELECT</span> '2025-03-30 01:30+01:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span>;</span></span></code></pre>
Result:</p>
        timezone</span></span>
---------------------</span></span>
 2025-03-30 09:30:00</span></span></code></pre>
This operation says: "Take this moment in time and show me what wall clock time it would be in the specified time zone."</p>
Practical usage</strong>
Probably the only use when correctly using timestamps</code> is to provide "wall clock" representation of the times at various time zones at once for (view) representational purposes.</p>
-- What time is the company-wide meeting in various offices?</span></span>
SELECT</span></span>
    '2025-03-30 15:00'</span>::</span>timestamptz AS</span> meeting_time_utc,</span></span>
    '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Europe/London'</span> AS</span> london_office_time,</span></span>
    '2025-03-30 15:00'</span>::</span>timestamptz AT TIME ZONE</span> 'Asia/Tokyo'</span> AS</span> tokyo_office_time;</span></span></code></pre>
Result:</p>
    meeting_time_utc    | london_office_time  |  tokyo_office_time</span></span>
------------------------+---------------------+---------------------</span></span>
 2025-03-30 15:00:00+02 | 2025-03-30 14:00:00 | 2025-03-30 22:00:00</span></span></code></pre>
It's important to note that in API based clients, it's always better to represent full time notation and leave the clients with the representational layer. Similar to that, unless you need to work with multiple time zones, and you depend on the correct local time representation it's always better to use local session timezone</code> setting instead of using AT TIME ZONE</code> operator. This applies for both time input and output.</p>
Common mistake</strong>
Let's consider the example where you might consider "magically casting" already stored time using AT TIME ZONE</code>.</p>
-- Setup example</span></span>
SET</span> timezone</span>=</span>'Europe/Berlin'</span>;</span></span>
CREATE TABLE</span> different_moments</span>(moment1 </span>timestamptz</span>, moment2 </span>timestamptz</span>);</span></span>
INSERT INTO</span> different_moments (moment1) </span>values</span> (</span>'2025-03-30 01:30'</span>);</span></span>
</span>
-- Apply double time zone conversion</span></span>
UPDATE</span> different_moments </span>SET</span> moment2 </span>=</span> moment1 </span>AT TIME ZONE</span> 'Europe/Berlin'</span> AT TIME ZONE</span> 'America/New_York'</span>;</span></span>
</span>
-- View the results</span></span>
SELECT * FROM</span> different_moments;</span></span></code></pre>
Result:</p>
        moment1         |        moment2</span></span>
------------------------+------------------------</span></span>
 2025-03-30 01:30:00+01 | 2025-03-30 07:30:00+02</span></span></code></pre>
Unless you are aware of the implicit conversion between timestamps</code> and timestamp</code> and vice-versa you might be set for a lot of surprises. In this particular case the first conversion performed redundant conversion and removed the time zone offset. While second "forced" it to NYC time, while the subsequent select rendered it in local session timezone.</p>
Timestamps and the storage</a>
</h2>
While you will use timestamp with time zone</code> from now on, PostgreSQL still comes with several nuances related to the way it can store the data. First lesser known feature might be timestamp precision. Let's consider following table.</p>
-- Table with various timestamp precision specifications</span></span>
CREATE TABLE</span> precision</span> (</span></span>
	t1 </span>timestamp with time zone</span>,        </span>-- default precision (6)</span></span>
	t2 </span>timestamp</span>(</span>0</span>)</span> with time zone</span>,     </span>-- seconds precision</span></span>
	t3 </span>timestamp</span>(</span>2</span>)</span> with time zone</span>,     </span>-- centiseconds precision</span></span>
	t4 </span>timestamp</span>(</span>4</span>)</span> with time zone</span>      -- 10 microseconds precision</span></span>
);</span></span></code></pre>
Effectively the timestamp</code> comes with default precision of 6 digits (microseconds), but you can specify anywhere from 0 to 6 for both timestamp</code> and timestamptz</code> types. At the same time it does not have any impact on the storage (8 bytes) regardless the precision specified.</p>
-- Insert the same timestamp into all columns</span></span>
INSERT INTO precision VALUES</span> (current_timestamp, current_timestamp, current_timestamp, current_timestamp);</span></span>
SELECT * FROM precision</span>;</span></span></code></pre>
Result:</p>
-[ RECORD 1 ]---------------------</span></span>
t1 | 2025-04-03 21:19:20.952354+02</span></span>
t2 | 2025-04-04 21:19:21+02</span></span>
t3 | 2025-04-04 21:19:20.95+02</span></span>
t4 | 2025-04-04 21:19:20.9524+02</span></span></code></pre>
Other than precision there are some other interesting aspects of the timestamps storage in PostgreSQL:</p>

It has finite range</strong> of timestamptz '4713-01-01 BC'</code> and timestamptz '294276-12-31'</code></li>
and at the same time supports positive and negative infinity</strong> with 'infinity'::timestamptz</code> and '-infinity'::timestamptz</code> that might provide alternative to NULL</code> for open ended intervals. If you choose to use infinity instead of NULL</code> values, you can test whatever the provided date/timestamp is finite value or not using isfinite(timestamp)</code>.</li>
</ul>
Getting the current time</a>
</h2>
Not many developers are aware that there are different ways to get the current datetime in SQL. These methods not only differ in syntax but also in their underlying logic.</p>
CURRENT_TIMESTAMP vs NOW()</a>
</h3>
CURRENT_TIMESTAMP</strong> is part of the SQL Standard and interestingly, it's defined as both a "time-varying variable" and a "datetime value function". In PostgreSQL, you can use it with or without parentheses, making both these forms valid:</p>
SELECT</span> CURRENT_TIMESTAMP;</span></span>
SELECT</span> CURRENT_TIMESTAMP</span>(</span>2</span>);  </span>-- (2) specifies precision</span></span></code></pre>
Along with CURRENT_TIMESTAMP, the SQL Standard also defines CURRENT_TIME</code> and CURRENT_DATE</code> for working with individual time and date components respectively.</p>
NOW()</code>, while not part of the SQL Standard, is a widely used alternative across many database systems. Unlike CURRENT_TIMESTAMP, it's strictly a function and therefore always requires parentheses when called. Its straightforward syntax - NOW()</code> - makes it a popular choice among developers.</p>
Transaction Behavior</a>
</h3>
Both NOW()</code> and CURRENT_TIMESTAMP</code> share the same transaction behavior: they return the timestamp from the start of the current transaction, not the moment of execution:</p>
BEGIN</span>;</span></span>
SELECT NOW</span>();                     </span>-- Returns transaction start time</span></span>
SELECT</span> pg_sleep(</span>5</span>);               </span>-- Wait 5 seconds</span></span>
SELECT NOW</span>();                     </span>-- Still returns the same time</span></span>
COMMIT</span>;</span></span></code></pre>
This behavior ensures data consistency within transactions but might be unexpected when measuring elapsed time.</p>
Alternative Time Functions</a>
</h3>
When you need the actual current time regardless of transaction status, CLOCK_TIMESTAMP()</code> is your best choice. It returns the precise moment of execution, making it particularly useful for measuring real elapsed time. Here's how it differs from NOW()</code>:</p>
BEGIN</span>;</span></span>
SELECT</span> pg_sleep(</span>1</span>);</span></span>
SELECT NOW</span>()</span> AS</span> transaction_time,</span></span>
       CLOCK_TIMESTAMP()</span> AS</span> actual_time;</span></span>
COMMIT</span>;</span></span></code></pre>
Result:</p>
       transaction_time        |          actual_time</span></span>
------------------------------+-------------------------------</span></span>
 2025-04-03 19:28:25.310254+00 | 2025-04-03 19:28:26.311917+00</span></span></code></pre>
Another useful function is STATEMENT_TIMESTAMP()</code>, which captures the time when the current statement began executing. This differs subtly from CLOCK_TIMESTAMP()</code>, which gives you the exact moment of function execution. The difference becomes clear in this example:</p>
SELECT</span></span>
    pg_sleep(</span>2</span>),</span></span>
    STATEMENT_TIMESTAMP()</span> AS statement</span>,</span></span>
    CLOCK_TIMESTAMP()</span> AS</span> wall_time;</span></span></code></pre>
Result:</p>
 pg_sleep |           statement           |          wall_time</span></span>
----------+-------------------------------+------------------------------</span></span>
          | 2025-04-03 19:32:15.984459+00 | 2025-04-03 19:32:17.98661+00</span></span></code></pre>
These timestamp functions serve different purposes and are invaluable when you need to measure transaction timing or analyze statement-level performance. Each provides a different perspective on time within your database operations.</p>
Intervals</a>
</h2>
Guess if you have worked with time in PostgreSQL you have come across with intervals. You probably just entered the it using something like interval '1 year 2 months 3 days 4 hours'</code>. Technical that's representation driven by setting intervalstyle</code> which (as in this case) is set to postgres_verbose</code>. But you have much more options.</p>
-- PostgreSQL default style</span></span>
SET</span> intervalstyle </span>=</span> 'postgres'</span>;</span></span>
SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre>
Result:</p>
           interval</span></span>
-------------------------------</span></span>
 1 year 2 mons 3 days 04:00:00</span></span></code></pre>-- SQL standard style</span></span>
SET</span> intervalstyle </span>=</span> 'sql_standard'</span>;</span></span>
SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre>
Result:</p>
     interval</span></span>
------------------</span></span>
 +1-2 +3 +4:00:00</span></span></code></pre>-- ISO 8601 style</span></span>
SET</span> intervalstyle </span>=</span> 'iso_8601'</span>;</span></span>
SELECT</span> interval </span>'1 year 2 months 3 days 4 hours'</span>;</span></span></code></pre>
Result:</p>
  interval</span></span>
------------</span></span>
 P1Y2M3DT4H</span></span></code></pre>
With the session intervalstyle</code> you only change the interval format representation, not the actual value.
Nevertheless the interval representation you might run into an interval definition that might not longer make it obvious (both in input or output) to interpret. That's where function justify_interval</code> comes in play - helping you to normalize time expression into conventional formats.</p>
-- Normalize complex interval</span></span>
SELECT</span> justify_interval(interval </span>'15 months 40 days 30 hours'</span>);</span></span></code></pre>
Result:</p>
        justify_interval</span></span>
---------------------------------</span></span>
 @ 1 year 4 mons 11 days 6 hours</span></span></code></pre>-- Practical example: normalize project durations</span></span>
SELECT</span></span>
    project_name,</span></span>
    total_hours </span>||</span> ' hours'</span> AS</span> raw_duration,</span></span>
    justify_interval(interval </span>'1 hour'</span> *</span> total_hours) </span>AS</span> normalized_duration</span></span>
FROM</span></span>
    (</span>VALUES</span> (</span>'Database Migration'</span>, </span>1850</span>),</span></span>
            (</span>'API Development'</span>, </span>720</span>),</span></span>
            (</span>'UI Redesign'</span>, </span>340</span>))</span></span>
    AS</span> projects(project_name, total_hours);</span></span></code></pre>
Result:</p>
    project_name    | raw_duration |   normalized_duration</span></span>
--------------------+--------------+--------------------------</span></span>
 Database Migration | 1850 hours   | @ 2 mons 17 days 2 hours</span></span>
 API Development    | 720 hours    | @ 1 mon</span></span>
 UI Redesign        | 340 hours    | @ 14 days 4 hours</span></span></code></pre>
One of the important specific with internal normalization is the fact it uses 30-days time periods as a month definition.</p>
-- Month definition in normalize intervals</span></span>
SELECT</span> justify_interval(interval </span>'30 days'</span>);</span></span></code></pre>
Result:</p>
 justify_interval</span></span>
------------------</span></span>
 @ 1 mon</span></span></code></pre>
Which might be understandable, but you need to beware of the edge cases when using days and months intervals.</p>
-- Edge case comparison: month vs 30 days</span></span>
SELECT</span></span>
	date</span> '2025-01-31'</span> +</span> interval </span>'1 month'</span> as</span> example1,</span></span>
	date</span> '2025-01-31'</span> +</span> interval </span>'30 days'</span> as</span> example2;</span></span></code></pre>
Result:</p>
-[ RECORD 1 ]-----------------</span></span>
example1 | 2025-02-28 00:00:00</span></span>
example2 | 2025-03-02 00:00:00</span></span></code></pre>
And one last thing, before wrapping up the interval - you can (same as with dates/timestamp) get specific parts using extract</code> function.</p>
-- Extract specific parts from intervals</span></span>
SELECT</span></span>
    extract(</span>days from</span> interval </span>'1 year 3 months 21 days'</span>) </span>AS days</span>,</span></span>
    extract(</span>hours from</span> interval </span>'25:30:45'</span>) </span>AS hours</span>;</span></span></code></pre>
Result:</p>
 days | hours</span></span>
------+-------</span></span>
 21   | 25</span></span></code></pre>Time ranges</a>
</h2>
While working with timestamp comes natural, it's time ranges which often feel like ugly ducklings. And quite unfairly so - timestamp ranges in PostgreSQL are powerful yet underutilized features that elegantly solve common time-based challenges. PostgreSQL offers dedicated range types that are perfect for modeling time periods, reservations, schedules, and any situation where you need to track intervals with defined start and end points:</p>

tsrange</code> - range of timestamp without time zone</li>
tstzrange</code> range of timestamp with time zone
And to follow the earlier advice - just as you should default to timestamps</code>, always use tstzrange</code> for time ranges unless you have VERY specific reason for it.
Timestamp ranges aren't merely convenient syntax - they provide a complete set of operations for determining relationships between time periods and enable powerful constraints that handle complex business rules without reinventing the wheel.
Timestamp ranges can be created using the range constructor syntax or string syntax.</li>
</ul>
-- Range constructor syntax</span></span>
SELECT</span> tstzrange(</span></span>
    '2025-04-01 09:00:00+02'</span>::</span>timestamptz</span>,</span></span>
    '2025-04-01 17:30:00+02'</span>::</span>timestamptz</span>,</span></span>
    '[)'</span></span>
) </span>AS</span> workday;</span></span>
</span>
-- String syntax</span></span>
SELECT</span> '[2025-04-01 09:00:00+02, 2025-04-01 17:30:00+02)'</span>::tstzrange </span>AS</span> workday;</span></span></code></pre>
Result:</p>
                       workday</span></span>
["2025-04-01 07:00:00+00","2025-04-01 15:30:00+00")</span></span></code></pre>
The default boundary notation for timestamp ranges is [)</code> - lower bound inclusive (includes the start time) and upper bound exclusive (excludes the end time).
For time-based applications, this convention is particularly valuable. Consider scheduling consecutive one-hour meetings from 9:00 to 10:00 and 10:00 to 11:00. With [)</code> notation, these are represented as [09:00, 10:00)</code> and [10:00, 11:00)</code>, creating perfect adjacency without overlap or gaps. If you used inclusive bounds [09:00, 10:00]</code> and [10:00, 11:00]</code>, the moment 10:00 would technically belong to both ranges - a logical impossibility for scheduling. This natural alignment with how we conceptualize time slots makes [)</code> notation the default and recommended choice for most of the application use cases.
Timestamp ranges support numerous operations for checking relationships:</p>
-- 1. Check if a timestamp is within a range</span></span>
SELECT</span></span>
    '[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange @</span>></span> '2025-04-01 12:30'</span>::</span>timestamptz AS</span> is_during_workday;</span></span></code></pre>
Result:</p>
 is_during_workday</span></span>
------------------</span></span>
 t</span></span></code></pre>-- 2. Check if two ranges overlap</span></span>
SELECT</span></span>
    '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange &&</span></span>
    '[2025-04-01 11:00, 2025-04-01 14:00)'</span>::tstzrange </span>AS</span> meetings_overlap;</span></span></code></pre>
Result:</p>
 meetings_overlap</span></span>
-----------------</span></span>
 t</span></span></code></pre>-- 3. Extract time range bounds</span></span>
SELECT</span></span>
    lower</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> start_time,</span></span>
    upper</span>(</span>'[2025-04-01 09:00, 2025-04-01 17:00)'</span>::tstzrange) </span>AS</span> end_time;</span></span></code></pre>
Result:</p>
        start_time        |         end_time</span></span>
--------------------------+--------------------------</span></span>
 2025-04-01 09:00:00+00   | 2025-04-01 17:00:00+00</span></span></code></pre>
As well as obvious manipulations:</p>
-- 1. Merge adjacent ranges</span></span>
SELECT</span></span>
    '[2025-04-01 09:00, 2025-04-01 12:00)'</span>::tstzrange </span>+</span></span>
    '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> full_day;</span></span></code></pre>
Result:</p>
                           full_day</span></span>
-------------------------------------------------------------</span></span>
 ["2025-04-01 09:00:00+00","2025-04-01 17:00:00+00")</span></span></code></pre>-- 2. Find intersection between overlapping ranges</span></span>
SELECT</span></span>
    '[2025-04-01 09:00, 2025-04-01 14:00)'</span>::tstzrange </span>*</span></span>
    '[2025-04-01 12:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> overlap_period;</span></span></code></pre>
Result:</p>
                        overlap_period</span></span>
-------------------------------------------------------------</span></span>
 ["2025-04-01 12:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre>-- 3. Find gap between non-overlapping ranges</span></span>
SELECT</span></span>
    '[2025-04-01 09:00, 2025-04-01 11:00)'</span>::tstzrange </span>-</span></span>
    '[2025-04-01 14:00, 2025-04-01 17:00)'</span>::tstzrange </span>AS</span> gap;</span></span></code></pre>
Result:</p>
                             gap</span></span>
-------------------------------------------------------------</span></span>
 ["2025-04-01 11:00:00+00","2025-04-01 14:00:00+00")</span></span></code></pre>
And if you followed the previously mentioned logic, you wouldn't double guess tstzrange</code> handling of the DST transitions.</p>
-- Testing DST transition handling</span></span>
SET</span> timezone </span>=</span> 'Europe/Berlin'</span>;</span></span>
SELECT</span></span>
    '[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange </span>AS</span> dst_transition_range,</span></span>
    upper</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>-</span></span>
    lower</span>(</span>'[2025-03-30 01:00:00, 2025-03-30 03:00:00)'</span>::tstzrange) </span>AS</span> actual_duration;</span></span></code></pre>
Result:</p>
                dst_transition_range                 | actual_duration</span></span>
-----------------------------------------------------+-----------------</span></span>
 ["2025-03-30 01:00:00+01","2025-03-30 03:00:00+02") | @ 1 hour</span></span></code></pre>
The area where time ranges win in almost all scenarios is indexing. If you consider a manual implementation of the range using start_at</code> and end_at</code> columns you typically rely on B-tree indexes. While they might work for simple queries on just one of the values, they fall short for time ranges. Consider this example:</p>
-- Traditional approach with two columns and B-tree indexes</span></span>
SELECT * FROM</span> events_columns</span></span>
WHERE</span> start_at </span><=</span> '2025-04-01 17:00'</span></span>
  AND</span> end_at </span>>=</span> '2025-04-01 09:00'</span>;</span></span></code></pre>
For this query, PostgreSQL might use one of the indexes if it's selective enough, but the fundamental problem remains: each B-tree index can only efficiently filter on its own column. The planner might use the start_at index to find events starting before 17:00, or the end_at index to find events ending after 09:00, but it can't use both indexes simultaneously to find the intersection.
With range types, we can use GiST</strong> (Generalized Search Tree) indexes that are specifically designed for range operations:</p>
-- Create specialized GiST index for time range operations</span></span>
CREATE INDEX</span> idx_events_range</span> ON</span> events_range </span>USING</span> GIST(time_period);</span></span></code></pre>
This allows you to transform the sample query into:</p>
-- Range-based approach using the && (overlap) operator</span></span>
SELECT * FROM</span> events_range</span></span>
WHERE</span> time_period && tstzrange(</span>'2025-04-01 09:00'</span>, </span>'2025-04-01 17:00'</span>);</span></span></code></pre>
The performance gap between these approaches is substantial – range queries with GiST indexes typically outperform</strong> the traditional approach by 2-10x on large datasets. This difference becomes even more pronounced as your data grows.</p>
Time to wrap up</a>
</h2>
Working with time in PostgreSQL has more edge cases than it first appears. To summarize:</p>

Always use timestamptz instead of timestamp</strong></li>
Be intentional about time zone handling</strong></li>
Leverage PostgreSQL's specialized time tools</strong></li>
Mind the edge cases</strong></li>
</ul>
Time handling has more edge cases than most software engineering problems, but PostgreSQL's time types cover nearly all of them. Understanding these "boring" concepts will save you from the pitfalls covered above.</p>


VIEW inlining in PostgreSQL
2025-02-08T00:00:00+00:00
Database VIEWs are powerful tools that often don't get the attention they deserve when building database-driven applications. They make our database work easier in several ways:</p>

They let us reuse common query patterns instead of writing them over and over</li>
They give us a place to define business rules once and use them everywhere</li>
They help us write cleaner, more organized queries</li>
</ul>
Let's see how this works with a practical example. Imagine we want to work with active users</em> - users who have used the application within the last 7 days. Instead of writing this condition in every query, we can define a view:</p>
CREATE VIEW</span> active_users</span> AS</span></span>
SELECT</span></span>
	*</span></span>
FROM</span> users;</span></span>
WHERE</span></span>
	last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span>;</span></span></code></pre>
Now we can easily use this view whenever we need active users. For example, if we want to find active users from Germany:</p>
SELECT</span></span>
	user_id</span></span>
FROM</span> active_users</span></span>
WHERE</span></span>
	country </span>=</span> 'Germany'</span>;</span></span></code></pre>
While this simple example shows how views can make developers' lives easier by organizing and reusing common logic, there's more to the story. In this article, we'll explore something even more interesting: how PostgreSQL can optimize these views through a process called "inlining" - making them not just convenient, but fast too.</p>
What is VIEW inlining</a>
</h2>
When you use a view, it acts like a building block in SQL queries. Taking our previous example, PostgreSQL effectively transforms the query by replacing the view with its underlying subquery.</p>
SELECT</span></span>
	user_id</span></span>
FROM</span> (</span></span>
	SELECT</span></span>
		*</span></span>
	FROM</span> users</span></span>
	WHERE</span></span>
		last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span>
) active users</span></span>
WHERE</span></span>
	country </span>=</span> 'Germany'</span>;</span></span></code></pre>
While we write the query using active_users</code> VIEW, PostgreSQL query planner will see it as an opportunity to optimize it further. Instead of treating the sub-query as separate step (and retrieving large subset of the users), it effectively transforms the query and inlines</strong> the view into the query itself. Behind the scenes, PostgreSQL will execute the query similar to:</p>
SELECT</span></span>
	user_id</span></span>
FROM</span> users</span></span>
WHERE</span></span>
	last_login </span>></span> current_date </span>-</span> INTERVAL </span>'7 days'</span></span>
	AND</span> country </span>=</span> 'Germany'</span>;</span></span></code></pre>
The inlining process</strong> allows the PostgreSQL query planner to optimize the entire query as a single unit. This allows it to select the best indexes, possible join strategies and other aspects together, rather than having to execute the individual parts separately and then filtering the data that won't be otherwise used.  This is exactly what VIEWs are for - we get the best of both worlds - clean, modular code for developers and optimal performance for the database.</p>
Inlining in action</a>
</h2>
The easiest way how to preview the VIEW inlining is to run EXPLAIN</p>
 Seq Scan on users  (cost=0.00..19.20 rows=1 width=4)</span></span>
   Filter: ((country = 'Germany'::text) AND (last_login > (CURRENT_DATE - '7 days'::interval)))</span></span></code></pre>
The Filter part here is the one showing how the filtering conditions from both the query and the view got merged together by the query planner.</p>
This works especially well with complex queries involving joins, aggregations and filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p>
CREATE VIEW</span> order_analytics</span> AS</span></span>
SELECT</span></span>
    o</span>.</span>customer_id</span>,</span></span>
    c</span>.</span>country</span>,</span></span>
    DATE_TRUNC(</span>'month'</span>, </span>o</span>.</span>created_at</span>) </span>as month</span>,</span></span>
    COUNT</span>(</span>*</span>) </span>as</span> orders,</span></span>
    SUM</span>(</span>o</span>.</span>total_amount</span>) </span>as</span> revenue</span></span>
FROM</span> orders o</span></span>
JOIN</span> customers c </span>ON</span> c</span>.</span>id</span> =</span> o</span>.</span>customer_id</span></span>
GROUP BY</span> 1</span>, </span>2</span>, </span>3</span>;</span></span></code></pre>
When you use this VIEW in the query</p>
SELECT</span> customer_id, revenue</span></span>
FROM</span> order_analytics</span></span>
WHERE</span> country </span>=</span> 'Germany'</span></span>
AND month >=</span> '2024-01-01'</span></span>
AND</span> revenue </span>></span> 1000</span>;</span></span></code></pre>
you will see how PostgreSQL effectively inlines the filter conditions where they belong.</p>
HashAggregate  (cost=205.97..207.97 rows=200 width=16)</span></span>
   Group Key: o.customer_id, c.country, (date_trunc('month'::text, o.created_at))</span></span>
   Filter: (sum(o.total_amount) > 1000)</span></span>
   ->  Hash Join  (cost=16.12..185.25 rows=1235 width=20)</span></span>
         Hash Cond: (o.customer_id = c.id)</span></span>
         ->  Seq Scan on orders o</span></span>
             Filter: (created_at >= '2024-01-01'::date)</span></span>
         ->  Hash</span></span>
               ->  Seq Scan on customers c</span></span>
                     Filter: (country = 'Germany'::text)</span></span></code></pre>
This works especially well with increasing query complexity. Particulary involving joins and further filters. PostgreSQL can optimize entire query chain as one unit instead of executing individual pieces separately.</p>
CREATE VIEW completed_orders AS</span></span>
SELECT</span></span>
   o.id,</span></span>
   o.customer_id,</span></span>
   o.total_amount,</span></span>
   o.created_at,</span></span>
   o.status,</span></span>
   p.name as product_name,</span></span>
   c.email,</span></span>
   c.country</span></span>
FROM orders o</span></span>
JOIN customers c ON c.id = o.customer_id</span></span>
JOIN order_items oi ON oi.order_id = o.id</span></span>
JOIN products p ON p.id = oi.product_id</span></span>
WHERE</span></span>
   o.status = 'completed';</span></span>
</span>
SELECT</span></span>
	*</span></span>
FROM completed_orders</span></span>
WHERE</span></span>
	country = 'Germany';</span></span>
</span>
CREATE INDEX idx_orders_customer_id ON orders(customer_id); CREATE INDEX idx_order_items_order_id ON order_items(order_id); CREATE INDEX idx_order_items_product_id ON order_items(product_id);</span></span></code></pre>
as demonstrated in the query plan</p>
 Nested Loop  (cost=0.30..19.70 rows=1 width=176)</span></span>
   ->  Nested Loop  (cost=0.15..13.52 rows=1 width=148)</span></span>
         Join Filter: (o.id = oi.order_id)</span></span>
         ->  Nested Loop  (cost=0.15..12.43 rows=1 width=144)</span></span>
               ->  Seq Scan on orders o  (cost=0.00..1.05 rows=1 width=80)</span></span>
                     Filter: (status = 'completed'::text)</span></span>
               ->  Index Scan using customers_pkey on customers c  (cost=0.15..8.17 rows=1 width=68)</span></span>
                     Index Cond: (id = o.customer_id)</span></span>
                     Filter: (country = 'Germany'::text)</span></span>
         ->  Seq Scan on order_items oi  (cost=0.00..1.04 rows=4 width=8)</span></span>
   ->  Index Scan using products_pkey on products p  (cost=0.15..6.17 rows=1 width=36)</span></span>
         Index Cond: (id = oi.product_id)</span></span></code></pre>Planner barriers</a>
</h2>
While PostgreSQL is very good at inlining views, certain operations create "planner barrier" that prevent the optimization. In case of views some of the conditions that will cause it are</p>

DISTINCT ON operations</li>
Window functions (OVER clauses)</li>
Set operations (UNION/INTERSECT/EXCEPT)</li>
More complex aggregations</li>
Common Table Expressions that are materialized (either with MATERIALIZED hint or by the decision of the query planner</a>)</li>
Use of VOLATILE functions</li>
And in some cases complex subqueries</li>
</ul>
When planner is not able to inline VIEWs it might lead to the unnecessary performance and resource impact. I.e. each sub query might materialize results separately, leading to the already retrieved rows to be discarded by another part of the query as one of the most common examples.</p>
When using EXPLAIN this is demostated by one of the following options:</p>

Subquery scan on...</code> nodes</li>
Materialization nodes</li>
Separate aggregation/sorting steps before the main execution</li>
</ul>
As mentioned above the window function acts as such a planner barrier.</p>
CREATE VIEW order_insights AS</span></span>
SELECT</span></span>
   o.customer_id,</span></span>
   o.total_amount,</span></span>
   o.created_at,</span></span>
   ROW_NUMBER() OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as order_sequence,</span></span>
   SUM(o.total_amount) OVER (PARTITION BY o.customer_id ORDER BY o.created_at) as running_total</span></span>
FROM orders o;</span></span>
</span>
EXPLAIN</span></span>
SELECT * FROM order_insights</span></span>
WHERE customer_id = 1</span></span>
AND total_amount > 500;</span></span></code></pre>
Giving us easy to spot Subquery scan</code> node.</p>
 Subquery Scan on order_insights  (cost=1.06..1.10 rows=1 width=84)</span></span>
   Filter: (order_insights.total_amount > '500'::numeric)</span></span>
   ->  WindowAgg  (cost=1.06..1.08 rows=1 width=84)</span></span>
         ->  Sort  (cost=1.06..1.06 rows=1 width=44)</span></span>
               Sort Key: o.created_at</span></span>
               ->  Seq Scan on orders o  (cost=0.00..1.05 rows=1 width=44)</span></span>
                     Filter: (customer_id = 1)</span></span></code></pre>Runtime Materialization (Not MATERIALIZED VIEWs)</a>
</h2>
Let's clear up a possible confusion - we're not talking about CREATE MATERIALIZED VIEW here. This is about PostgreSQL's runtime decision to cache view results in memory during query execution. Query planner might use explicit materialization when it needs to reference view results multiple times or prevent repeated expensive computations. The query planner shows this through a Materialize node:</p>
CREATE VIEW</span> customer_summary</span> AS</span></span>
SELECT</span></span>
    customer_id,</span></span>
    COUNT</span>(</span>*</span>) </span>as</span> orders,</span></span>
    SUM</span>(total_amount) </span>as</span> total_spent</span></span>
FROM</span> orders</span></span>
GROUP BY</span> 1</span>;</span></span>
</span>
EXPLAIN </span>SELECT</span></span>
    a</span>.</span>customer_id</span>,</span></span>
    a</span>.</span>total_spent</span>,</span></span>
    b</span>.</span>total_spent</span> as</span> other_customer_spent</span></span>
FROM</span> customer_summary a</span></span>
JOIN</span> customer_summary b </span>ON</span> b</span>.</span>total_spent</span> ></span> a</span>.</span>total_spent</span>;</span></span></code></pre>
Giving the perfect example of materialization.</p>
 Nested Loop  (cost=46.00..200.75 rows=3333 width=68)</span></span>
   Join Filter: (b.total_spent > (sum(orders.total_amount)))</span></span>
   ->  HashAggregate  (cost=23.00..24.25 rows=100 width=44)</span></span>
         Group Key: orders.customer_id</span></span>
         ->  Seq Scan on orders  (cost=0.00..18.00 rows=1000 width=15)</span></span>
   ->  Materialize  (cost=23.00..25.75 rows=100 width=32)</span></span>
         ->  Subquery Scan on b  (cost=23.00..25.25 rows=100 width=32)</span></span>
               ->  HashAggregate  (cost=23.00..24.25 rows=100 width=44)</span></span>
                     Group Key: orders_1.customer_id</span></span>
                     ->  Seq Scan on orders orders_1  (cost=0.00..18.00 rows=1000 width=15)</span></span></code></pre>
Next to the materialization - the materialized VIEWs</strong> (CREATE MATERIALIZED VIEW</code>) are for practical purposes "just another table" they present an ultimate planner barrier. PostgreSQL must scan the materialized data directly, trading query flexibility for performance</p>
Tips for writing inlining friendly VIEWs</a>
</h2>
Well, there's not a single "right" way how to write views. And don't forget - VIEWs should make your database easier to work with, not harder. When designing views, focus on single responsibility - each view should handle one aspect of business logic rather than trying to optimize everything at once. Simple views are more likely to get inlined by PostgreSQL's query planner.</p>
There are several common patterns that prevent inlining</p>

Avoid window functions, distinct and set operations completely</li>
Split complex logic into multiple views if possible</li>
Minimize subquery usage</li>
Consider materialized views for aggregations</li>
</ol>
View dependencies are a critical consideration. Even minor schema changes can trigger cascading view modifications across your database. Instead of creating deep view hierarchies, split complex logic into smaller, composable views. For critical views that many applications depend on, consider versioning (v1, v2) rather than modifying existing ones.</p>
When in doubt, write the plain SQL first, then extract common patterns into views. This helps avoid overengineering and keeps your database schema maintainable.</p>
Inlining is only half the story. The other half is what views are made of in the catalog, and why every column change to the underlying table turns into a drop-and-recreate exercise. That trade-off, and the workarounds, is covered in Strong views on PostgreSQL VIEWs</a>.</p>


DELETEs are difficult
2024-11-23T00:00:00+00:00
Your database is ticking along nicely - until a simple DELETE brings it to its
knees. What went wrong? While we tend to focus on optimizing SELECT and INSERT
operations, we often overlook the hidden complexities of DELETE. Yet, removing
unnecessary data is just as critical. Outdated or irrelevant data can bloat your
database, degrade performance, and make maintenance a nightmare. Worse,
retaining some types of data without valid justification might even lead to
compliance issues.</p>
At first glance, the DELETE command seems straightforward. Even the PostgreSQL
documentation</a> provides
simple examples like:</p>
DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span>
DELETE FROM</span> films;</span></span></code></pre>
These queries might work effortlessly on your development machine, where only a
few hundred records exist. But what happens when you try running a similar
DELETE in production, where datasets are orders of magnitude larger?</p>
In this article, we’ll uncover why DELETE operations
demand careful consideration and explore how to handle them effectively.</p>
What really happens when you DELETE the data?</a>
</h2>
At first glance, a DELETE query might seem straightforward. However, once the query is executed, a series of intricate steps occur:</p>

Row Identification</strong>: Similar to a SELECT operation, the query identifies rows visible to the current transaction (considering MVCC) and checks for locks.</li>
Lock Acquisition</strong>: The database acquires row-level exclusive locks to prevent other operations on the targeted rows.</li>
BEFORE DELETE trigger</strong>: If a BEFORE DELETE trigger is defined, it is executed at this point.</li>
Marking Rows as Deleted</strong>: Instead of being physically removed, the rows are marked as deleted in the current transaction, rendering them invisible to future queries (depending on transaction isolation). If table has large data objects, TOAST table will have to be involved too.</li>
Index Updates</strong>: The corresponding index entries are also marked for deletion (if applicable).</li>
Cascaded Actions</strong>: Cascading operations, such as ON DELETE CASCADE, are performed on related tables.</li>
AFTER DELETE Trigger</strong>: If an AFTER DELETE trigger is defined, it is executed.</li>
Write-Ahead Log (WAL)</strong>: Changes are recorded in the WAL-first at the row level, followed by index-level updates.</li>
</ol>
Only when the transaction is committed do these changes become permanent and visible to transactions starting afterward. However, even at this point, the data is not physically removed</strong>. This is how bloat</strong> is created.</p>
Until the autovacuum</strong> process or a manual VACUUM operation reclaims the space, the “deleted” data remains. This leftover data contributes to bloat, which can degrade query performance over time.</p>
The key question now is whether DELETEs are truly the hardest operation we can subject our database to. The answer? Quite possibly. While UPDATEs come close in complexity, they’re typically designed in ways that make them less challenging:</p>

UPDATEs usually modify only a limited number of columns, reducing the potential number of index updates lower.</li>
Not all UPDATEs trigger a full row (COLD) update, where the old row is marked as dead and a new row is created. With careful table and query design, you can minimise these cases. HOT (Heap-Only Tuple) updates</a>, for instance, are easier to achieve with fixed-length columns.</li>
Unlike DELETEs, UPDATEs don’t trigger cascaded actions - they only involve triggers that are explicitly defined.</li>
</ul>
And then comes AUTOVACUUM</a>
</h2>
When autovacuum kicks in (and you really want it to kick in) - typically triggered by thresholds for the number of dead tuples or changes to the table - a significant amount of work is required to clean them up. Let’s break it down step by step:</p>

The process begins by scanning the table. While it’s not always a full table scan, the autovacuum checks the visibility map and pages where dead tuples might exist. This can happen incrementally, allowing even large tables to be processed - provided your autovacuum settings allow it to run frequently enough.</li>
Each tuple is checked to ensure it’s no longer visible to any active or pending transactions.</li>
Dead tuples that pass the visibility check are physically removed from the table.</li>
Corresponding index entries for the removed tuples are updated.</li>
The now-empty space is marked for reuse in future INSERT or UPDATE operations.</li>
Table statistics are updated to reflect the current state, helping the query planner make better decisions.</li>
The changes, including tuple removals and index updates, are logged in the Write-Ahead Log (WAL) for durability and replication.</li>
If the table has TOASTed data (large objects), the associated TOAST tables are processed.</li>
The visibility map is updated to mark cleaned pages as fully visible again.</li>
Autovacuum resets thresholds to determine when the next vacuum operation should occur.</li>
</ul>
This process continues until it reaches the configured vacuum cost limit (in the case of autovacuum), at which point it pauses or stops. While autovacuum helps keep your database in check, it’s clear that reclaiming dead tuples is no small task - and it underscores why DELETE operations can have lasting effects on database performance.</p>
While the AUTOVACUUM</code> might sounds as a bad news, without it, your database would quickly become bloated with dead tuples, leading to degraded performance, slower queries, increased storage usage, and even the potential for out-of-disk errors as unused space cannot be reclaimed.</p>
Further considerations</a>
</h2>
For what seems like a simple DELETE, a surprising amount of work has already taken place, but the complexity doesn’t stop there. DELETE operations can introduce additional challenges, particularly when replication, resource contention, or the size of the operation comes into play.</p>
In environments with replication to hot standby or replicas, DELETEs become more time-sensitive. The transaction cannot complete until the corresponding WAL (Write-Ahead Log) records have been written to disk on the standby. This is a fundamental requirement for maintaining data consistency in high-availability setups, where at least one standby server is typically involved. Additionally, if the standby is actively serving read operations, it must account for DELETEs before confirming the changes, potentially introducing further delays.</p>
The size of the DELETE operation also plays a critical role. Small DELETEs, such as removing a single row, tend to have minimal impact. However, as the size of the operation grows, so does the volume of WAL records generated. Large DELETEs can overwhelm the system, slowing down transactions and straining the replication process. Standby servers must work harder to process the incoming WAL stream, which can bottleneck performance if their throughput is insufficient.</p>
Resource contention adds yet another layer of complexity, particularly for large DELETEs. Generating WAL records, handling regular transactional workloads, and running background processes can collectively push the system towards I/O saturation. This creates competition for CPU and memory resources, leading to slower operations across the board.</p>
Finally, once the data is marked for deletion, the autovacuum process must eventually step in to physically remove it. This introduces its own set of challenges, as autovacuum must deal with the same resource contention and I/O demands, compounding the overall impact of the initial DELETE operation.</p>
Soft-deletes are not the solution</a>
</h2>
Soft-deletes might seem like an easy way to sidestep the complexities of
traditional DELETE operations. After all, updating a deleted_at</code> field is
straightforward and unlikely to trigger a COLD update. However, soft-deletes are
not a true mechanism for data removal, and they come with their own set of
complications.</p>
While soft-deletes can provide a simple way to implement “undo” functionality,
they raise serious questions about data consistency. For instance, do you only
mark the main entity as deleted, or do you also cascade the status to all
related records in referenced tables? Failing to cascade properly can leave your
database in an inconsistent state, making it difficult to maintain data
integrity.</p>
Soft-deletes also require consideration in your application logic. Every query
must include appropriate filters to exclude “deleted” rows, which can complicate
query design and increase the risk of oversights. One missed filter could expose
data that should no longer be visible, leading to potential security or business
logic issues.</p>
Finally, soft-deletes don’t solve the problem - they do merely postpone it. The
data is still in your database, consuming storage and potentially contributing
to performance degradation over time. Sooner or later, you’ll need to deal with
the actual removal of this data, bringing you back to the same challenges that
DELETEs pose in the first place.</p>
At the time of writing this article we can only speculate how much will support
of temporal PRIMARY KEY and UNIQUE constriants in PostgreSQL 18 will transform
the balances in future. But given the complexity of the feature I wouldn't bet
on it just yet.</p>
Batching is the answer</a>
</h2>
Giving PostgreSQL time to process and catch up with large-scale changes is critical when dealing with operations like DELETEs. The core issue here is the duration and magnitude of the transaction. The shorter the transaction and the fewer changes made, the better PostgreSQL can manage and reconcile those changes.</strong> This principle is universal across all database operations and underscores the importance of minimizing the impact of individual transactions.</p>
While you can optimize certain aspects, like row identification (using indexes, clustering, or similar techniques), larger datasets demand a more strategic approach - batching. For example, deleting 1 million rows in a single transaction is a textbook case of what not to do. Instead, splitting the operation into smaller batches, such as deleting 10,000 rows across 100 iterations, is far more effective.</p>
Will this method be faster than performing one massive DELETE? Likely not, especially if you include a wait time between batches to allow PostgreSQL to handle other workloads. However, the trade-off is worthwhile. By batching, you give PostgreSQL more breathing room to manage changes without overwhelming regular transactional workloads - unless, of course, you’ve scheduled dedicated maintenance time for the operation.</p>
How to Batch DELETEs</a>
</h3>
The easiest way to batch DELETEs is to use a subquery or a Common Table
Expression (CTE)</a> to limit the number of rows affected in each iteration. For
example, instead of executing a bulk DELETE like this:</p>
DELETE FROM</span> films </span>WHERE</span> kind </span><></span> 'Musical'</span>;</span></span></code></pre>
You can break the operation into smaller chunks. Using a query like the
following, you can repeatedly delete rows in manageable batches (for instance,
using \watch in psql to automate iterations):</p>
DELETE FROM</span> films</span></span>
WHERE</span> ctid </span>IN</span> (</span></span>
    SELECT</span> ctid </span>FROM</span> films</span></span>
    WHERE</span> kind </span><></span> 'Musical'</span></span>
    LIMIT</span> 250</span></span>
);</span></span></code></pre>
The use of ctid</code> in this example is PostgreSQL system column which provides a
unique identifier for each row. By selecting ctid</code> values in the subquery, you
can limit the number of rows affected in each iteration. This approach is more
efficient than using LIMIT</code> directly in the main query, as it avoids the need
to re-scan the table for each batch.</p>
If you don't feel comfortable with ctid</code> (which might deserve article on its
own) you can use regular lookup by primary key and LIMIT</code>.</p>
Planning for Autovacuum</a>
</h3>
Batching alone doesn’t directly solve the issue of autovacuum catching up with the changes. You’ll need to plan for that separately. Adjusting autovacuum settings or triggering manual VACUUM</code> and VACUUM ANALYZE</code> runs can help manage bloat created during the DELETE process. However, disabling autovacuum is rarely advisable unless you’ve carefully planned for manual maintenance throughout the batch operations. Skipping this step risks leaving behind performance-impacting bloat that will require even more effort to address later.</p>
Drop whole range of data with partitioning</a>
</h2>
Data that is naturally segmented - for example by time of the creation - makes it an excellent candidate for removal through partitioning. Partitioning allows you to bypass DELETE operations altogether by simply dropping or truncating the relevant partitions. This approach is far more efficient and avoids the overhead of scanning, locking, and marking rows as deleted, effectively eliminating the problem with the bloat.</p>
While partitioning adds some complexity to schema design and query planning, it can provide significant performance benefits for DELETE-heavy workloads, especially when combined with automated partition management.</p>
Conclusion</a>
</h2>
DELETE operations are often a source of unpleasant surprises - not
just by affecting performance and creating bloat, but by striking back at the
times we least expect. To handle them effectively, focus on strategies such as
batching, monitoring autovacuum, or leveraging partitioning for large datasets.
By considering DELETE operations during schema design, you can maintain an
efficient database, reduce maintenance headaches, and ensure it continues to run
smoothly as your data grows.</p>


Text identifiers in PostgreSQL database design
2024-11-09T00:00:00+00:00
Whether you are designing a standalone application or a microservice, you will inevitably encounter the topic of sharing identifiers. Whether it’s URLs of web pages, RESTful API resources, JSON documents, CSV exports, or something else, the identifier of specific resources will be exposed.</p>
/orders/123</span></span>
/products/345/variants/1</span></span></code></pre>
While an identifier is just a number and does not carry any negative connotations, there are valid reasons why you might want to avoid exposing them. These reasons include:</p>

Security and Data Exposure</strong>: Numerical identifiers are sequential and predictable, which can expose information about the underlying data source (e.g., the volume of the data) and provide a basis for ID enumeration.</li>
Privacy and Confidentiality</strong>: Concerns may arise about concealing the volume of referenced data. For example, the number of customers, clients, or orders might be information a business prefers to keep private.</li>
Non-descriptive Nature</strong>: Integers as identifiers can lead to confusion. An ID like 123 does not convey any additional information, making debugging edge cases more challenging.</li>
</ol>
These and other reasons (like SEO optimisation) have led to the increased use of text-based identifiers. Their readability and versatility make them ideal for external data sharing.</p>
However, in database (or data model) design, the advantages of text identifiers are often overshadowed by the problems they introduce. While text identifiers improve interoperability, they frequently come with performance and storage trade-offs. In contrast, integers are naturally faster and more efficient to process, resulting in lower storage requirements and faster indexing, sorting, and searching-tasks that computers are optimised for.</p>
In this article, we will explore scenarios where using text identifiers directly in the database design might seem natural and discuss strategies for using them effectively.</p>
What Makes Text Identifiers Appealing?</a>
</h3>
Let’s be honest-text identifiers are popular for a reason. For humans, they are much more readable and, in some cases, add extra context. (Raise your hand if you don’t appreciate Heroku’s quirky and memorable names!) If chosen carefully, they can also be easier to recall.</p>
Text identifiers can embed additional context. Consider the order number APAC-20241103-8237, which encodes both the region and the order date. Another popular reason for using text identifiers is their uniqueness in distributed environments.</p>
They’re especially handy when people need to interact with them directly. For example, customers copying an order number from an email or support teams discussing an issue benefit from a readable, meaningful identifier. It’s simpler, more intuitive, and less likely to cause headaches when someone tries to recall or share it.</p>
When an Identifier Isn’t Just an Identifier</a>
</h3>
Problems with text identifiers arise when they are used as natural keys in your data or database model. Despite their benefits, text identifiers often make poor primary keys for several reasons:</p>

Context Changes</strong>: The additional context provided by text identifiers will likely change, necessitating updates. Despite assurances to the contrary, changes are inevitable.</li>
Sorting Issues</strong>: Sorting text identifiers can be tricky, particularly with locale-based sorting or when numbers are embedded within the identifier (e.g., order1434 vs. order349).</li>
</ul>
The ultimate issue is efficiency:</p>

Text identifiers typically require more storage space</strong> than numeric ones. Each character in a text field occupies more bytes than a simple integer, leading to larger database sizes and slower performance, especially during indexing or handling large datasets.</li>
Databases are optimised for numeric operations, making searches, joins, and indexing on text fields inherently slower</strong>. This performance gap can significantly affect large datasets, impacting application efficiency.</li>
Text identifiers complicate managing relationships between tables</strong>. The additional storage requirements affect not only the source table but also all referencing entities. Multiply this increased storage by the number of referencing tables, and you’ll get a sense of the overall impact.</li>
</ul>
As databases grow, these issues become more pronounced. The combination of increased storage needs and slower operations contributes to database bloat</strong>, which can degrade system performance. Additionally, bloated indexes can mislead the query planner into making suboptimal choices, such as favouring sequential scans over index scans, complicating regular operations further.</p>
Aren't UUIDs the Solution to All This?</a>
</h3>
It depends. While UUIDs offer numerical representation and are excellent for unique key generation across distributed systems, they are not always the best choice:</p>

In comparison to BIGINT, there are few real-world scenarios that necessitate the full range of UUIDs. Premature optimisation most often isn’t worthwhile for most solutions.</li>
UUIDs’ 16-byte (128-bit) storage size can be less efficient than many text identifiers. Even BIGINT, at 8 bytes (64 bits), is more storage-efficient. This inefficiency extends to indexes, joins, and other operations.</li>
</ul>

A matter of opinion: UUIDs are plain ugly.</li>
</ul>
Moreover, different versions of UUIDs offer varying benefits. For example, UUIDv1 includes a timestamp component, making it somewhat sortable, while UUIDv4 is entirely random. Even sortable UUIDs may not offer significant advantages over more streamlined options like BIGINT or carefully structured text identifiers.</p>
In most cases, the trade-offs in performance and readability make UUIDs less appealing unless their globally unique nature is essential across distributed systems.</p>
Real-Life Examples</a>
</h3>
Let’s move beyond theory and explore practical examples:</p>
CREATE TABLE</span> sessions</span> (</span></span>
    token </span>TEXT PRIMARY KEY</span>,</span></span>
    user_id </span>INT NOT NULL REFERENCES</span> users(user_id),</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> products</span> (</span></span>
    sku </span>TEXT PRIMARY KEY</span>,</span></span>
    label </span>TEXT NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> documents</span> (</span></span>
    document_id </span>TEXT PRIMARY KEY</span>,</span></span>
    ...</span></span>
);</span></span></code></pre>
In these cases, using text identifiers as primary keys might seem logical because:</p>

They serve as natural keys for the entity.</li>
</ul>

They are likely propagated outside the service scope, hence needing storage anyway.</li>
They are not expected to change.</li>
The storage impact for a single table seems negligible.</li>
</ul>
However, this logic falters under scrutiny. Text identifiers often propagate beyond the service scope, leading to pressure to update them. Updating primary keys is no trivial matter. Consider:</p>

While SKUs ideally never change, real-world scenarios like rebranding, product consolidation, or supplier changes can necessitate updates. Despite their appeal, SKUs are poor primary key candidates.</li>
Randomly generated text identifiers (like session tokens) will… TBD.</li>
</ul>
The real issue arises when referencing these identifiers across tables:</p>
CREATE TABLE</span> session_logs</span> (</span></span>
    log_id </span>BIGINT GENERATED ALWAYS AS IDENTITY</span>,</span></span>
    token </span>TEXT NOT NULL REFERENCES sessions</span>(token),</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> product_reviews</span> (</span></span>
    review_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    product_sku </span>TEXT NOT NULL REFERENCES</span> products(sku),</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> customer_orders</span> (</span></span>
    order_id </span>TEXT PRIMARY KEY</span>,</span></span>
    customer_id </span>TEXT NOT NULL REFERENCES</span> customers(customer_id),</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> document_revisions</span> (</span></span>
    revision_id </span>INTEGER GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    document_id </span>TEXT NOT NULL REFERENCES</span> documents(document_id),</span></span>
    ...</span></span>
);</span></span></code></pre>
In such scenarios, text identifiers can become problematic. Firstly, you lose the ability to modify them</strong>. Secondly, the increased storage requirements</strong> become evident. For instance, an additional 100 bytes per reference across a million records results in an extra 100 MB of storage for just one reference.</p>
However, storage isn’t the biggest concern; indexing</strong> is. In PostgreSQL, indexing text fields - whether as primary or foreign keys - requires significantly more space than indexing numeric fields. This leads to bloated indexes, slower lookups, and more fragmented operations, particularly in queries relying on foreign key relationships. As a result, the query planner might resort to full table scans instead of index scans, further degrading performance.</p>
These performance issues often remain undetected in development or testing environments but can cause significant disruptions in production.</p>
Reintroducing Text Identifiers Effectively</a>
</h3>
The goal of this post is not to discourage the use of text identifiers entirely but to highlight why they are unsuitable as primary keys. Here are some strategies to handle them effectively:</p>
1. Introduce a Surrogate Key</a>
</h4>
One straightforward solution is to introduce a surrogate primary key, replacing references to text identifiers:</p>
CREATE TABLE</span> products</span> (</span></span>
    product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span>
    sku </span>TEXT NOT NULL</span>,</span></span>
    label </span>TEXT NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE INDEX</span> products_by_sku</span> ON</span> products(sku);</span></span>
</span>
CREATE TABLE</span> product_reviews</span> (</span></span>
    review_id </span>SERIAL PRIMARY KEY</span>,</span></span>
    product_id </span>INT REFERENCES</span> products(product_id),</span></span>
    ...</span></span>
);</span></span></code></pre>
This approach maintains efficient retrieval by SKU via an index.</p>
2. Use Mapping Tables for Greater Flexibility</a>
</h4>
Mapping tables allow you to:</p>
* 	Update identifiers without affecting the parent entity.</span></span>
* 	Maintain a history of text identifiers linked to a specific entity.</span></span></code></pre>CREATE TABLE</span> products</span> (</span></span>
    product_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span>
    label </span>TEXT NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> product_skus</span> (</span></span>
    product_sku_id </span>INT GENERATED BY DEFAULT AS IDENTITY</span>,</span></span>
    product_id </span>INT REFERENCES</span> products(product_id),</span></span>
    sku </span>TEXT NOT NULL</span>,</span></span>
    created_at </span>TIMESTAMPTZ DEFAULT</span> CURRENT_TIMESTAMP,</span></span>
    deleted_at </span>TIMESTAMPTZ</span></span>
);</span></span>
</span>
CREATE UNIQUE INDEX</span> unique_product_skus</span> ON</span> product_skus (product_id, sku) </span>WHERE</span> deleted_at </span>IS NULL</span>;</span></span></code></pre>
This approach accommodates changing SKUs without compromising data integrity. A related use case could be linking a mapping table to a product variant:</p>
CREATE TABLE</span> product_variants</span> (</span></span>
    variant_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span>
    product_id </span>INT NOT NULL REFERENCES</span> products(product_id),</span></span>
    sku </span>TEXT NOT NULL</span>,</span></span>
    name TEXT NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span></code></pre>3. Decompose the Text Identifier’s Meaning</a>
</h4>
Text identifiers often embed meaningful information, such as regions, dates, or categories. By decomposing the identifier, you can store this contextual data separately, improving both flexibility and performance.</p>
For example, instead of storing an order identifier like ORD-EMEA-00789 directly, you can design a more robust schema:</p>
CREATE TABLE</span> orders</span> (</span></span>
    order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span>
    region_id </span>INT NOT NULL REFERENCES</span> regions(region_id),</span></span>
    order_date </span>DATE NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span>
</span>
CREATE TABLE</span> regions</span> (</span></span>
    region_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span>
    name TEXT NOT NULL</span>,</span></span>
    code </span>TEXT NOT NULL</span>,</span></span>
    ...</span></span>
);</span></span>
</span>
INSERT INTO</span> regions (</span>name</span>, code) </span>VALUES</span></span>
(</span>'Europe, the Middle East, and Africa'</span>, </span>'EMEA'</span>),</span></span>
(</span>'Asia Pacific'</span>, </span>'APAC'</span>);</span></span>
</span>
To</span> generate a user</span>-</span>friendly order </span>number</span>, you can </span>create</span> a </span>function</span>:</span></span>
</span>
CREATE OR REPLACE FUNCTION</span> get_order_number</span>(p_order_id </span>INT</span>)</span></span>
RETURNS TEXT AS</span> $$</span></span>
DECLARE</span></span>
    v_region_code </span>TEXT</span>;</span></span>
    v_formatted_order_id </span>TEXT</span>;</span></span>
    v_order_number </span>TEXT</span>;</span></span>
BEGIN</span></span>
    SELECT</span> r</span>.</span>code</span> INTO</span> v_region_code</span></span>
    FROM</span> orders o</span></span>
    JOIN</span> regions r </span>ON</span> o</span>.</span>region_id</span> =</span> r</span>.</span>region_id</span></span>
    WHERE</span> o</span>.</span>order_id</span> =</span> p_order_id;</span></span>
</span>
    IF NOT</span> FOUND </span>THEN</span></span>
        RETURN NULL</span>;</span></span>
    END IF</span>;</span></span>
</span>
    v_formatted_order_id :</span>=</span> TO_CHAR(p_order_id, </span>'FM00000'</span>);</span></span>
    v_order_number :</span>=</span> 'ORD-'</span> ||</span> v_region_code </span>||</span> '-'</span> ||</span> v_formatted_order_id;</span></span>
</span>
    RETURN</span> v_order_number;</span></span>
END</span>;</span></span>
$$ </span>LANGUAGE</span> plpgsql;</span></span></code></pre>
This method enables you to store and retrieve structured data efficiently while still providing a readable and meaningful identifier for external use.</p>
4. Reversible Text IDs</a>
</h4>
For scenarios requiring enhanced security or privacy, where identifiers should not be easily enumerable or predictable, you can use reversible text-based IDs. These allow you to present users with seemingly random text representations while maintaining efficient numerical storage internally.</p>
Sqids</a></strong> is a project that can help achieve this. It generates URL-friendly unique identifiers from numbers and can encode multiple numerical identifiers into a single string. Here’s an example using the Sqids project:</p>
[42] -> JgaEBgznCpUZo3Kk</span></span>
[42, 430004] -> lTiYlvsGkh59m1PQ</span></span></code></pre>
The generated identifiers are reversible with knowledge of the shared alphabet, allowing you to decode requests without hitting the database, which can be beneficial in high-throughput environments. This technique is particularly useful for user, account, or session identifiers, balancing the need for security with operational efficiency.</p>
However, it’s crucial to remember that this is not a substitute for robust security practices. Proper authentication and authorisation mechanisms are still necessary to secure your application.</p>
Use text identifiers where they genuinely help, and stick to numeric primary keys where they don't - that keeps the schema both efficient and maintainable.</p>
5. Leverage Generate columns</a>
</h4>
In edge case scenarios, where keeping text identifiers with embedded context is necessary, generated columns</strong> in PostgreSQL can be a valuable feature. Since version 12, PostgreSQL allows defining columns whose values are automatically computed from other columns in the table. This ensures consistency without manual intervention.</p>
For example, you can define a function to handle the formatting logic:</p>
CREATE OR REPLACE FUNCTION</span> get_formatted_order_id</span>(p_region_id </span>INT</span>, p_order_id </span>INT</span>)</span></span>
RETURNS TEXT AS</span> $$</span></span>
DECLARE</span></span>
    v_region_code </span>TEXT</span>;</span></span>
BEGIN</span></span>
    v_region_code :</span>= CASE</span> p_region_id</span></span>
        WHEN</span> 1</span> THEN</span> 'EMEA'</span></span>
        WHEN</span> 2</span> THEN</span> 'APAC'</span></span>
        ELSE</span> 'OTHER'</span></span>
    END</span>;</span></span>
</span>
    RETURN</span> 'ORD-'</span> ||</span> v_region_code </span>||</span> '-'</span> ||</span> LPAD(p_order_id::</span>TEXT</span>, </span>5</span>, </span>'0'</span>);</span></span>
END</span>;</span></span>
$$ </span>LANGUAGE</span> plpgsql IMMUTABLE;</span></span></code></pre>
The function is marked IMMUTABLE because PostgreSQL requires generated columns to use only immutable functions-those guaranteed to return the same result for the same input every time.</p>
CREATE TABLE</span> orders</span> (</span></span>
    order_id </span>INT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY</span>,</span></span>
    region_id </span>INT NOT NULL</span>,</span></span>
    order_date </span>DATE NOT NULL</span>,</span></span>
    formatted_order_id </span>TEXT GENERATED ALWAYS AS</span> (</span></span>
        get_formatted_order_id(region_id, order_id)</span></span>
    ) STORED</span></span>
);</span></span></code></pre>
This setup ensures that formatted_order_id</code> is automatically updated whenever region_id</code> or order_id</code> changes. The column is physically stored, enhancing read performance for frequently queried data.</p>
Using generated columns can simplify maintaining consistency in derived values like formatted text identifiers. However, be mindful of their characteristics, such as:</p>

Automated Updates</strong>: The system automatically recomputes the column’s value when the referenced columns change.</li>
Immutability Requirements</strong>: Only immutable functions can be used, ensuring reliable and consistent computation.</li>
</ul>
Wrapping up</a>
</h3>
Text identifiers will stay with us, and that's great. They’re human-readable, memorable, and can pack a lot of meaningful context into a simple string. They make external interactions smoother, whether it’s a customer referencing an order number or a support team tracing an issue. They even add a bit of charm and personality to otherwise sterile identifiers.</p>
However it's important to keep their usage in control. The nice rule I've heard is</p>

Use text identifiers when communicating externally</strong> - for example, in URLs or API responses</li>
Internally, always rely on numerical IDs</strong> - surrogate keys like INT or BIGINT (even UUID if you go for planet dominanance) where necessary, to maintain database efficiency and integrity.</li>
</ul>
This approach allows you to leverage the strengths of text identifiers for external communication while keeping your database optimised for performance and scalability. By storing text identifiers in dedicated fields and using numeric primary keys for internal operations, you achieve a the right balance and do best to maintain the system performance.</p>


We need to talk about ENUMs
2024-09-04T00:00:00+00:00
Designing a database schema, whether for a new application or a new feature, always raises a lot of questions. The choices you make can have a big impact on how well your database performs and how easy it is to maintain and scale. Whether you’re just getting started with PostgreSQL or consider yourself a seasoned pro, it’s easy to rely on old habits or outdated advice. In this article, I want to take a fresh look at one of those topics that often sparks debate: the use of ENUMs in PostgreSQL.</p>
I have to admit, not so long ago, I would advise "don't use ENUMs" without thinking about it too much. Relying only on random articles and some personal but outdated experience, this had been my go-to answer for some years. And while there were several limitations of ENUMs in PostgreSQL in the (distant) past, the support has improved a long time ago.</p>
The improvements are so long-standing that:</p>

PostgreSQL 9.1 (released in 2011) introduced the option to add new values to ENUMs without a table rewrite</strong>. From what I can say this fact alone remained the source of biggest misconception when thinking about ENUMs.</li>
Renaming of values</strong> was added in version 10 (2017).</li>
The ability to ALTER TYPE ... ADD VALUE</code> in a transaction block was introduced in PostgreSQL 12 (2019).</li>
</ul>
And new features coming (at the time of writing this article) with PostgreSQL 17</strong> allow the use of newly added values within the same transaction block (previously not possible without an explicit commit).</p>
Therefore, I want to correct even myself and say—let's give ENUMs another chance. This article will go into detail and help us correctly decide when it makes sense to use them or not.</p>
How are ENUMs Implemented?</a>
</h2>
Every stored ENUM value occupies 4 bytes on disk. This is the size of the OID, representing the actual ENUM value, stored as a row within the pg_enum</code> table. You can see this yourself by querying the table directly, but it will come by default with no data defined.</p>
# select * from pg_catalog.pg_enum;</span></span>
</span>
 oid | enumtypid | enumsortorder | enumlabel</span></span>
-----+-----------+---------------+-----------</span></span>
(0 rows)</span></span></code></pre>
But once you define a new ENUM data type, using:</p>
CREATE TYPE</span> order_status</span> AS</span> ENUM (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>);</span></span></code></pre>
You will get the actual OIDs.</p>
# select * from pg_catalog.pg_enum;</span></span>
</span>
  oid   | enumtypid | enumsortorder | enumlabel</span></span>
--------+-----------+---------------+------------</span></span>
 211018 |    211016 |             1 | new</span></span>
 211020 |    211016 |             2 | pending</span></span>
 211022 |    211016 |             3 | processing</span></span>
 211024 |    211016 |             4 | shipped</span></span>
 211026 |    211016 |             5 | delivered</span></span></code></pre>
The table straightforwardly identifies both the type and individual values, but also the enumsortorder</code>, which is the foundation for BEFORE/AFTER</code> new value(s) definition.</p>
ALTER TYPE</span> order_status </span>ADD VALUE</span> 'cancelled'</span> BEFORE</span> 'shipped'</span>;</span></span></code></pre>
Resulting in:</p>
  oid   | enumtypid | enumsortorder | enumlabel</span></span>
--------+-----------+---------------+------------</span></span>
 211018 |    211016 |             1 | new</span></span>
 211020 |    211016 |             2 | pending</span></span>
 211022 |    211016 |             3 | processing</span></span>
 211024 |    211016 |             4 | shipped</span></span>
 211026 |    211016 |             5 | delivered</span></span>
 211027 |    211016 |           3.5 | cancelled</span></span></code></pre>
To confirm the actual storage size of the ENUM value, you can use a sample table, the previously defined type order_status</code>, a single data row, and the pg_column_size</code> function.</p>
CREATE TABLE</span> orders</span> (</span></span>
    id </span>SERIAL PRIMARY KEY</span>,</span></span>
    status</span> order_status </span>NOT NULL</span>,</span></span>
    order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE</span></span>
);</span></span>
</span>
INSERT INTO</span> orders (</span>status</span>, order_date) </span>VALUES</span></span>
(</span>'new'</span>, </span>'2024-09-01'</span>);</span></span></code></pre>
This gives you confirmation of the actual 4-byte storage.</p>
# SELECT id, status, pg_column_size(status) AS status_size</span></span>
</span>
 id | status | status_size</span></span>
----+--------+-------------</span></span>
  1 | new    |           4</span></span></code></pre>
Returning to the pg_enum</code> table, the structure of the table provides other important clues:</p>

enumlabel</code> is of type name</code>, which is effectively a 63-byte varchar for storing system identifiers.</li>
The UNIQUE CONSTRAINT pg_enum_typid_label_index</code> prevents us from creating two values with the same name.</li>
However, given the nature of VARCHAR</code>, it is case-sensitive, allowing you to define new</code> and NEW</code> as two distinct values.</li>
</ul>
Life with ENUMs</a>
</h2>
Now that we understand how ENUMs are implemented, let's review the lifecycle of such a data type in a typical database.</p>
As demonstrated above, creation of new ENUMs</strong> is simple, and since version 12 (and expanded in 17), it's possible even in transaction blocks, making it easier to work with database migration tools without needing to explicitly worry about transaction management.</p>
Adding new values</strong> does not require a table rewrite and can be done without further hesitation. You can also safely rename existing values</strong>. Similarly, you can use ENUMs in arrays, indexes, or compare them as needed (based on their order).</p>
ALTER TYPE name ADD VALUE</span> [ IF NOT EXISTS ] new_enum_value [ { BEFORE | AFTER } neighbor_enum_value ]</span></span>
</span>
ALTER TYPE name</span> RENAME </span>VALUE</span> existing_enum_value </span>TO</span> new_enum_value</span></span></code></pre>
It's also important to mention that ENUMs allow you to use a default value, like this:</p>
status order_status NOT NULL DEFAULT 'new'</span></span></code></pre>
Unfortunately, this is where the flexibility ends at the moment. There's no easy way to remove or re-order existing values (please, don't modify pg_enum</code>—or any system tables, for that matter). The topic of removing existing ENUM value(s) can easily open up a whole discussion about the best way to 'deprecate' and drop a value over time. The only suitable way to perform these operations is by creating a new type and altering the column. Please be aware of the ways described in How not to change PostgreSQL column type</a></strong> will apply in this scenario.</p>
Using CHECK Constraints as an Alternative to ENUMs</a>
</h2>
While ENUMs are a powerful tool in PostgreSQL for enforcing a fixed set of values in a column, they come with the limitations mentioned above. An alternative approach to achieving similar functionality is to use a CHECK constraint with a TEXT column. This method offers more flexibility when managing the allowed values, especially in scenarios where the set of valid values might change frequently.</p>
A CHECK constraint can be applied to a column to enforce that its value must be one of a predefined set of values. Here’s how you can define a table using a CHECK constraint as an alternative to an ENUM:</p>
CREATE TABLE</span> orders</span> (</span></span>
    id </span>SERIAL PRIMARY KEY</span>,</span></span>
    status TEXT NOT NULL</span>,</span></span>
    order_date </span>DATE NOT NULL DEFAULT</span> CURRENT_DATE,</span></span>
    CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>))</span></span>
);</span></span></code></pre>
Modification of the constraints is possible, but you will always have to DROP CONSTRAINT</code> and then ADD CONSTRAINT</code>.</p>
ALTER TABLE</span> orders </span>DROP CONSTRAINT</span> orders_status_check;</span></span>
</span>
ALTER TABLE</span> orders </span>ADD CONSTRAINT</span> orders_status_check </span>CHECK</span> (</span>status IN</span> (</span>'new'</span>, </span>'pending'</span>, </span>'processing'</span>, </span>'shipped'</span>, </span>'delivered'</span>, </span>'returned'</span>));</span></span></code></pre>
While this operation is not as 'heavyweight' as changing the data type, please be aware that an ACCESS EXCLUSIVE</strong> lock will be acquired for the entire table—meaning no other operations can be performed on the table. With a growing table size and increasing concurrency, this might still lead to application-level downtime.</p>
Another limitation is the lack of direct support for sorting, meaning you will have to implement it manually in every query or abstract the sorting detail using a view.</p>
SELECT</span> id, </span>status</span>, order_date</span></span>
FROM</span> orders</span></span>
ORDER BY</span></span>
    ARRAY_POSITION(</span></span>
        ARRAY</span>['new', 'pending', 'processing', 'shipped', 'delivered', 'returned'],</span></span>
        status</span></span>
    );</span></span></code></pre>Reference Tables as the Real Alternative</a>
</h2>
It might be surprising, but one of the motivations for using either ENUMs or CHECK constraints is to avoid additional JOINs. It's no surprise that JOINs are a source of pain for many people who struggle with SQL, and therefore they try to limit their use. While there might be negligible impact on performing such an operation, there are no other reasons not to use reference tables—it’s not storage (unless you over-optimise with smallint</code> and extremely large datasets) and definitely not operational considerations.</p>
In fact, reference tables are often better in most cases. They not only address all the use cases supported by both ENUMs and CHECK constraints, but they can also resolve the limitations of these methods.</p>
Creating and Using a Reference Table</a>
</h3>
Let’s create a reference table similar to the ENUM data type we’ve discussed above:</p>
CREATE TABLE</span> order_statuses</span> (</span></span>
    status_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    status_name </span>TEXT NOT NULL</span></span>
);</span></span>
</span>
CREATE UNIQUE INDEX</span> unique_status_name</span> ON</span> order_statuses (</span>LOWER</span>(status_name));</span></span>
</span>
INSERT INTO</span> order_statuses (status_name) </span>VALUES</span></span>
(</span>'new'</span>),</span></span>
(</span>'pending'</span>),</span></span>
(</span>'processing'</span>),</span></span>
(</span>'shipped'</span>),</span></span>
(</span>'delivered'</span>),</span></span>
(</span>'cancelled'</span>);</span></span></code></pre>Updating the Orders Table to Use the Reference Table</a>
</h3>
To use the reference table, you’ll need to alter your existing orders</code> table. This involves dropping the current status</code> column if it’s using an ENUM or CHECK constraint, and replacing it with a foreign key that references the order_statuses</code> table:</p>
ALTER TABLE</span> orders </span>DROP</span> COLUMN </span>IF EXISTS status</span>;</span></span>
</span>
ALTER TABLE</span> orders </span>ADD</span> COLUMN status_id </span>INT</span>,</span></span>
ADD CONSTRAINT</span> fk_order_status</span></span>
FOREIGN KEY</span> (status_id) </span>REFERENCES</span> order_statuses(status_id);</span></span></code></pre>Impact of Using Reference Tables</a>
</h3>
While reference tables do not provide anything for free, such as ENUM ordering or simple CHECK constraints, they offer far greater flexibility. Here are some examples of what you can achieve with reference tables:</p>

Custom Sorting Logic:</strong> You can implement specific sorting by adding an additional sort_order</code> column to the order_statuses</code> table. This allows you to easily change the order of statuses without altering the schema.</li>
Deprecating Values:</strong> By adding a deleted_at</code> column (or a similar field), you can deprecate certain status values. Combined with a trigger, you can prevent these deprecated values from being used in new data while maintaining historical records.</li>
Renaming Values:</strong> Unlike ENUMs, renaming a status is straightforward - simply update the value in the order_statuses</code> table.</li>
Internationalisation:</strong> If your application needs to support multiple languages, you can extend the order_statuses</code> table with additional columns for different languages or even create a separate lookup table that stores translations.</li>
Additional Metadata:</strong> Reference tables can store extra information about each status, such as descriptions, colours (for UI purposes), or associated icons, which can be particularly useful in applications with rich user interfaces.</li>
</ul>
Having mentioned all advantages of using Reference Tables, it's also important to mention the fact it's not straightforward to set the default values for such a references. Definitely not in obvious way which ENUMs provide. The real alternative is to create table/column without default value and alter it specifically for given database using ALTER COLUMN ... SET DEFAULT</code></p>
ALTER TABLE</span> orders</span></span>
ALTER</span> COLUMN status_id </span>SET DEFAULT</span> (</span>SELECT</span> status_id </span>FROM</span> order_statuses </span>WHERE</span> status_name </span>=</span> 'new'</span>);</span></span></code></pre>Performance Considerations: ENUMs, CHECK Constraints, and Reference Tables</a>
</h2>
Thanks to their internal representation as OIDs, ENUMs offer predictable performance and indexing strategies, often outperforming text-based CHECK constraints. Since ENUM comparisons rely on integers, they tend to be faster than string comparisons required for CHECK constraints. Reference tables, on the other hand, inherently require a JOIN, which introduces the overhead of accessing two tables and potentially working with multiple indexes.</p>
While I initially planned to publish benchmarks comparing the performance of all three approaches, I decided not to include specific numbers. It’s true that ENUMs consistently performed the fastest, and reference tables were somewhat slower due to the necessary JOIN. However, as with most benchmarks, the scenarios were artificial. The actual performance difference, though measurable, is unlikely to be significant in practical applications. In any reasonably complex query, other factors are far more likely to impact performance than whether you use ENUMs or not.</p>
Conclusion: Use ENUMs Where They Make Sense</a>
</h2>
As with many decisions in software development, the use of ENUMs comes down to "it depends." This article has provided clarity on how ENUMs work, where their limitations lie, and how to make an informed decision about when to use them.</p>
To recap:</p>

ENUMs</strong> are ideal for small, static sets of values that require strict type enforcement and minimal changes, preferably data types that are not directly tied to the business logic.</li>
CHECK constraints</strong> offer more flexibility than ENUMs, making them a good choice when the allowed values might change, but the data type remains simple, and there's no</li>
Reference tables</strong> provide the most flexibility and scalability, making them the go-to choice for dynamic or complex value sets and when you want to avoid the limitations of ENUMs and CHECK constraints.</li>
</ul>
But in the end, no plan ever survives first contact with reality. It’s essential to revisit and reconsider your decisions as your application evolves, ensuring that your database schema continues to serve the needs of your project as effectively as possible.</p>
And while it's not meant as advice - personally I'm still going to gravitate towards the advice "don't use ENUMs" though. I'm yet to see value-based field that does not change over time.</em></p>


Beyond Simple Upserts with MERGE in PostgreSQL
2024-08-25T00:00:00+00:00
Understanding how comfortable someone is with databases and SQL often comes down to the features they use. In PostgreSQL, one such feature that distinguishes more advanced users is the MERGE</code> command, introduced in version 15 and expanded in version 17 (in beta at the time of writing this article). Before MERGE</code>, developers typically relied on INSERT ... ON CONFLICT DO UPDATE</code> for upserts—a method introduced in PostgreSQL 9.5 that has since become a staple in many developers' toolkits.</p>
While ON CONFLICT</code> offers a straightforward solution for simple upsert scenarios, it can quickly become limiting as business logic grows in complexity. This is where the MERGE</code> command excels. Introduced in the SQL:2003 standard, MERGE</code> allows for more sophisticated data synchronisation tasks by combining multiple operations—such as conditional inserts, updates, and deletes—into a single, atomic statement.</p>
In this article, we’ll explore the capabilities of the MERGE</code> command, comparing it with traditional upsert methods and examining how it can streamline database operations. Through practical examples, we'll illustrate how MERGE</code> can simplify even the most complex workflows, making it an indispensable tool for developers working with PostgreSQL.</p>
Hands-on Example: Managing a Score Points System for a Mobile Game</a>
</h2>
Imagine you’re managing a tournament score system for an online game, where players earn special status by participating in weekly tournaments. The simple business logic would be:</p>

Players can earn score points by participating in the tournament.</li>
Players must participate in all tournaments to maintain their score (i.e., maintain the streak).</li>
Players who participate in a second consecutive tournament can achieve the status of veteran</code> (if they reach 1,000 score points) or gain the status of star</code> (if they reach 100 or more score points).</li>
</ul>
The score points are updated weekly, based on the outcome of the previous week's participation.</p>
We will start with a simple database schema consisting of a table tournament_scores</code> that tracks (as the name suggests) only active users.</p>
CREATE TABLE</span> tournament_scores</span> (</span></span>
    player_id </span>INT GENERATED ALWAYS AS IDENTITY PRIMARY KEY</span>,</span></span>
    player_name </span>TEXT UNIQUE NOT NULL</span>,</span></span>
    score </span>INT NOT NULL DEFAULT</span> 0</span>,</span></span>
    status TEXT NOT NULL DEFAULT</span> 'newbie'</span> CHECK</span> (</span>status IN</span> (</span>'newbie'</span>, </span>'veteran'</span>, </span>'star'</span>))</span></span>
);</span></span></code></pre>
And populate it with some sample data:</p>
INSERT INTO</span> tournament_scores (player_name, score, </span>status</span>)</span></span>
VALUES</span></span>
    (</span>'PlayerOne'</span>, </span>900</span>, </span>'newbie'</span>),   </span>-- Regular player, close to Veteran promotion</span></span>
    (</span>'PlayerTwo'</span>, </span>1200</span>, </span>'veteran'</span>), </span>-- Already a Veteran player</span></span>
    (</span>'PlayerThree'</span>, </span>300</span>, </span>'newbie'</span>); </span>-- Regular player with a lower score</span></span></code></pre>Upsert using ON CONFLICT</code></a>
</h2>
When activity data is received, there are multiple ways to process it. ON CONFLICT</code> is used here for demonstration purposes.</p>
INSERT INTO</span> tournament_scores (player_name, score)</span></span>
VALUES</span></span>
    (</span>'PlayerOne'</span>, </span>50</span>),    </span>-- Add points for PlayerOne</span></span>
    (</span>'PlayerTwo'</span>, </span>120</span>),   </span>-- Add points for PlayerTwo</span></span>
    (</span>'PlayerFour'</span>, </span>70</span>)    </span>-- New player without an account, PlayerFour</span></span>
ON</span> CONFLICT (player_name)</span></span>
DO </span>UPDATE SET</span></span>
    score </span>=</span> tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>,</span></span>
    status = CASE WHEN</span> (</span>tournament_scores</span>.</span>score</span> +</span> EXCLUDED</span>.</span>score</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span>;</span></span></code></pre>
This performs the basic requirements, updating the scores and possibly evaluating the status for existing tournament users. To remove users who no longer participated (and hence broke the streak) and apply other conditional logic, you would need to use separate statements.</p>
Handling Upserts with MERGE</code></a>
</h2>
Now, let’s introduce the MERGE</code> command and implement the same logic as above.</p>
MERGE INTO</span> tournament_scores ts</span></span>
USING</span> (</span></span>
    VALUES</span></span>
        (</span>'PlayerOne'</span>, </span>50</span>),    </span>-- Add points for PlayerOne</span></span>
        (</span>'PlayerTwo'</span>, </span>120</span>),   </span>-- Add points for PlayerTwo</span></span>
        (</span>'PlayerFour'</span>, </span>70</span>)    </span>-- New player without an account, PlayerFour</span></span>
) </span>AS</span> v(player_name, score_added)</span></span>
ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span>
WHEN MATCHED THEN</span></span>
    UPDATE SET</span></span>
      score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span>
      status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span>
WHEN NOT MATCHED THEN</span></span>
    INSERT</span> (player_name, score)</span></span>
    VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre>
Here, we define the table tournament_scores</code> as the target</strong> and the list of VALUES as the source</strong>. Using a conditional clause, we define the logic for both matched and unmatched entries, giving us both INSERT and UPDATE paths.</p>
The evaluation paths in this case are called when_clauses</code>. The MERGE</code> statement allows you to specify multiple clauses with different conditions, for example, allowing you to award users star</code> status if they gain more than 100 score points within a given tournament.</p>
MERGE INTO</span> tournament_scores ts</span></span>
USING</span> (</span></span>
    VALUES</span></span>
        (</span>'PlayerOne'</span>, </span>50</span>),    </span>-- Add points for PlayerOne</span></span>
        (</span>'PlayerTwo'</span>, </span>120</span>),   </span>-- Add points for PlayerTwo</span></span>
        (</span>'PlayerFour'</span>, </span>70</span>)    </span>-- New player without an account, PlayerFour</span></span>
) </span>AS</span> v(player_name, score_added)</span></span>
ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span>
WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span>
    UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span>
WHEN MATCHED THEN</span></span>
    UPDATE SET</span></span>
      score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>,</span></span>
      status = CASE WHEN</span> (</span>ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>) </span>></span> 1000</span> THEN</span> 'veteran'</span> ELSE</span> 'newbie'</span> END</span></span>
WHEN NOT MATCHED THEN</span></span>
    INSERT</span> (player_name, score)</span></span>
    VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>);</span></span></code></pre>
Here’s a summary of how it works:</p>

The MERGE</code> command evaluates the when_clauses</code> in the order they are written.</li>
If a row matches the condition specified in a clause, the action defined in that clause is performed, and the row is no longer eligible to be matched against subsequent when_clauses</code>.</li>
</ol>
While we introduced multiple evaluation paths, the MERGE</code> command in PostgreSQL goes beyond that and expands the WHEN NOT MATCHED</code> clause, which effectively becomes WHEN NOT MATCHED [BY TARGET]</code>. PostgreSQL allows you to specify WHEN NOT MATCHED BY SOURCE</code> to perform the necessary merge statement for the data not present in the source table.</p>
Handling DELETEs with MERGE</code></a>
</h2>
The functionality completely missing from regular upserts with ON CONFLICT</code> is the ability to delete missing entries. In our sample scenario, we want to penalise players who broke their streak and did not participate in last week's tournament by effectively deleting their entries from the target table.</p>
MERGE INTO</span> tournament_scores ts</span></span>
USING</span> (</span></span>
    VALUES</span></span>
        (</span>'PlayerOne'</span>, </span>50</span>),    </span>-- Add points for PlayerOne</span></span>
        (</span>'PlayerTwo'</span>, </span>120</span>),   </span>-- Add points for PlayerTwo</span></span>
        (</span>'PlayerFour'</span>, </span>70</span>)    </span>-- New player without an account, PlayerFour</span></span>
) </span>AS</span> v(player_name, score_added)</span></span>
ON</span> ts</span>.</span>player_name</span> =</span> v</span>.</span>player_name</span></span>
WHEN MATCHED AND</span> v</span>.</span>score_added</span> ></span> 100</span> THEN</span></span>
    UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span>, </span>status =</span> 'star'</span></span>
WHEN MATCHED THEN</span></span>
    UPDATE SET</span> score </span>=</span> ts</span>.</span>score</span> +</span> v</span>.</span>score_added</span></span>
WHEN NOT MATCHED THEN</span></span>
    INSERT</span> (player_name, score)</span></span>
    VALUES</span> (</span>v</span>.</span>player_name</span>, </span>v</span>.</span>score_added</span>)</span></span>
WHEN NOT MATCHED BY</span> SOURCE </span>THEN</span></span>
    DELETE</span>;</span></span></code></pre>
The introduction of the DELETE</code> merge operation complements all possible MERGE</code> outcomes:</p>

The first is merge_update</code>, applicable to WHEN MATCHED</code> and WHEN NOT MATCHED</code> clauses, consisting of regular UPDATE</code> operations.</li>
The second is merge_insert</code> for the WHEN NOT MATCHED</code> clause, allowing (as the name implies) data to be inserted.</li>
The DELETE</code> we introduce is part of merge_delete</code>.</li>
</ul>
Technically, there's also the ability to specify DO NOTHING</code> for all available when_clauses</code>, similar to upserts using ON CONFLICT</code>.</p>
Using MERGE</code> Output</a>
</h2>
Starting from PostgreSQL 17, the MERGE</code> command has been extended to support RETURNING</code> clauses to process merged data further. When an INSERT</code> or UPDATE</code> action is performed, the new values of the target table's columns are used. When a DELETE</code> is performed, the old values of the target table's columns are used.</p>
Conclusion</a>
</h2>
The MERGE</code> command significantly enhances the ability to handle complex INSERT</code> and UPDATE</code> logic by allowing multiple operations within a single query while also capturing even the most intricate scenarios. In this article, we have explored the syntax and common use cases of MERGE</code>. For further exploration, note that the source</code> dataset is where you can perform any data transformations required, making MERGE</code> particularly well-suited for ETL operations, data archival, cleanup tasks, and more.</p>
Incorporating MERGE</code> into your PostgreSQL toolkit can simplify database operations, reduce the risk of data inconsistencies, and streamline your</p>


Gentle Introduction to Window Functions in PostgreSQL
2024-07-07T00:00:00+00:00
Understanding the relationship between data points is crucial. For instance, you might need to identify the most recent orders for each customer or track changes in sensor readings over time. Unlike aggregate functions, which summarise data into a single row, it is window functions that allow you to analyse data while preserving each row’s details. This is the core of the logic, but don’t worry if you struggle to imagine the difference, as we will cover all of it in this article.</p>
PostgreSQL supports SQL window functions, facilitating complex calculations across related rows within a table. These functions are particularly useful for tasks such as ranking entries, calculating running totals, finding moving averages, and comparing individual entries. Mastering window functions can significantly enhance your data analysis capabilities.</p>
You can easily use similar window functions as you would with aggregation. Let’s take a simple example of sensor readings:</p>
CREATE TABLE</span> sensor_readings</span> (</span></span>
    sensor_id </span>bigint</span>,</span></span>
    reading_value </span>decimal</span>,</span></span>
    reading_time </span>timestamp with time zone default</span> current_timestamp</span></span>
);</span></span>
</span>
INSERT INTO</span> sensor_readings (sensor_id, reading_value, reading_time) </span>VALUES</span></span>
(</span>1</span>, </span>32</span>.</span>7</span>, </span>'2024-07-01 11:24:34'</span>),</span></span>
(</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 11:29:01'</span>),</span></span>
(</span>1</span>, </span>33</span>.</span>1</span>, </span>'2024-07-02 12:03:59'</span>),</span></span>
(</span>1</span>, </span>33</span>.</span>0</span>, </span>'2024-07-03 10:12:15'</span>),</span></span>
(</span>2</span>, </span>32</span>.</span>8</span>, </span>'2024-07-01 13:17:01'</span>),</span></span>
(</span>2</span>, </span>35</span>.</span>8</span>, </span>'2024-07-02 09:18:11'</span>),</span></span>
(</span>3</span>, </span>29</span>.</span>1</span>, </span>'2024-07-01 13:54:03'</span>),</span></span>
(</span>3</span>, </span>30</span>.</span>3</span>, </span>'2024-07-01 14:12:09'</span>),</span></span>
(</span>3</span>, </span>31</span>.</span>5</span>, </span>'2024-07-02 16:07:43'</span>);</span></span></code></pre>
When you start with the aggregation functions, it is straightforward for anybody familiar with SQL:</p>
SELECT</span></span>
    sensor_id,</span></span>
    AVG</span>(reading_value)</span></span>
FROM</span> sensor_readings</span></span>
GROUP BY</span> sensor_id;</span></span></code></pre>
with the output</p>
 sensor_id |         avg</span></span>
-----------+---------------------</span></span>
         3 | 30.3000000000000000</span></span>
         2 | 34.3000000000000000</span></span>
         1 | 32.9750000000000000</span></span></code></pre>
While similar, using the window function AVG we can get almost same result:</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_value,</span></span>
    AVG</span>(reading_value) </span>OVER</span> (</span>PARTITION BY</span> sensor_id) </span>AS</span> avg_reading_value</span></span>
FROM</span> sensor_readings;</span></span></code></pre>
such as</p>
 sensor_id | reading_value |  avg_reading_value</span></span>
-----------+---------------+---------------------</span></span>
         1 |          32.7 | 32.9750000000000000</span></span>
         1 |          33.1 | 32.9750000000000000</span></span>
         1 |          33.1 | 32.9750000000000000</span></span>
         1 |          33.0 | 32.9750000000000000</span></span>
         2 |          32.8 | 34.3000000000000000</span></span>
         2 |          35.8 | 34.3000000000000000</span></span>
         3 |          29.1 | 30.3000000000000000</span></span>
         3 |          30.3 | 30.3000000000000000</span></span>
         3 |          31.5 | 30.3000000000000000</span></span></code></pre>
This reiterates the fundamental difference between the two sets of functions. As mentioned earlier, while the results for sensor_id</code> are the same in both cases, the aggregate function</strong> summarised it into a single row (grouped by sensor_id</code>), whereas the window function</strong> provides the value for the set of rows in the partition defined by sensor_id</code>.</p>
After showing the average in a window function, it’s important to note that traditional aggregation functions might not be the most helpful in the context of window function logic. Despite functions like AVG</code>, COUNT</code>, and SUM</code>—which are the most used aggregation functions—being available as window functions, we used the above only to demonstrate the difference.</p>
OVER clause</a>
</h2>
Before diving into the individual functions, let’s cover the syntax first. From the sample query above, you already get the basic syntax of the window functions, with the OVER</code> clause being a primary identification of windowing functionality.</p>
window_function ([expression...]) OVER window_definition</span></span></code></pre>
In our example, the basic window functions can be similar to the aggregate ones— like AVG</code>, SUM</code> and COUNT</code> - and a number of the functions we will cover shortly.</p>
The window definition part specifies how the window function will see the data it works over. It can include:</p>

Partitioning</strong> to divide the result sets into separate partitions (think groups in aggregate functions).</li>
Definition of the ordering</strong> of the rows within each partition.</li>
Specification of how to apply framing</strong> of the subset of rows for each row’s calculation.</li>
</ol>
From all the components of the window syntax, only partitioning is mandatory.</p>
If we revisit our first window function example above, you can identify the partitioning</strong> part (PARTITION BY sensor_id</code>). As already mentioned, you can easily compare the partitioning logic to the grouping used in aggregate functions, with the same properties—like partitioning data segments by multiple fields, using functions and other logic.</p>
With partitioning comes hand in hand ordering</strong> of the data. When you start thinking of row properties, instead of grouping data into a single row, order starts to make a difference. Let’s take the following example:</p>
window_function ([expression...]) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time)</span></span></code></pre>
This will provide different data compared to unsorted ones. If you struggle to find the application for this, think of the moving average or row numbering.</p>
The last component of the window definition is framing</strong>, allowing you to further limit the data over which the window function is calculated. There are two framing expressions: ROWS BETWEEN</code> and RANGE BETWEEN</code>. Using the framing, we can turn AVG from the above example to provide our first real use case when you might want to use window functions.</p>
SELECT</span></span>
    reading_time,</span></span>
    sensor_id,</span></span>
    reading_value,</span></span>
    AVG</span>(reading_value) </span>OVER</span> (</span></span>
        PARTITION BY</span> sensor_id</span></span>
        ORDER BY</span> reading_time</span></span>
        ROWS BETWEEN</span> 1</span> PRECEDING AND</span> 1</span> FOLLOWING</span></span>
    ) </span>AS</span> moving_avg_reading_value</span></span>
FROM</span> sensor_readings;</span></span></code></pre>
Implementing the moving average of the individual readings, taking into account a maximum of 3 rows including the current one.</p>
      reading_time      | sensor_id | reading_value | moving_avg_reading_value</span></span>
------------------------+-----------+---------------+--------------------------</span></span>
 2024-07-01 11:24:34+02 |         1 |          32.7 |      32.9000000000000000</span></span>
 2024-07-02 11:29:01+02 |         1 |          33.1 |      32.9666666666666667</span></span>
 2024-07-02 12:03:59+02 |         1 |          33.1 |      33.0666666666666667</span></span>
 2024-07-03 10:12:15+02 |         1 |          33.0 |      33.0500000000000000</span></span>
 2024-07-01 13:17:01+02 |         2 |          32.8 |      34.3000000000000000</span></span>
 2024-07-02 09:18:11+02 |         2 |          35.8 |      34.3000000000000000</span></span>
 2024-07-01 13:54:03+02 |         3 |          29.1 |      29.7000000000000000</span></span>
 2024-07-01 14:12:09+02 |         3 |          30.3 |      30.3000000000000000</span></span>
 2024-07-02 16:07:43+02 |         3 |          31.5 |      30.9000000000000000</span></span></code></pre>Exploring Window functions</a>
</h2>
Now that we have a foundational understanding of window functions and their components, let’s dive into specific window functions. We’ll cover some of the most commonly used functions, such as ROW_NUMBER()</code>, RANK()</code>, DENSE_RANK()</code>, LAG()</code>, LEAD()</code>, FIRST_VALUE()</code>, LAST_VALUE()</code> and more. For each function, we’ll provide examples to illustrate their practical applications.</p>
ROW_NUMBER</a>
</h3>
The ROW_NUMBER()</code> function assigns a unique sequential integer to rows within a partition of a result set, starting with one for the first row in each partition.</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_time,</span></span>
    reading_value,</span></span>
    ROW_NUMBER</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> row_num</span></span>
FROM</span> sensor_readings;</span></span></code></pre>
This makes it easy to find the first/last entries for a specified window. As an example, you can experiment with getting only the last reading for each hour.</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_time,</span></span>
    reading_value</span></span>
FROM</span> (</span></span>
    SELECT</span></span>
        sensor_id,</span></span>
        reading_time,</span></span>
        reading_value,</span></span>
        ROW_NUMBER</span>() </span>OVER</span> (</span></span>
            PARTITION BY</span> sensor_id, date_trunc(</span>'hour'</span>, reading_time)</span></span>
            ORDER BY</span> reading_time </span>DESC</span></span>
        ) </span>AS</span> row_num</span></span>
    FROM</span> sensor_readings</span></span>
) </span>AS</span> ranked_readings</span></span>
WHERE</span> row_num </span>=</span> 1</span>;</span></span></code></pre>RANK and DENSE_RANK</a>
</h3>
The RANK()</code> and DENSE_RANK()</code> functions are used to assign a rank to each row within a partition, based on the order of one or more values. These functions are useful when scoring the values and handling ties. The main difference between RANK</code> and DENSE_RANK</code> is how they deal with gaps.</p>
Example use to find the highest reading_value per sensor:</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_time,</span></span>
    reading_value,</span></span>
    RANK</span>() </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_value </span>DESC</span>) </span>AS</span> rank</span></span>
FROM</span> sensor_readings;</span></span></code></pre>
If you consider the sorting for the sensor_id</code> 1 in our sample seed the difference between RANK</code> and DENSE_RANK</code> is easy to demonstrate.</p>
 sensor_id |      reading_time      | reading_value | rank</span></span>
-----------+------------------------+---------------+------</span></span>
         1 | 2024-07-02 11:29:01+02 |          33.1 |    1</span></span>
         1 | 2024-07-02 12:03:59+02 |          33.1 |    1</span></span>
         1 | 2024-07-03 10:12:15+02 |          33.0 |    3</span></span>
         1 | 2024-07-01 11:24:34+02 |          32.7 |    4</span></span></code></pre>
Giving a natural ranking with tie on the reading value 33.1 and leaving a gap on 2nd rank, whereas DENSE_RANK</code> wouldn't include the gap.</p>
 sensor_id |      reading_time      | reading_value | rank</span></span>
-----------+------------------------+---------------+------</span></span>
         1 | 2024-07-02 11:29:01+02 |          33.1 |    1</span></span>
         1 | 2024-07-02 12:03:59+02 |          33.1 |    1</span></span>
         1 | 2024-07-03 10:12:15+02 |          33.0 |    2</span></span>
         1 | 2024-07-01 11:24:34+02 |          32.7 |    3</span></span></code></pre>LAG</code> and LEAD</code></a>
</h3>
To evaluate previous or subsequent rows without the need to self-join the dataset, you can utilise the power of the window functions LAG</code> and LEAD</code>. These functions are particularly useful for calculating differences between rows, comparing current and previous values, or fetching future values for comparison.</p>
The LAG</code> function provides access to a value in a previous row within the partition, while LEAD</code> provides access to a subsequent row.</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_time,</span></span>
    reading_value,</span></span>
    LAG</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> previous_value,</span></span>
    LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time) </span>AS</span> next_value</span></span>
FROM</span> sensor_readings;</span></span></code></pre>
In this query, we are using a value of 1, but you can choose any position necessary. By using LAG()</code> and LEAD()</code>, you can perform advanced analyses that require looking backward or forward within your dataset, making it easier to derive meaningful insights and trends.</p>
FIRST_value</code> and LAST_VALUES</code></a>
</h3>
The functions FIRST_value</code> and LAST_VALUES</code> are similar, except they (as the name says) give the first/last value of the partition.</p>
The FIRST_VALUE</code> is useful when comparing the partition values to the first value (for example opening price for a day), and LAST_VALUE</code> to identity the closing values.</p>
Other Window functions</a>
</h3>
The complete list of the window functions is available in the documentation</a>.</p>
Re-using the window definition</a>
</h3>
As you might have noticed, the query we used to demonstrate LAG</code> and LEAD</code> was rather verbose. This was due to the repeated definition of the window for both columns. Luckily, the syntax of the window functions allows you to define and re-use the window definition.</p>
SELECT</span></span>
    sensor_id,</span></span>
    reading_time,</span></span>
    reading_value,</span></span>
    LAG</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> previous_value,</span></span>
    LEAD</span>(reading_value, </span>1</span>) </span>OVER</span> readings_window </span>AS</span> next_value</span></span>
FROM</span> sensor_readings</span></span>
WINDOW</span> readings_window </span>AS</span> (</span>PARTITION BY</span> sensor_id </span>ORDER BY</span> reading_time);</span></span></code></pre>
This way you can ensure the consistent window definition and consistency in complex queries.</p>

The NULL in your NOT IN

Welcome to ORDER BY jungle

Strong views on PostgreSQL VIEWs

Good CTE, bad CTE

The hidden cost of PostgreSQL arrays

RegreSQL: Regression Testing for PostgreSQL Queries

Beyond Start and End: PostgreSQL Range Types

Time to Better Know The Time in PostgreSQL

VIEW inlining in PostgreSQL

DELETEs are difficult

Text identifiers in PostgreSQL database design

We need to talk about ENUMs

Beyond Simple Upserts with MERGE in PostgreSQL

Gentle Introduction to Window Functions in PostgreSQL

boringSQL | Supercharge your SQL & PostgreSQL powers - sql

The DISTINCT in your COUNT

The tests passed. The plan didn't.

Same rows, different SUM

boringSQL | Supercharge your SQL & PostgreSQL powers - sql

The DISTINCT in your COUNT

The tests passed. The plan didn't.

Same rows, different SUM

The NULL in your NOT IN

Welcome to ORDER BY jungle

Strong views on PostgreSQL VIEWs

Good CTE, bad CTE

The hidden cost of PostgreSQL arrays

RegreSQL: Regression Testing for PostgreSQL Queries

Beyond Start and End: PostgreSQL Range Types

Time to Better Know The Time in PostgreSQL

PostgreSQL timestamp vs timestamptz</a> </h2> The next possible source of confusion when working with time in PostgreSQL is presence of two distinct data types:</p> timestamp</code> (or timestamp without time zone</code>)</li>

VIEW inlining in PostgreSQL