boringSQL | Supercharge your SQL & PostgreSQL powers - internals

How TABLESAMPLE picks rows

2026-10-11T20:30:00+00:00

Most exploratory questions against a large table only need a rough answer, but without an index on status, even "roughly how many shipped orders" costs a full scan. On the two-million-row orders table, Postgres has to read every single 8 kB page, all 18,085 of them (141 MB), just to count matching rows.

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) FROM orders WHERE status = 'shipped';

 Aggregate
   Buffers: shared hit=18085
   ->  Seq Scan on orders (actual rows=500000.00 loops=1)
         Filter: (status = 'shipped'::text)
         Rows Removed by Filter: 1500000
 Execution Time: 56.188 ms

shared hit=18085 is one buffer access per page, every page found already in memory (Reading Buffer statistics in EXPLAIN output goes through the counters). At 141 MB of heap size that's nothing. Take a 1 TB table with 134 million pages: a scan reads all of them from disk, every time, however small the answer is.

Postgres reads tables larger than a quarter of shared_buffers through a small ring buffer. See more in introduction to buffers.

At least Postgres avoids flushing its own cache by reading large tables through a small ring buffer. Nevertheless the disk still perform full read. On a primary server handling production traffic is shared with everything else. When all you need is exploratory confirmation "roughly how many" or "are there any representative samples" reading the whole terabyte is too expensive.

SQL Standard comes with TABLESAMPLE to help with that. It goes in the FROM clause right after the table name, and the rest of the query runs against the subset of pages it picks.

SELECT
  id, status, created_at, amount
FROM orders TABLESAMPLE SYSTEM (1) REPEATABLE (7)
LIMIT 5;

 id  |  status   |       created_at       | amount
-----+-----------+------------------------+--------
 429 | delivered | 2024-01-01 00:07:09+00 |  38.46
 430 | delivered | 2024-01-01 00:07:10+00 | 150.72
 431 | delivered | 2024-01-01 00:07:11+00 | 295.59
 432 | delivered | 2024-01-01 00:07:12+00 | 488.23
 433 | delivered | 2024-01-01 00:07:13+00 | 238.84

The value (1) is a percentage, and the size of the sample isn't fixed: five different seeds returned between 18,107 and 20,416 rows. If you need an exact number of rows, tsm_system_rows in contrib adds a SYSTEM_ROWS (n) method. The rows come back in physical order, which is why these five are neighbours, ids 429 to 433. And whatever you compute is computed over the sample, so totals have to be scaled back up by hand, count(*) and sum() times 100, while averages, ratios and extremes are read as they come.

Here's the shipped-orders count again, scaled up, with its buffer numbers:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) * 100 FROM orders TABLESAMPLE SYSTEM (1)
WHERE status = 'shipped';

 Aggregate
   Buffers: shared hit=177
   ->  Sample Scan on orders (actual rows=4920.00 loops=1)
         Sampling: system ('1'::real)
         Filter: (status = 'shipped'::text)
 Execution Time: 0.829 ms

177 pages instead of 18,085, and the estimate comes out at 492,000 against the real 500,000. For counts like this one it holds up. The rows are inserted one per second in created_at order, though, so the oldest row sits at the very start of the table. Ask the same 1% sample for it:

SELECT min(created_at) FROM orders TABLESAMPLE SYSTEM (1);

Six runs:

 2024-01-01 02:51:13+00
 2024-01-01 01:36:19+00
 2024-01-01 01:27:24+00
 2024-01-01 04:41:47+00
 2024-01-01 02:51:13+00
 2024-01-01 00:44:36+00

The real answer is 00:00:01. The sample is off by anything from 44 minutes to almost five hours, and runs one and five agree to the second, which a sample of random rows would almost never do. Everything odd about TABLESAMPLE comes from one fact: it doesn't pick rows, it picks addresses.

Setup

Everything in this article was captured on PostgreSQL 18.6 using the postgres:18 image.shared_buffers = 1GB and a warm cache, so the timings are for comparing methods against each other, not for predicting yours.

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE orders (
    id          bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    customer_id bigint        NOT NULL,
    status      text          NOT NULL,
    created_at  timestamptz   NOT NULL,
    amount      numeric(10,2) NOT NULL
);

SELECT setseed(0.42);
INSERT INTO orders (customer_id, status, created_at, amount)
SELECT (random() * 50000)::bigint + 1,
       CASE WHEN i <= 1300000 THEN 'delivered'
            WHEN i <= 1800000 THEN 'shipped'
            WHEN i <= 1900000 THEN 'cancelled'
            ELSE 'pending' END,
       timestamptz '2024-01-01 00:00:00+00' + i * interval '1 second',
       round((random() * 500)::numeric, 2)
FROM generate_series(1, 2000000) AS i;
VACUUM ANALYZE orders;

-- the same rows in random physical order
CREATE TABLE orders_shuffled AS SELECT * FROM orders ORDER BY random();
VACUUM ANALYZE orders_shuffled;

CREATE TABLE customers AS SELECT id FROM generate_series(1, 50001) AS id;
ALTER TABLE customers ADD PRIMARY KEY (id);

-- a small copy with room on every page, for the pageinspect step
CREATE TABLE h (LIKE orders INCLUDING ALL) WITH (fillfactor = 70);
INSERT INTO h OVERRIDING SYSTEM VALUE
SELECT * FROM orders WHERE id <= 1000 ORDER BY id;

Every row has an address

The pages are numbered from 0, and right after its header each page keeps an array of line pointers, one per row, numbered from 1 (Inside PostgreSQL's 8KB Page takes one apart byte by byte). Page number and line pointer together are the row's ctid:

SELECT ctid, id, created_at FROM orders ORDER BY id LIMIT 3;

 ctid  | id |       created_at
-------+----+------------------------
 (0,1) |  1 | 2024-01-01 00:00:01+00
 (0,2) |  2 | 2024-01-01 00:00:02+00
 (0,3) |  3 | 2024-01-01 00:00:03+00

Both built-in sampling methods hash part of that address together with a seed and keep whatever hashes under a 1% cutoff. They differ only in which part. SYSTEM, in src/backend/access/tablesample/system.c, hashes the page number (trimmed to the lines that matter):

hashinput[0] = nextblock;          /* page */
hashinput[1] = sampler->seed;
hash = DatumGetUInt32(hash_any((const unsigned char *) hashinput,
                               (int) sizeof(hashinput)));
if (hash < sampler->cutoff)        /* keep the whole page */

BERNOULLI, in bernoulli.c, hashes the page and the line pointer number:

hashinput[0] = blockno;            /* page */
hashinput[1] = tupoffset;          /* line pointer */
hashinput[2] = sampler->seed;

So SYSTEM flips one coin per page and takes every row on a winning page. BERNOULLI flips one coin per line pointer, which means it has to open every page to see how many line pointers there are. That's the whole performance story: on this table SYSTEM answers in 1.2 ms and BERNOULLI in 10.4 ms.

Pages come as bundles

A page here holds about 107 rows, and since the rows went in by time, one page is 107 consecutive seconds of orders. Group a SYSTEM sample by the page half of ctid and the bundles are right there:

SELECT (ctid::text::point)[0]::int AS page, count(*) AS rows,
       min(created_at)::time AS first, max(created_at)::time AS last
FROM orders TABLESAMPLE SYSTEM (1) REPEATABLE (7)
GROUP BY 1 ORDER BY 1 LIMIT 4;

 page | rows |  first   |   last
------+------+----------+----------
    4 |  107 | 00:07:09 | 00:08:55
   38 |  107 | 01:07:47 | 01:09:33
   57 |  107 | 01:41:40 | 01:43:26
  265 |  107 | 07:52:36 | 07:54:22

The sample isn't 20,000 scattered seconds. It's about 200 short strips of the timeline, and min() returns the start of whichever strip the hash kept first. Runs one and five above kept the same first page.

Shuffle the table and the problem disappears. orders_shuffled holds the same rows written in ORDER BY random() order, so every page is a handful of seconds from all over the three weeks, and the same query lands within a few minutes of the truth every time:

 2024-01-01 00:00:06+00
 2024-01-01 00:01:20+00
 2024-01-01 00:01:43+00
 2024-01-01 00:01:24+00
 2024-01-01 00:02:50+00
 2024-01-01 00:05:07+00

BERNOULLI on the original, time-ordered table does just as well, because a coin per line pointer never bundles anything.

Nobody samples for a minimum, but the bundles hurt averages the same way. Across 200 seeds, avg(created_at) from a SYSTEM sample wanders ten times as far as the same average from BERNOULLI. The reason is plain once you think in pages: 107 rows from one page tell you about as much about the timeline as one row from that page would, so a SYSTEM sample of 20,000 rows has roughly the precision of a 200-row one. A column that has nothing to do with where rows sit, like a random amount, averages identically under both methods. Time columns, ever-increasing ids, and anything loaded in batches are the columns where it matters.

REPEATABLE remembers addresses

REPEATABLE (42) fixes the seed. Same seed, same hash, same addresses. Whether those addresses still hold the same rows depends on what happened to the table in between, and the address model predicts every line of this table. Each run captures a sample with seed 42, changes the table, captures again, and counts how many rows from the first sample are still in it:

after	SYSTEM kept	BERNOULLI kept
append 200k rows	17,595 of 17,595	19,800 of 19,800
delete half the rows	8,795 of 8,795	10,003 of 10,003
`VACUUM`	8,795 of 8,795	10,003 of 10,003
`UPDATE` every row	8,740 of 8,795	109 of 10,003
`VACUUM FULL`	113 of 8,795	113 of 10,003
`CLUSTER` on a reversed index	0 of 8,795	95 of 10,003

Appends create new addresses and deletes empty old ones; no surviving row moves, so nothing changes. VACUUM marks dead line pointers unused and compacts the tuple data inside each page, but it never renumbers a line pointer that still points at a live row (VACUUM at the Page Level walks through each phase). VACUUM FULL and CLUSTER build a fresh copy of the table, every row gets a new address, and both samples are effectively redrawn.

UPDATE is the interesting line. Postgres never overwrites a row in place: an update writes a new version, and when the page has room and no indexed column changed, it puts that version on the same page without touching any index. That's a HOT update (HOT Updates in Postgres), and here 99% of them were. Same page means SYSTEM doesn't notice. But the new version gets a new line pointer, and pageinspect shows it for one row of h:

SELECT ctid FROM h WHERE id = 3;   -- (0,3)
UPDATE h SET amount = amount + 1 WHERE id = 3;
SELECT ctid FROM h WHERE id = 3;   -- (0,76)

SELECT lp, lp_flags, lp_off, t_ctid
FROM heap_page_items(get_raw_page('h', 0)) WHERE lp IN (3, 76);

 lp | lp_flags | lp_off | t_ctid
----+----------+--------+--------
  3 |        2 |     76 |
 76 |        1 |   2792 | (0,76)

Line pointer 3 is now LP_REDIRECT (lp_flags = 2), a signpost that says "moved to 76" so index entries pointing at (0,3) still find the row. BERNOULLI doesn't read signposts. It hashes line pointer 76, which is a fresh 1% coin flip, so almost every updated row drops out of the sample and a different 1% drops in.

Hashing the key instead

When people pin a seed, they usually want a development or test subset that stays the same from one day to the next. An address can't promise that, because updates and rewrites move rows. A primary key doesn't move, so hash the key:

SELECT * FROM orders WHERE (hashint8extended(id, 42) & 1023) < 10;

hashint8extended hashes a bigint with a seed; keeping 10 of 1,024 buckets returns 0.98% of the rows. Through the same changes it kept every surviving row at every step, 9,916 of 9,916 after the delete, UPDATE, VACUUM FULL and CLUSTER included. It has no bundles either: its 200-seed spread on avg(created_at) matches BERNOULLI.

A key can also be a foreign key. Sample customers on hashint8extended(id, 42) and orders on hashint8extended(customer_id, 42), and you get 484 customers with all 19,432 of their orders. The same two tables sampled with SYSTEM (1) REPEATABLE (42) leave 17,517 of the 17,595 sampled orders pointing at a customer who isn't in the sample.

The price is a full scan, since the hash has to be computed for every row: 44.4 ms serially, 21.5 ms with two parallel workers. For a subset you pull often, an expression index on (hashint8extended(id, 42) & 1023) (14 MB here) turns it into a bitmap scan at 8.5 ms, with the seed fixed in the index definition.

The join doesn't shrink

Everything so far read one table. Join the orders to their customers and count:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) FROM orders o JOIN customers c ON c.id = o.customer_id;

The full join answers in 84.4 ms from a parallel hash join with two workers: 18,751 buffer hits, all but 666 of them on orders. Sample the orders side:

EXPLAIN (ANALYZE, BUFFERS)
SELECT count(*) * 100
FROM orders o TABLESAMPLE SYSTEM (1) REPEATABLE (7)
JOIN customers c ON c.id = o.customer_id;

 Aggregate
   Buffers: shared hit=424
   ->  Hash Join (actual rows=22316.00 loops=1)
         Hash Cond: (o.customer_id = c.id)
         ->  Sample Scan on orders o (actual rows=22316.00 loops=1)
               Sampling: system ('1'::real)
               Buffers: shared hit=202
         ->  Hash
               Buckets: 65536  Batches: 1  Memory Usage: 2270kB
               Buffers: shared hit=222
               ->  Seq Scan on customers c (actual rows=50001.00 loops=1)
                     Buffers: shared hit=222
 Execution Time: 25.590 ms

The sample scan did its part, 202 pages against the full 18,085. The hash over customers still reads all 222 of its pages, because TABLESAMPLE only shrinks the table it sits on, and building the hash is 11.5 ms of the 25.6. The query gets 3.3× faster, not 90×. Most of what remains is the table the sample never touched. The workers are gone too: a sample scan runs in the leader process, so the plan falls back to serial however big the table is.

Forcing the other join shape moves the cost rather than removing it:

SET enable_hashjoin = off;
SET enable_mergejoin = off;

The same query then runs 22,316 index-only probes against customers_pkey, 44,636 buffer hits against the sample's own 202, and finishes in 21.6 ms. Every sampled row pays full price for its customer lookup.

A predicate on the far side works normally: adding WHERE c.id <= 500 builds the hash over 500 rows and answers in 1.5 ms. Some questions break outright, though. Every one of the 50,001 customers has orders, but the 1% draw sees:

SELECT count(DISTINCT customer_id) FROM orders TABLESAMPLE SYSTEM (1) REPEATABLE (7);

 count
-------
 18034

Multiplying by 100 can't bring back a customer whose orders all landed on losing pages. A sample scales sums and counts of what it kept; anything you have to see to count comes out low.

Sampling the join output directly isn't an option: TABLESAMPLE attaches to one table in the FROM clause, and putting it on a join or a subquery is a syntax error. Sampling both sides is as close as you can get, and it's close to useless:

SELECT count(*)
FROM orders o TABLESAMPLE SYSTEM (1) REPEATABLE (7)
JOIN customers c TABLESAMPLE SYSTEM (1) REPEATABLE (7) ON c.id = o.customer_id;

 count
-------
   288

The real join has two million rows. Two independent address draws keep 288 of them, because a sampled order finds its customer only when both page hashes kept the matching pair. When a sampled pair of tables has to stay joinable, hashing the foreign key on both sides is the way to do it.

Which to reach for

SYSTEM for a quick number over a column that has nothing to do with insert order. BERNOULLI when it might, and reading the whole table once is affordable. A hashed key when the subset has to be the same rows tomorrow, or has to keep its foreign keys intact. None of them when you need a filtered subset: the sample is chosen before your WHERE runs, so the filter only trims a draw that was already made.

ANALYZE builds pg_stats from a sample too, with its own block sampler and a seed you don't get to set, which is why the numbers move between runs even when the data hasn't.

Postgres relation files, filed under base/

2026-10-09T09:30:00+00:00

The buffers article followed an 8KB page from disk into shared memory and back. What it didn't do is name the file. Every page that passes through the buffer pool is block number N of some file under the data directory, and the file has a naming scheme, a size limit, and companion files that share its name. Before the next article reads the inside of a page, this one finds it on disk with ls and od, and shows that the bytes are the same ones.

Nothing here needs more than a scratch database and shell access to its data directory. The examples were captured on PostgreSQL 18.6 with data checksums on, which is the default since 18.

Setup

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE files_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    title text NOT NULL,
    value numeric(10,2)
);

INSERT INTO files_demo (title, value)
SELECT 'row_' || i, i * 1.5 FROM generate_series(1, 1000) AS i;

A thousand short rows and a primary key. The text column is there so a TOAST table gets created.

From table name to file name

The function that answers "where is this table on disk" ships with the server:

SELECT pg_relation_filepath('files_demo');

 pg_relation_filepath
----------------------
 base/5/16431
(1 row)

The path is relative to the data directory (SHOW data_directory prints it). Three parts: base is the directory for the default tablespace, 5 is the OID of the database, and 16431 is the file's name. That last number is the relfilenode, and it comes from pg_class:

SELECT oid, relfilenode, relname
FROM pg_class
WHERE relname LIKE 'files_demo%';

  oid  | relfilenode |      relname
-------+-------------+-------------------
 16431 |       16431 | files_demo
 16430 |       16430 | files_demo_id_seq
 16438 |       16438 | files_demo_pkey
(3 rows)

Three relations, three files. The identity column's sequence and the primary key index each get their own, because to the storage layer a sequence is one-page relation and an index is a relation like any other. The oid and relfilenode columns are equal for all three, which stays true only until the table is first rewritten (see OID versus relfilenode below).

The middle part of the path, 5, is the OID of the current database from pg_database. Each database gets its own subdirectory under base/, and a table's file lives in the directory of the database it belongs to. That is the whole reason the database OID is part of the path: relfilenodes are only unique within a tablespace and database, so two databases can each have a file called 16431 without colliding.

On disk, the directory looks like the catalog:

$ ls -l base/5/16431 base/5/16438 base/5/16430
-rw------- 1 postgres postgres  8192 Sep 11 20:44 base/5/16430
-rw------- 1 postgres postgres 57344 Sep 11 20:44 base/5/16431
-rw------- 1 postgres postgres 40960 Sep 11 20:44 base/5/16438

pg_relation_size gets its numbers the same way: it does nothing more than stat the files.

The base/5 directory holds a few hundred files on a fresh database before any of ours, almost all of them system catalogs and their indexes, and every one follows the same scheme.

The files you didn't create

There is a function that goes the other way, from a file number back to a name. Point it at the numbers between and around ours:

SELECT n, pg_filenode_relation(0, n)
FROM unnest(ARRAY[16430, 16431, 16436, 16437, 16438]) AS n;

   n   |     pg_filenode_relation
-------+-------------------------------
 16430 | files_demo_id_seq
 16431 | files_demo
 16436 | pg_toast.pg_toast_16431
 16437 | pg_toast.pg_toast_16431_index
 16438 | files_demo_pkey
(5 rows)

The first argument is the tablespace OID, where 0 means the database's default. Two relations we didn't ask for appear: a TOAST table and its index. Because files_demo has a text column, PostgreSQL created a place to put values too large for a page. They're ordinary files, 16436 and 16437, sitting next to the main table, and nothing has been toasted yet, so the TOAST heap has no pages. The catalog links them:

SELECT oid, relname, relkind, relfilenode, reltoastrelid
FROM pg_class
WHERE oid IN (16431, 16436, 16437);

  oid  |       relname        | relkind | relfilenode | reltoastrelid
-------+----------------------+---------+-------------+---------------
 16431 | files_demo           | r       |       16431 |         16436
 16436 | pg_toast_16431       | t       |       16436 |             0
 16437 | pg_toast_16431_index | i       |       16437 |             0
(3 rows)

So one CREATE TABLE with a primary key produced five relations and five files. pg_table_size covers the heap, its TOAST relations, and the forks described next; pg_indexes_size covers the indexes; pg_total_relation_size is the sum of those two. The sequence is its own relation and is counted by none of them.

Two kinds of relation break the one-relation-one-file pattern. Views, partitioned tables, and foreign tables have no storage at all: their relfilenode is 0 and pg_relation_filepath() returns NULL. Temporary tables do have files, in the same directory under names of the form t<backend>_<relfilenode>. pg_relation_filepath() resolves another session's temp table too, by OID or by its schema-qualified pg_temp_N name; only reading its contents fails.

Forks

Each relation has a main file and up to three companion files that share its relfilenode and differ by a suffix. PostgreSQL calls them forks. The one with no suffix is the main fork, the table's actual pages. After the INSERTs and a VACUUM, files_demo has three of the four:

VACUUM files_demo;

SELECT fork,
       pg_relation_size('files_demo', fork) AS bytes,
       pg_relation_size('files_demo', fork) / 8192 AS pages
FROM unnest(ARRAY['main', 'fsm', 'vm', 'init']) AS fork;

 fork | bytes | pages
------+-------+-------
 main | 57344 |     7
 fsm  | 24576 |     3
 vm   |  8192 |     1
 init |     0 |     0
(4 rows)

$ ls -l base/5/16431*
-rw------- 1 postgres postgres 57344 Sep 11 20:44 base/5/16431
-rw------- 1 postgres postgres 24576 Sep 11 20:44 base/5/16431_fsm
-rw------- 1 postgres postgres  8192 Sep 11 20:44 base/5/16431_vm

16431_fsm is the free space map, a small tree that records roughly how much room each heap page has left so an INSERT can find a page without scanning them all. It's three pages here for a seven-page table, which looks disproportionate until you know the map is a three-level tree and each level needs at least one page. 16431_vm is the visibility map, two bits per heap page: all-visible, so a scan can skip visibility checks, and all-frozen, so VACUUM can skip the page for wraparound purposes. The VACUUM article reads both of these with pg_freespacemap and pg_visibility.

The fourth fork, init, is empty for this table because files_demo is a normal, logged table. An unlogged table gets one:

CREATE UNLOGGED TABLE files_unlogged (id int, note text);
INSERT INTO files_unlogged VALUES (1, 'scratch');

SELECT pg_relation_filepath('files_unlogged');

 pg_relation_filepath
----------------------
 base/5/16444
(1 row)

$ ls -l base/5/16444* base/5/16447* base/5/16448*
-rw------- 1 postgres postgres 8192 Sep 11 20:44 base/5/16444
-rw------- 1 postgres postgres    0 Sep 11 20:44 base/5/16444_init
-rw------- 1 postgres postgres    0 Sep 11 20:44 base/5/16447
-rw------- 1 postgres postgres    0 Sep 11 20:44 base/5/16447_init
-rw------- 1 postgres postgres 8192 Sep 11 20:44 base/5/16448
-rw------- 1 postgres postgres 8192 Sep 11 20:44 base/5/16448_init

The init fork is the state the relation is reset to after a crash. An unlogged table skips the write-ahead log, so after unclean shutdown its contents can't be trusted and PostgreSQL replaces the main fork with a copy of the init fork. For a heap that copy is an empty file, which is why 16444_init is zero bytes. 16447 and 16448 are the unlogged table's TOAST heap and TOAST index, created the same way 16436 and 16437 were, and the pattern repeats: the two heaps get a zero-byte init fork, while the TOAST index at 16448 gets a full 8192 bytes, because an empty B-tree still needs its metapage.

Forks are per relation. The TOAST table 16436, being a heap, can grow its own _fsm and _vm. The index 16438 can grow an _fsm, which B-trees use to track pages freed by deletion, but never a _vm, because visibility is a heap concept.

The same bytes, two ways

pageinspect's get_raw_page returns a page as it sits in shared buffers. The file on disk is supposed to contain the same page at the same offset. Read the first 24 bytes of block 0 both ways, starting with the buffer:

SELECT lsn, checksum, flags, lower, upper, special, pagesize, version, prune_xid
FROM page_header(get_raw_page('files_demo', 0));

    lsn    | checksum | flags | lower | upper | special | pagesize | version | prune_xid
-----------+----------+-------+-------+-------+---------+----------+---------+-----------
 0/186E9B0 |        0 |     4 |   680 |   712 |    8192 |     8192 |       4 |         0
(1 row)

SELECT encode(substring(get_raw_page('files_demo', 0) from 1 for 24), 'hex');

                      encode
--------------------------------------------------
 00000000b0e9860100000400a802c8020020042000000000
(1 row)

Now the file. od prints bytes from an offset, and block 0 starts at offset 0:

$ od -A d -t x1 -N 24 base/5/16431
0000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0000016 00 00 00 00 00 00 00 00
0000024

Nothing. The file has the right length, because the relation was extended with zeroed space, but the INSERT and VACUUM only dirtied pages in shared memory and nobody has written them back. The buffers article's point, seen from the filesystem side. Force it:

CHECKPOINT;

$ od -A d -t x1 -N 24 base/5/16431
0000000 00 00 00 00 b0 e9 86 01 dd f2 04 00 a8 02 c8 02
0000016 00 20 04 20 00 00 00 00
0000024

Now the bytes match the buffer, with one exception picked apart below. Two fields are worth reading by hand:

Bytes 0 to 7, 00 00 00 00 b0 e9 86 01, are pd_lsn. It is stored as two 32-bit little-endian halves, so the high half is 0 and the low half is 0x0186E9B0, which is how page_header printed 0/186E9B0. That value is the write-ahead log position of the last change to this page.
Bytes 8 and 9, dd f2, are pd_checksum, little-endian 0xF2DD.

The remaining 14 bytes match page_header field for field.

Except the checksum. The buffer says 0, the file says f2dd. PostgreSQL computes the checksum when it writes a page out, into the copy it hands to the kernel, and never touches the shared buffer. (Pages written outside the pool, by index builds or the VACUUM FULL rewrite, get theirs in place.) A page that was initialised in memory and has never been read from disk carries a zero checksum in the buffer; one that was read and then modified carries whatever checksum the file had when it was read. Read this page back after it's been evicted and reloaded and the buffer will show the on-disk value, which page_header prints as signed smallint: 0xF2DD comes out as -3363, not 62173.

pageinspect never re-verifies a checksum; it reads the buffer, not the file. Verification happens when a page is read from disk into the pool (so get_raw_page on an uncached page does trip over a corrupt one), and offline with pg_checksums. If you want the on-disk bytes from SQL, pg_read_binary_file on the relation path reads the file rather than the shared buffer. It still goes through the OS page cache, and it needs superuser or the pg_read_server_files role.

OID versus relfilenode

Back to the pair of numbers in pg_class that started out equal. The OID is the relation's identity, the number every catalog reference points at, and it never changes. The relfilenode is just a file name. Most of the time they match, because CREATE TABLE assigns the same number to both. Operations that rewrite the table break the tie:

SELECT oid, relfilenode, pg_relation_filepath('files_demo')
FROM pg_class WHERE relname = 'files_demo';

VACUUM FULL files_demo;

SELECT oid, relfilenode, pg_relation_filepath('files_demo')
FROM pg_class WHERE relname = 'files_demo';

  oid  | relfilenode | pg_relation_filepath
-------+-------------+----------------------
 16431 |       16431 | base/5/16431
(1 row)

  oid  | relfilenode | pg_relation_filepath
-------+-------------+----------------------
 16431 |       16449 | base/5/16449
(1 row)

VACUUM FULL writes a compacted copy of the table into a brand new file and then swaps the catalog entry to point at it. The OID stays 16431 so nothing that references the table notices; the relfilenode moves to 16449. The primary key was rebuilt too, and moved from 16438 to 16454. Before the next checkpoint, the old file is still there:

$ ls -l base/5/16431* base/5/1645*
-rw------- 1 postgres postgres     0 Sep 11 20:45 base/5/16431
-rw------- 1 postgres postgres     0 Sep 11 20:45 base/5/16452
-rw------- 1 postgres postgres  8192 Sep 11 20:45 base/5/16453
-rw------- 1 postgres postgres 40960 Sep 11 20:45 base/5/16454

16452 and 16453 are the rebuilt TOAST table and its index; the gap before them is OIDs the rewrite consumed for its transient relation. The old 16431 is still there at zero bytes, and its _fsm and _vm aren't: PostgreSQL unlinks the companion forks and any extra segments immediately, but truncates the main fork's first segment to nothing and keeps the empty name until the next checkpoint, which unlinks it for good.

md.c explains why: under wal_level = minimal, relation contents are fsynced rather than logged. A relfilenode freed before a checkpoint could be reused by a new relation, and replay of the old relation's drop after a crash would then delete the new relation's file. Holding the name until the checkpoint closes that window.

TRUNCATE does the same swap, to an empty file:

TRUNCATE files_demo;

SELECT oid, relfilenode, pg_relation_filepath('files_demo')
FROM pg_class WHERE relname = 'files_demo';

  oid  | relfilenode | pg_relation_filepath
-------+-------------+----------------------
 16431 |       16455 | base/5/16455
(1 row)

Assigning a fresh file rather than emptying the old one is what makes TRUNCATE transactional. If the transaction rolls back, the catalog row reverts to the old relfilenode and the old file is still there, untouched. CLUSTER, REINDEX, ALTER TABLE ... SET TABLESPACE, and any ALTER TABLE that rewrites rows all take the same route, because each of them builds a new copy of the relation.

Don't rely on oid = relfilenode. Tools and scripts that build paths by hand from pg_class.oid work on a fresh database and break after the first VACUUM FULL. Use pg_relation_filepath() to get the path and pg_filenode_relation() to get the name back. This matters most when reading WAL, where records name relations by their file number, not their OID, because the log describes physical files.

Tablespaces

Everything so far lived under base/, the home of the pg_default tablespace. A tablespace is a directory somewhere else, with the same layout inside:

CREATE TABLESPACE fast LOCATION '/var/lib/postgresql/fast';
CREATE TABLE files_ts (id int, note text) TABLESPACE fast;

SELECT pg_relation_filepath('files_ts');

          pg_relation_filepath
-----------------------------------------
 pg_tblspc/16459/PG_18_202506291/5/16460
(1 row)

$ ls -l pg_tblspc/
lrwxrwxrwx 1 postgres postgres 24 Sep 11 20:45 16459 -> /var/lib/postgresql/fast

pg_tblspc holds one symlink per tablespace, named by its OID. Under the link sits PG_18_202506291, the server and catalog version, so two major versions can share a location during an upgrade; below that it's <database OID>/<relfilenode> again, exactly like base/. Tablespace OID, database OID and relfilenode together name a relation file uniquely in the cluster, and pg_waldump identifies every block it touches by that triple.

Segment files

A single file is capped at 1GB (the --with-segsize build default; packaged builds use it). When a relation grows past that, PostgreSQL starts a second file with the same name and a .1 suffix, then .2, and so on. Each holds 131072 pages of 8192 bytes. The cap exists so the server works on platforms with file-size limits.

To see one, the table has to grow past 1GB:

CREATE TABLE files_big (
    id integer GENERATED ALWAYS AS IDENTITY,
    payload text
);

INSERT INTO files_big (payload)
SELECT repeat('x', 1000) FROM generate_series(1, 1200000);

SELECT pg_size_pretty(pg_relation_size('files_big')) AS size,
       pg_relation_size('files_big') / 8192 AS pages,
       pg_relation_filepath('files_big');

  size   | pages  | pg_relation_filepath
---------+--------+----------------------
 1339 MB | 171429 | base/5/16466
(1 row)

$ ls -l base/5/16466*
-rw------- 1 postgres postgres 1073741824 Sep 11 20:45 base/5/16466
-rw------- 1 postgres postgres  330604544 Sep 11 20:45 base/5/16466.1
-rw------- 1 postgres postgres     368640 Sep 11 20:46 base/5/16466_fsm

The first segment is exactly 1073741824 bytes, 131072 pages. The remaining 171429 minus 131072 is 40357 pages, times 8192 is 330604544, the size of .1. pg_relation_filepath still returns the base name; the segment suffix is added by the storage manager when it computes which file holds a given block. pg_relation_size sums all segments.

Block numbers don't know about segments. Block 131072 is simply the first page of 16466.1:

SELECT encode(substring(get_raw_page('files_big', 131072) from 1 for 24), 'hex');

                      encode
--------------------------------------------------
 00000000304b1e3c63ad00003400c8030020042000000000
(1 row)

$ od -A d -t x1 -N 24 base/5/16466.1
0000000 00 00 00 00 30 4b 1e 3c 63 ad 00 00 34 00 c8 03
0000016 00 20 04 20 00 00 00 00
0000024

Same bytes, checksum included this time. This table is ten times larger than this instance's 128MB shared_buffers, so the page was evicted after being written and read back from disk before get_raw_page saw it, and it carries the checksum the write computed.

Where PostgreSQL stores row locks

2026-09-17T06:40:00+00:00

In PostgreSQL, every tuple starts with 23-byte header, and the first eight bytes are two transaction IDs. t_xmin for the transaction that created the row and t_xmax for the one that deleted or updated it. That is the visibility story covered in PostgreSQL MVCC, Byte by Byte. For now we have discussed t_xmax acting as the delete marker.

t_xmax has a second job. When you run SELECT ... FOR UPDATE or an insert checks a foreign key, PostgreSQL has nowhere else to record the row lock. The shared memory lock table is limited by max_locks_per_transaction. Locking a million rows would exceed its capacity. PostgreSQL works around this by storing the locking transaction ID in t_xmax and marking the row as locked with flags in t_infomask, while readers can still see it, so every row lock in PostgreSQL ends up as a write to the page.

Setup

The setup is one parent table in the usual shape, plus a child table with a foreign key, since foreign key checks lock parent rows. Everything below was captured on a single PostgreSQL 18.6 cluster using the postgres:18 image. Transaction IDs will be different on your cluster; compare the bits instead.

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE lock_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    owner text NOT NULL,
    balance numeric(12,2)
);

CREATE TABLE lock_demo_tx (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    account_id integer NOT NULL REFERENCES lock_demo (id),
    amount numeric(12,2)
);

INSERT INTO lock_demo (owner, balance)
VALUES ('alice', 100.00), ('bob', 200.00), ('carol', 300.00);

SELECT count(*) FROM lock_demo;

`heap_tuple_infomask_flags()` from `pageinspect` turns the two raw bitmask integers into flag names.

Every page read below uses this query:

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                        raw_flags
----+--------+--------+--------+----------------------------------------------------------
  1 |    767 |      0 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
(3 rows)

The insert itself stamps HEAP_XMAX_INVALID. HEAP_XMIN_COMMITTED was set later by the count(*); a page read straight after the insert would still show it unset.

What FOR UPDATE writes

Call the original session A. Open a second psql session, B, lock the first row there and keep the transaction open:

-- session B
BEGIN;
SELECT pg_current_xact_id();   -- 768
SELECT id, owner FROM lock_demo WHERE id = 1 FOR UPDATE;

Back in session A, read the page:

-- session A
SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                            raw_flags
----+--------+--------+--------+--------------------------------------------------------------------------------------------------
  1 |    767 |    768 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_KEYS_UPDATED}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
(3 rows)

Session B's xid sits in t_xmax exactly where a DELETE would put it. Three bits say what kind of lock it is, two of them in t_infomask:

HEAP_XMAX_LOCK_ONLY says this xmax is a locker and the row isn't going anywhere.
HEAP_XMAX_EXCL_LOCK gives the strength: exclusive.

HEAP_KEYS_UPDATED in t_infomask2 is set even though nothing was updated. A lock always sets at least one of the two t_infomask bits, which leaves three patterns for four strengths, so this bit does double duty: it separates FOR UPDATE from FOR NO KEY UPDATE. It is the same bit a key-column UPDATE or a DELETE sets, which is why FOR UPDATE conflicts with the key-share locks foreign keys take.

The HEAP_XMAX_INVALID hint no longer applies, yet the row is still readable from session A:

SELECT owner, balance FROM lock_demo WHERE id = 1;

 owner | balance
-------+---------
 alice |  100.00
(1 row)

A reader checks visibility, sees HEAP_XMAX_LOCK_ONLY, and treats the row as live without even looking at whether 768 committed. Readers never wait for row locks in any case, because snapshot visibility decides what they see; what the bit does is keep the row alive after the locker commits. Without it, a committed lock-only xmax would read as a committed delete.

Now look at what the lock manager knows about all this, asked from session B itself:

-- session B, still inside the open transaction
SELECT locktype, relation::regclass AS relation, mode, granted
FROM pg_locks WHERE pid = pg_backend_pid() ORDER BY locktype;

   locktype    |    relation    |      mode       | granted
---------------+----------------+-----------------+---------
 relation      | lock_demo      | RowShareLock    | t
 relation      | pg_locks       | AccessShareLock | t
 relation      | lock_demo_pkey | RowShareLock    | t
 transactionid |                | ExclusiveLock   | t
 virtualxid    |                | ExclusiveLock   | t
(5 rows)

The transactionid lock is the queue. Every transaction that has an xid holds an ExclusiveLock on it until it ends. Anyone who needs to wait for that transaction, for any row it has locked, requests a ShareLock on the same xid and blocks.

There is no entry for row 1, because the row lock exists only in the tuple header. A second session that wants the row finds 768 in t_xmax, sees it is still in progress, and waits on the transaction ID lock instead.

The stamp outlives the lock

Commit session B and read the page again:

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                            raw_flags
----+--------+--------+--------+--------------------------------------------------------------------------------------------------
  1 |    767 |    768 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_KEYS_UPDATED}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
(3 rows)

Nothing changed. I'd assumed commit would tidy the tuple up, but COMMIT doesn't revisit the pages a transaction locked; that would mean remembering every one of them. The lock is released because 768 is no longer in progress, and anybody who looks at the row will work that out for themselves by checking the transaction's status. The stamp sits there until something that cares about xmax comes past:

VACUUM lock_demo;

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                                     raw_flags
----+--------+--------+--------+--------------------------------------------------------------------------------------------------------------------
  1 |    767 |    768 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID,HEAP_KEYS_UPDATED}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}
(3 rows)

VACUUM checks lock-only xmax values too, finds 768 finished, and sets the hint, which is enough to tell every future visitor to ignore the rest of the bits. The next locker overwrites them.

Four modes, four bit patterns

PostgreSQL has four row lock strengths, encoded in three bits across the two infomask fields. Repeat the experiment with each of the other three modes, one per row, each in its own transaction that session B holds open while session A reads the page, then commits before the next:

-- session B, one at a time, each in its own open transaction
SELECT id FROM lock_demo WHERE id = 2 FOR NO KEY UPDATE;   -- xid 769
SELECT id FROM lock_demo WHERE id = 3 FOR SHARE;           -- xid 770
SELECT id FROM lock_demo WHERE id = 1 FOR KEY SHARE;       -- xid 771

 lp | t_xmax |                                   raw_flags
----+--------+--------------------------------------------------------------------------------
  2 |    769 | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED}

 lp | t_xmax |                                              raw_flags
----+--------+------------------------------------------------------------------------------------------------------
  3 |    770 | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED}

 lp | t_xmax |                                    raw_flags
----+--------+----------------------------------------------------------------------------------
  1 |    771 | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED}

The FOR KEY SHARE lock on row 1 overwrote the stale FOR UPDATE stamp completely. Side by side:

SQL lock mode	`KEYSHR_LOCK`	`EXCL_LOCK`	`KEYS_UPDATED`	Taken implicitly by
`FOR KEY SHARE`	1	0	0	foreign key checks
`FOR SHARE`	1	1	0	nothing
`FOR NO KEY UPDATE`	0	1	0	`UPDATE` that leaves key columns alone
`FOR UPDATE`	0	1	1	`UPDATE` of a key column, `DELETE`

The source calls the FOR SHARE combination HEAP_XMAX_SHR_LOCK, but it is not a separate bit. In practice, the last column explains most row-lock behaviour: every UPDATE that does not touch a key column (a column under a unique index that a foreign key could reference) takes FOR NO KEY UPDATE on the old row version, and every foreign key check takes FOR KEY SHARE. The two are compatible by design. That's what lets you update a customer's balance while someone else inserts an order for them.

Which modes block which:

Requested, versus held	`KEY SHARE`	`SHARE`	`NO KEY UPDATE`	`UPDATE`
`FOR KEY SHARE`				blocks
`FOR SHARE`			blocks	blocks
`FOR NO KEY UPDATE`		blocks	blocks	blocks
`FOR UPDATE`	blocks	blocks	blocks	blocks

FOR UPDATE conflicts with everything, so a DELETE waits behind a foreign key check but a balance update doesn't. The same conflict applies to explicit locks: an application that runs SELECT ... FOR UPDATE on a parent row before changing a non-key column blocks every child insert that references it. FOR NO KEY UPDATE would have done the job. FOR UPDATE should rarely be anyone's default, and it mostly is only because it's the one everybody remembers. Laurenz Albe covers the consequences in SELECT FOR UPDATE considered harmful in PostgreSQL.

Two lockers: the MultiXactId

t_xmax is one 32-bit field. Two sessions can hold FOR SHARE on the same row at the same time, and PostgreSQL has to record both. With the three mode transactions committed, take the lock in session B, then in a third session C, and read the page from session A:

-- session B (xid 772) and session C (xid 773), both left open
SELECT id FROM lock_demo WHERE id = 3 FOR SHARE;

-- session A
SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                                        raw_flags
----+--------+--------+--------+-------------------------------------------------------------------------------------------------------------------------
  1 |    767 |    771 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED}
  2 |    767 |    769 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED}
  3 |    767 |      1 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_EXCL_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMAX_IS_MULTI}
(3 rows)

Row 3's t_xmax of 1 isn't a transaction ID. HEAP_XMAX_IS_MULTI marks it as a MultiXactId, the first this cluster has created, pointing at a member list stored outside the page:

SELECT * FROM pg_get_multixact_members('1');

 xid | mode
-----+------
 772 | sh
 773 | sh
(2 rows)

The other mode values are keysh, fornokeyupd and forupd for the remaining lockers, and nokeyupd and upd for members that actually wrote a new row version. The infomask bits on the tuple summarise the strongest mode held by any member, so a compatible locker can be granted without walking the member list; deciding whether to wait still means reading it.

That structure is on disk, in two directories under the data directory:

$ ls -l $PGDATA/pg_multixact/offsets $PGDATA/pg_multixact/members
pg_multixact/members:
-rw------- 1 postgres postgres 8192 Sep 11 21:06 0000

pg_multixact/offsets:
-rw------- 1 postgres postgres 8192 Sep 11 21:06 0000

offsets maps a MultiXactId to a position in members; members is the flat list of (xid, mode) pairs. Both are SLRU files, the same family as the commit log. offsets grows by one entry per multixact, members by one entry per member.

Both lockers commit. As with the single-xid case, nothing on the page changes until VACUUM visits:

VACUUM lock_demo;

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                 raw_flags
----+--------+--------+--------+----------------------------------------------------------------------------
  1 |    767 |      0 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
(3 rows)

All members finished, none of them an updater, so the multixact carries no information any more. VACUUM replaced row 3's t_xmax with 0 instead of setting a hint. A multixact reference costs more to leave around than a plain xid: resolving it means a lookup in pg_multixact, and the multixact can't be recycled while any tuple points at it. HEAP_XMIN_INVALID next to HEAP_XMIN_COMMITTED means VACUUM also froze the page, which clearing an old multixact forces.

Inserting into lock_demo_tx locks a parent row without any FOR ... clause in the statement. The foreign key check has to make sure that alice's id still exists and does not change until the insert commits, and it does that by locking the parent row. Start an insert in session B and leave it open:

-- session B (xid 774)
BEGIN;
INSERT INTO lock_demo_tx (account_id, amount) VALUES (1, 25.00);

-- session A
SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                             raw_flags
----+--------+--------+--------+----------------------------------------------------------------------------------------------------
  1 |    767 |    774 | (0,1)  | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
(3 rows)

The foreign key check took a FOR KEY SHARE lock on alice's row, in a table the statement never wrote to.

Now, still with session B open, update alice's balance from session A, in autocommit mode. Balance is not a key column, so the UPDATE wants FOR NO KEY UPDATE, which doesn't conflict with FOR KEY SHARE:

-- session A, autocommit (xid 775)
UPDATE lock_demo SET balance = 150.00 WHERE id = 1;

UPDATE 1

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                                    raw_flags
----+--------+--------+--------+------------------------------------------------------------------------------------------------------------------
  1 |    767 |      2 | (0,4)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_IS_MULTI,HEAP_HOT_UPDATED}
  2 |    767 |      0 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  4 |    775 |    774 | (0,4)  | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_UPDATED,HEAP_ONLY_TUPLE}
(4 rows)

The old version at line pointer 1 has to record session B's key-share lock and session A's update in one field, so t_xmax became MultiXactId 2:

SELECT * FROM pg_get_multixact_members('2');

 xid |   mode
-----+----------
 774 | keysh
 775 | nokeyupd
(2 rows)

With an updater among the members, HEAP_XMAX_LOCK_ONLY is gone and this version has been dead since 775 committed. HEAP_HOT_UPDATED is there because the new version landed on the same page with no index change, the subject of the next chapter.

The UPDATE carried session B's key-share lock forward to the new version at (0,4), because session B still relies on alice's id not changing and the current row now lives there.

Try, from session C, to delete alice while the child insert is still uncommitted:

-- session C
SET statement_timeout = '2s';
DELETE FROM lock_demo WHERE id = 1;

ERROR:  canceling statement due to statement timeout
CONTEXT:  while deleting tuple (0,4) in relation "lock_demo"

DELETE needs FOR UPDATE strength, so it queued behind 774 on (0,4), the version carrying the inherited lock. While it was waiting, pg_locks did show a row, the one exception to it never showing row locks:

-- session A, while C is waiting
SELECT locktype, page, tuple, mode, granted FROM pg_locks WHERE locktype = 'tuple';

 locktype | page | tuple |        mode         | granted
----------+------+-------+---------------------+---------
 tuple    |    0 |     4 | AccessExclusiveLock | t
(1 row)

pg_locks shows you what a blocked session is waiting on, never what a session already holds at row level. That's also the answer to "why can't I delete this row, nothing has it locked". Something inserted a child row and hasn't committed yet, and this tuple entry plus the transactionid it is waiting for are the only traces.

A waiter takes a heavyweight tuple lock on the row it is queued for before sleeping on the holder's transaction ID, so that later waiters line up behind it in order rather than racing for the row when the holder finishes. It disappears the moment the waiter gets the row or gives up.

Once session B commits, a delete goes through on a different row to show the last pattern:

-- session C
DELETE FROM lock_demo WHERE id = 2;

SELECT lp, t_xmin, t_xmax, t_ctid,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('lock_demo', 0));

 lp | t_xmin | t_xmax | t_ctid |                                                    raw_flags
----+--------+--------+--------+------------------------------------------------------------------------------------------------------------------
  1 |    767 |      2 | (0,4)  | {HEAP_HASVARWIDTH,HEAP_XMAX_EXCL_LOCK,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_IS_MULTI,HEAP_HOT_UPDATED}
  2 |    767 |    777 | (0,2)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_KEYS_UPDATED}
  3 |    767 |      0 | (0,3)  | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  4 |    775 |    774 | (0,4)  | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_UPDATED,HEAP_ONLY_TUPLE}
(4 rows)

A delete sets HEAP_KEYS_UPDATED and no lock bits, because it has to conflict with every lock mode, key share included.

Multixacts on disk

Every multixact allocated is a permanent record until VACUUM says otherwise, so it matters how many a workload creates. Take a parent table with 20,000 rows and a child table referencing it, in the same shape as before:

CREATE TABLE mx_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    owner text NOT NULL,
    balance numeric(12,2)
);

CREATE TABLE mx_child (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    parent_id integer NOT NULL REFERENCES mx_demo (id)
);

INSERT INTO mx_demo (owner, balance)
SELECT 'user_' || i, i FROM generate_series(1, 20000) AS i;

SELECT relname, relminmxid FROM pg_class WHERE relname = 'mx_demo';

 relname | relminmxid
---------+------------
 mx_demo |          3

relminmxid is the oldest multixact any tuple in this table might still reference. A new table starts at the oldest multixact still in use, which on an idle cluster is the next one to be allocated; after the two created above, that's 3. Now have two transactions insert 5,000 child rows each for the same parents, overlapping in time:

-- session B (xid 781) and session C (xid 782), both left open
-- (778 to 780 went to the DDL above, 776 to the DELETE that timed out)
BEGIN;
INSERT INTO mx_child (parent_id) SELECT i FROM generate_series(1, 5000) AS i;

-- session A
SELECT lp, t_xmax,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('mx_demo', 0)) LIMIT 3;

SELECT * FROM pg_get_multixact_members('3');

 lp | t_xmax |                                              raw_flags
----+--------+-----------------------------------------------------------------------------------------------------
  1 |      3 | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMAX_IS_MULTI}
  2 |      3 | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMAX_IS_MULTI}
  3 |      3 | {HEAP_HASVARWIDTH,HEAP_XMAX_KEYSHR_LOCK,HEAP_XMAX_LOCK_ONLY,HEAP_XMIN_COMMITTED,HEAP_XMAX_IS_MULTI}
(3 rows)

 xid | mode
-----+-------
 781 | keysh
 782 | keysh
(2 rows)

A backend caches the multixacts it created or read in the current transaction (up to 256 entries, dropped at transaction end), so transaction 782 built {781 keysh, 782 keysh} once and reused it for all 5,000 rows. The counters confirm it after a checkpoint:

CHECKPOINT;
SELECT next_multixact_id, next_multi_offset FROM pg_control_checkpoint();

 next_multixact_id | next_multi_offset
-------------------+-------------------
                 4 |                 7

Offsets start at 1, so six members leave next_multi_offset at 7.

The expensive shape is the opposite one: a single long transaction holding key-share locks, and many short transactions each adding their own member. Commit session C, keep session B's 5,000-row insert open, and from session C run 5,000 separate one-row inserts, each its own transaction:

-- session C, autocommit, 5,000 times
INSERT INTO mx_child (parent_id) VALUES (1);
INSERT INTO mx_child (parent_id) VALUES (2);
...

SELECT * FROM pg_get_multixact_members('4');
SELECT * FROM pg_get_multixact_members('5');

 xid | mode
-----+-------
 781 | keysh
 784 | keysh
(2 rows)

 xid | mode
-----+-------
 781 | keysh
 785 | keysh
(2 rows)

Each short transaction forms a new member set with 781, so each row gets a new multixact (783 went to a stray statement):

CHECKPOINT;
SELECT next_multixact_id, next_multi_offset FROM pg_control_checkpoint();

SELECT pg_size_pretty(sum(size)) AS members_size
FROM pg_ls_dir('pg_multixact/members') f, pg_stat_file('pg_multixact/members/' || f) s;

 next_multixact_id | next_multi_offset
-------------------+-------------------
              5004 |             10007

 members_size
--------------
 56 kB

A long-running transaction that touches many parent rows, alongside a steady stream of short inserts, produces exactly this. A common source is a nightly batch job holding key-share locks on a hot reference table while the application keeps inserting. At roughly five bytes per member, the hard limit of 2³² members works out to about 20 GB.

Multixact IDs are 32-bit counters like transaction IDs, and they wrap around the same way:

SELECT relname, relminmxid, mxid_age(relminmxid)
FROM pg_class WHERE relname = 'mx_demo';

 relname | relminmxid | mxid_age
---------+------------+----------
 mx_demo |          3 |     5001

mxid_age is how far the counter has run ahead of the oldest multixact the table still references. VACUUM advances relminmxid the same way it advances relfrozenxid, by removing or replacing old multixacts in tuple headers, and it has its own set of thresholds for when to do so:

SELECT name, setting FROM pg_settings WHERE name LIKE '%multixact%age';

                name                 |  setting
-------------------------------------+------------
 autovacuum_multixact_freeze_max_age | 400000000
 vacuum_multixact_failsafe_age       | 1600000000
 vacuum_multixact_freeze_min_age     | 5000000
 vacuum_multixact_freeze_table_age   | 150000000

These mirror the transaction ID settings but track a separate counter, so a database can be nowhere near xid wraparound and still hit autovacuum_multixact_freeze_max_age. Autovacuum then starts an anti-wraparound vacuum you didn't schedule, on a table whose n_dead_tup is zero, which is confusing the first time you see it. Once session B commits, freezing clears it:

VACUUM FREEZE mx_demo;

SELECT relname, relminmxid, mxid_age(relminmxid)
FROM pg_class WHERE relname = 'mx_demo';

SELECT lp, t_xmax,
       (heap_tuple_infomask_flags(t_infomask, t_infomask2)).raw_flags
FROM heap_page_items(get_raw_page('mx_demo', 0)) LIMIT 3;

 relname | relminmxid | mxid_age
---------+------------+----------
 mx_demo |       5004 |        0

 lp | t_xmax |                                 raw_flags
----+--------+----------------------------------------------------------------------------
  1 |      0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  2 |      0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
  3 |      0 | {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMIN_INVALID,HEAP_XMAX_INVALID}
(3 rows)

The files in pg_multixact don't shrink yet. They are truncated once every database's datminmxid has moved past a segment. On this cluster that has not happened. datminmxid is the minimum relminmxid over a database's tables, and the system catalogs still carry the 1 they got from initdb, in every database including this one. Then there's template0, not connectable by default, whose value moves only when autovacuum's wraparound pass gets around to it.

The multixact members limit
When the 2³² member slots run out, any statement that needs a new multixact fails with multixact "members" limit exceeded. Autovacuum starts freezing multixacts more aggressively once half the space is used. On a system with many foreign key lockers, watch next_multi_offset in pg_control_checkpoint(), not just next_multixact_id.

PostgreSQL's MVCC is bad. So is everyone else's.

2026-07-27T16:01:00+00:00

The first thing you will probably learn about Postgres, if you follow people who don't like Postgres, is that MVCC is bad. The 40-year-old design mistake. It's signatures are everywhere. Bloated tables that double in size, 32-bit transaction counter limit, the never ending struggle with VACCUM, dead tuples nightmares. It comes with credentials, too: Uber measured the write amplification in 2016 and left for MySQL over it; Andy Pavlo's database group called MVCC the part of PostgreSQL they hate the most. It's a real thing. Postgres is as bad as it gets.

While none of this is exaggerated, it comes down to a real design choice. The bloat, the amplified writes, the vacuum babysitting: every charge traces to a decision, not a defect, and we reproduce each one below on a live PostgreSQL 19 beta2 instance, so you can watch the damage happen yourself. But the verdict that spreads from community to community always stops one question early: compared to what? What does every other engine do instead, and what does that cost?

Because MVCC is not optional. Any database that wants readers to not block writers has to keep multiple versions of rows somewhere, and every engine that does so answers the same four questions:

Where do old versions live? In the table itself, or in a separate structure?
Which way do version chains point? From old to new, or new to old?
What do indexes point at? A physical row location, or a logical key?
Who cleans up, and when? A background process later, or the transaction itself?

PostgreSQL's answers: in the table, old to new, physical location, background process later. Every cost the critics list follows from those four answers. And every alternative is a different set of answers with the bill sent to someone else, the writer, the reader of history, tempdb, the cache, the compactor. One of them spent years of engineering to buy the one property PostgreSQL's design has had for free since day one. All of them fail, differently, when a transaction stays open over lunch.

The charges against PostgreSQL

If you want the full mechanism, PostgreSQL MVCC, Byte by Byte walks through it with pageinspect. The short version: an UPDATE in PostgreSQL never modifies a row. It writes a complete new copy of the row into the heap, stamps the old version's t_xmax, and leaves both versions sitting on disk. Visibility is decided at read time, tuple by tuple. Cleanup is somebody else's problem, specifically VACUUM's.

Four charges follow, reproduced one at a time below.

Charge 1: write amplification

The heart of Uber's complaint. Because every index on the table points at the row's physical location (a page number plus a slot, the ctid), and an UPDATE creates a new physical row, every index needs a new entry pointing at the new location. Even indexes on columns you didn't touch.

Set up two copies of the same 1M-row table. One with just a primary key, one with four additional secondary indexes:

CREATE TABLE accounts (
  id bigint PRIMARY KEY,
  email text NOT NULL,
  status text NOT NULL,
  balance numeric NOT NULL,
  created_at timestamptz NOT NULL,
  last_seen timestamptz
);

INSERT INTO accounts
SELECT g, 'user' || g || '@example.com', 'active', 100, now(), now()
FROM generate_series(1, 1000000) g;

CREATE INDEX ON accounts (email);
CREATE INDEX ON accounts (status);
CREATE INDEX ON accounts (balance);
CREATE INDEX ON accounts (created_at);

CREATE TABLE accounts_lean (LIKE accounts);
ALTER TABLE accounts_lean ADD PRIMARY KEY (id);
INSERT INTO accounts_lean SELECT * FROM accounts;

Now update 100,000 rows, touching only last_seen. Note that last_seen is not in any index, on either table. Measure the WAL generated (pg_stat_wal, after a CHECKPOINT and pg_stat_reset_shared('wal') to get a clean window):

UPDATE accounts_lean SET last_seen = now()
WHERE id > 100000 AND id <= 200000;

 wal_records | wal_fpi | wal_bytes | wal_pretty
-------------+---------+-----------+------------
      302510 |    1419 |  38218531 | 36 MB

UPDATE accounts SET last_seen = now()
WHERE id > 100000 AND id <= 200000;

 wal_records | wal_fpi | wal_bytes | wal_pretty
-------------+---------+-----------+------------
      709440 |    1981 |  72394573 | 69 MB

Same logical change, 100,000 timestamps. The lean table produced about 3.0 WAL records per row. The indexed table produced 7.1: the heap update, plus one new entry in the primary key, plus one new entry in each of the four secondary indexes, none of which index the column we changed. They all got rewritten anyway, because the row moved and they all point at physical locations.

This is the amplification Uber measured, and it compounds: more WAL means more checkpoint work, more full-page writes, and more bytes shipped to every replica. A single-column update on a heavily indexed table is one of the most expensive things you can do per byte of useful change.

PostgreSQL's mitigation is the HOT update (Heap-Only Tuples): if no indexed column changed and the new version fits on the same page, indexes are left alone. last_seen isn't indexed, so that condition held; the pages were just packed full, and every new version landed on a different page. Give the table breathing room and try again:

ALTER TABLE accounts SET (fillfactor = 70);
VACUUM FULL accounts;

UPDATE accounts SET last_seen = now()
WHERE id > 200000 AND id <= 300000;

 wal_records | wal_bytes | n_tup_hot_upd (this batch)
-------------+-----------+----------------------------
      455812 |     51 MB | 41,996 of 100,000

Better, but note the number: 42% HOT, not 100%. Each page has 30% free space and roughly 80 rows; once the first ~30 updates on a page consume the reserve, the rest of that page's rows spill elsewhere, indexes and all. Run the same batch a second time, after opportunistic pruning has recycled the dead versions in-page:

 wal_records | wal_bytes | n_tup_hot_upd (this batch)
-------------+-----------+----------------------------
      304990 |     22 MB | 66,314 of 100,000

66% HOT and 3.0 WAL records per row, nearly back to lean-table cost. That's the honest shape of the mitigation: HOT is opportunistic, arrives in steady state rather than on demand, and quietly costs you 30% of your table size in reserved space. It works, and OLTP tables in the wild routinely see 90%+ HOT ratios with well-chosen fillfactor. But the default behavior is the 7.1 records per row, and the critics are describing the default.

Charge 2: your table is an accident scene

Every UPDATE is an INSERT plus a deferred DELETE. Do nothing about it and the dead versions accumulate:

VACUUM (FULL, ANALYZE) accounts_lean;  -- reset: 89 MB, 0% dead

\timing on
BEGIN;
UPDATE accounts_lean SET balance = balance + 1;  -- all 1M rows
ROLLBACK;

SELECT pg_size_pretty(pg_relation_size('accounts_lean'));
SELECT tuple_count, dead_tuple_count, round(dead_tuple_percent)
FROM pgstattuple('accounts_lean');

BEGIN
Time: 0.032 ms
UPDATE 1000000
Time: 1590.112 ms (00:01.590)
ROLLBACK
Time: 0.124 ms

 pg_size_pretty
----------------
 178 MB

 tuple_count | dead_tuple_count | dead_pct
-------------+------------------+----------
     1000000 |          1000000 |       47

One statement, and a transaction that didn't even commit, doubled the table. A million dead tuples wait for vacuum, and as VACUUM Is a Lie covers, the indexes that grew along the way will never shrink back on their own. This is the bloat treadmill: the storage design guarantees garbage is created at write speed and collected at vacuum speed, and keeping those two rates matched is an operations job that PostgreSQL outsources to you, with autovacuum's defaults as the starting point.

Charge 3: one idle transaction poisons the well

Keep the experiment going. In a second session, open a transaction and just... hold a snapshot:

-- session B
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT count(*) FROM accounts_lean;
-- go to lunch

Back in session A, update 100,000 rows and vacuum:

INFO:  finished vacuuming "mvcc_critics.public.accounts_lean":
tuples: 14 removed, 1099945 remain,
        100000 are dead but not yet removable
removable cutoff: 718, which was 1 XIDs old when operation ended

"Dead but not yet removable." Session B's snapshot might still need those versions, so vacuum must preserve them, and not just in this table: the xmin horizon is held back for every table in the database, whether session B ever reads them or not. PostgreSQL will never cancel the reader and never make it fail.Strictly true at the MVCC level, but production setups routinely add that cancellation back in from outside: idle_in_transaction_session_timeout and statement_timeout exist specifically to kill the lunch-break transaction before it holds back the xmin horizon. It just silently stops collecting garbage, database-wide, for as long as the transaction idles. Every Postgres operator eventually learns to hunt long-running transactions and stale replication slots.

Charge 4: the 32-bit debt

Every tuple header stores its creator transaction ID in 32 bits. Four billion transactions and the counter wraps, so PostgreSQL must periodically freeze old tuples, rewriting pages that haven't been touched in months, purely to reclaim XID space. Fall too far behind and the database stops accepting writes to protect itself. Ask Sentry, who wrote up their 2015 wraparound outage, or the several companies that have re-learned it since. But only PostgreSQL makes you periodically rewrite untouched pages to keep the clock from running out.

To be fair, Oracle's counter is finite too: the SCN was 48 bits for decades, and Oracle DBAs spent 2019 patching for SCN headroom after database links were found to jump counters forward. The widening came first, and caused the scramble: 12.2 grew the SCN to 64 bits ("big SCN") back in 2016, and the 2019 patches were what let older databases accept the larger values a linked 12.2 could now send, at the price of a version-compatibility dance across database links.

So: charges stick. Amplified writes by default, garbage created at write speed and collected by a process you have to tune, cleanup held hostage by any idle snapshot, and a 32-bit clock that must be wound forever. PostgreSQL's MVCC is bad, in four specific and demonstrable ways.

The undo camp: Oracle and InnoDB

Oracle and MySQL's InnoDB answer the four questions the opposite way: versions live outside the table, chains point new to old, cleanup is partly the writer's problem. This is the design most "Postgres MVCC is bad" articles implicitly hold up as the fix, so it deserves the closest look.

An UPDATE in InnoDB modifies the row in place, inside the clustered index (in InnoDB, the table is a B-tree ordered by primary key). Before overwriting, it copies the old values into an undo log, a separate storage area, and stamps the row with a pointer to that undo record. Every row carries two hidden system columns for this: DB_TRX_ID, the last transaction to touch it, and DB_ROLL_PTR, the pointer into undo. Oracle's structure differs in detail (undo segments, block-level consistent reads) but the shape is identical: the table holds only the newest version, and history hangs off it in a chain of deltas.

A reader with an older snapshot finds the current row, notices DB_TRX_ID is too new, and walks the undo chain backwards, applying deltas until it reconstructs the version it's entitled to see. Oracle does this at block granularity, rebuilding a whole consistent image of the block in memory.

Look at what this buys, point by point against the four charges:

The table stays compact. One version per row, ever. No bloat treadmill, no pgstattuple showing 47% dead. Undo is recycled in a fixed-size ring (Oracle) or purged continuously (InnoDB).
Secondary indexes are spared. InnoDB's secondary indexes point at the primary key, a logical reference; Oracle's secondary indexes store the ROWID, a physical address, but Oracle keeps that address stable during in-place updates and row migration. That way updating last_seen leaves the four secondary indexes untouched. This single decision is most of the write-amplification gap Uber measured.
No vacuum. Cleanup is woven into normal operation: InnoDB's purge threads trim undo history continuously, Oracle recycles undo segments.
No wraparound rituals. InnoDB's transaction IDs are 48-bit and Oracle's SCN is 64-bit since 12.2 (the 48-bit era ended with the headroom scare from charge 4), wide enough that neither engine ever has to freeze a tuple.

Four for four. Case closed?

One caveat before we count the wins: Oracle's default READ COMMITTED takes a snapshot at the start of each statement, not at the start of the transaction. An idle transaction does not pin undo the way a PostgreSQL REPEATABLE READ transaction does. The undo pressure, and the risk of ORA-01555, comes from a long-running statement, an open cursor, or an explicit SERIALIZABLE or READ ONLY transaction.

One caveat to "in place": it only works while the new version fits in the block. Oracle leaves headroom at insert time with PCTFREE (default 10%). If the update fits there, the row expands in place and indexes are untouched. If it does not fit, Oracle moves the row to another block and leaves a forwarding pointer in the original slot. The ROWID stays the same, so the indexes still need no maintenance.

The cost moves to reads: an index lookup follows the pointer to a second block. Enough growth-heavy updates create migrated rows, and the table is eventually reorganized (ALTER TABLE ... MOVE). PostgreSQL pays expansion as write amplification; Oracle pays it as read amplification and maintenance.

Here is the bill.

Rollback costs what the transaction cost. PostgreSQL rolled back a one-million-row UPDATE in 0.124 milliseconds, because in a heap-versioning system "abort" means doing nothing: the old versions are already there. (It took 1.59 seconds to make the mess and 0.124 milliseconds to disown it; the mess remains, for vacuum, but your transaction is done.) In an undo-based engine, rollback walks the undo chain and reverses every change one row at a time, at roughly the cost of the forward direction, sometimes more. Every MySQL operator eventually kills a long-running UPDATE and then waits longer for the rollback than the statement ran, because the undo has to be applied; crash recovery replays the same job before the engine is fully healthy. It is precisely to escape this that SQL Server later grew a Postgres-style version store, but we're getting ahead of ourselves.

Readers pay for that history at read time, too. In PostgreSQL, an old row version stays in the heap. Reading it costs the same as reading anything else. When a reader accesses a recently committed tuple, it may first check pg_xact to set hint bits. This is a one-time cost per tuple, after which it remains in the heap. In the undo model, every version except the latest must be reconstructed for each read. This involves following pointers to another storage area and applying deltas. A long-running report on a hot table slows down as the snapshot gets older. This delay puts pressure on the buffer pool due to the undo pages.

The idle-snapshot problem didn't go away. It changed its failure mode. Undo space is finite. When a long-running query needs a version whose undo has already been recycled, Oracle doesn't pause cleanup like PostgreSQL does. It kills the query: ORA-01555: snapshot too old, one of the most familiar error codes in databases. InnoDB makes the opposite choice within the same design: purge waits for the oldest read view, history piles up (watch the history list length climb in SHOW ENGINE INNODB STATUS), and undo tablespaces balloon.MySQL 8's automatic undo truncation does shrink tablespaces whose logs have gone inactive, but the long read that caused the growth is exactly what keeps the logs active, so the garbage stays put for as long as the reader does. That is bloat, relocated to a place pgstattuple can't see. Same fundamental tension, choose your poison: cancel the reader, or keep the garbage.

InnoDB's purge threads are, functionally, vacuum under another name: they scan undo history, remove delete-marked index records, and can fall behind under write bursts exactly the way autovacuum can. The war-story threads on the MySQL side (history list length in the millions, purge lag stalls) read like Postgres bloat threads with the nouns changed.

So the undo camp didn't eliminate MVCC's costs. It moved them: away from the future (background cleanup, table bloat) and onto the writer's critical path (rollback, in-place update bookkeeping) and the reader of old data (reconstruction). If your workload commits nearly always, updates heavily indexed rows, and rarely runs long reads against hot tables, that trade is genuinely better, that is, roughly, the workload Uber had.

SQL Server: versioning as an optional extra

SQL Server is the reminder that MVCC itself was a choice. Its default READ COMMITTED isn't versioned at all: readers take shared locks, writers take exclusive locks, and they block each other. The concurrency problems MVCC exists to solve are simply... allowed to happen, managed with lock hints and shorter transactions. For years the standard fix for "my readers are blocked" was WITH (NOLOCK), which is opting into reading uncommitted rows.

Since 2005, versioning is available by opt-in (READ_COMMITTED_SNAPSHOT or full SNAPSHOT isolation). Flip it on and SQL Server starts copying pre-update row versions into a version store that lives in tempdb, stamping each modified row with a version pointer.The pointer is 14 bytes, added to every versioned row, and used to chase that row's chain back through tempdb. It's an undo-style design bolted onto a locking engine, with one memorable operational property: the version store is shared, instance-wide, and lives in the same tempdb as everyone's sort spills and temp tables. One forgotten long-running snapshot transaction and the version store cannot be trimmed; tempdb grows until it runs out of disk, at which point the entire instance starts failing queries, including databases that had nothing to do with it. The idle-snapshot problem again, with the widest blast radius of any design in this post.

The most interesting SQL Server development is recent: Accelerated Database Recovery (2019, on by default in Azure SQL) added a persistent version store, and note where it lives: inside the user database, not tempdb. Row versions move into a per-database PVS, which both shrinks the instance-wide blast radius described above and makes rollback and crash recovery instant regardless of transaction size, reading old versions back instead of replaying undo. The flagship locking-and-undo engine spent significant engineering effort to acquire the one property, constant-time abort, that PostgreSQL's much-maligned design has had for free since day one. Trade-offs run in both directions.

MongoDB: versions in the cache

MongoDB's WiredTiger engine answers the four questions with an accent you haven't seen yet: old versions live in RAM. A read runs against a snapshot taken when it starts; a writer appends its change as an in-memory delta hanging off the page's current image, and readers walk the chain back to the version their snapshot is entitled to see. Checkpoints periodically flush clean page images to disk, a journal provides durability, and, note this, coming out of the undo section, rollback is free: uncommitted versions were never on disk, so aborting a transaction means discarding memory. WiredTiger has had constant-time abort since day one, without Microsoft's engineering program.

The design buys real things: no dead tuples on disk, no undo tablespace to size, readers never blocked. The bill arrives through a single fact: the cache is finite, shared, and cannot evict a version that some snapshot still needs.

The oldest snapshot pins history. Versions pushed out of the cache while still visible to an old reader spill into a history store (WiredTigerHS.wt), which grows for exactly as long as the pinning reader lives. InnoDB's history list, as a file: bloat, relocated somewhere you have to know to look.
Under pressure, everyone pays. Let the cache fill with dirty or pinned content and eviction can't keep up; WiredTiger then recruits application threads into eviction work, and the failure mode is not an error but a latency smear across every operation on the node, readers included. In the worst cases the engine starts rolling transactions back for you, cache full. The idle-snapshot problem, failing by slowdown instead of by bloat.
Snapshot too old, again. Ask for a read at a timestamp older than the pinned history can serve and MongoDB answers with SnapshotTooOld. Third engine in this post to have that error.
The physical-pointer charge survives. Documents are indexed by RecordId, a physical slot; grow a document past its space and it moves, and every secondary index updates to follow it. Uber's complaint, in a document store.

Cleanup is eviction plus checkpoints: vacuum, running continuously, in RAM, mostly invisible, until a snapshot stays open over lunch.

The LSM generation: garbage collection as a lifestyle

The distributed SQL newcomers, CockroachDB and YugabyteDB most prominently, store data in LSM trees (RocksDB or derivatives), and their MVCC answer is different again: a row version is simply a key, the logical key suffixed with a commit timestamp. Updates never touch anything in place, they just write a newer key. Reads scan for the newest version at or below their snapshot timestamp. Deletes write a tombstone key.

In one sense this is PostgreSQL's philosophy taken further: append versions, never modify, clean later. The cleanup story differs. Old versions are removed by compaction, the LSM's normal background rewriting of storage layers, governed by a garbage-collection window: versions younger than the TTL are kept for time-travel reads and follower reads; CockroachDB's default window has historically been measured in hours (originally 25, since reduced). No vacuum command exists, and the marketing will mention that. But compaction is vacuum, running continuously, paying the same deferred-cleanup tax as write amplification inside the storage engine, and the failure modes rhyme with everything above:

Read too far in the past, below the GC window, and the query fails. Snapshot too old, rediscovered.
Heavy update or delete churn on a narrow key range leaves layers of tombstones and stale versions that every scan must wade through until compaction catches up. The classic symptom is the queue table that gets slower the emptier it is, an antipattern LSM users know as intimately as Postgres users know bloated tables.

Nothing was solved. The four questions got new answers, and the costs moved into the compactor.

Outside the RDBMS entirely: etcd

Leave the relational world and the same four questions reappear, verbatim. etcd, the consensus-backed key-value store that holds the desired state of every Kubernetes cluster, is MVCC by name and by design. Every write bumps a cluster-wide revision counter, and values are stored under (key, revision) in a bbolt file. An update never overwrites; it appends a newer revision of the key. A read at a given revision gets the newest version at or below it. A delete writes a tombstone. If that sounds like the LSM section you just read, it should. And if it sounds like the four charges at the top of this post, keep reading.

Versions live in the store; cleanup is a separate job. Old revisions sit in the same backend file as current data until compaction removes them, run by hand (etcdctl compact) or on a schedule (--auto-compaction-retention). Skip it and the backend grows without bound: garbage created at write speed, collected at compaction speed. The bloat treadmill, minus the SQL.
Cleanup doesn't return the disk. bbolt, like PostgreSQL's heap, never shrinks its file; compacted pages go to an internal freelist for reuse. Getting the space back means etcdctl defrag, run member by member, each node briefly stalling while it rewrites. VACUUM FULL, one quorum member at a time.
"Snapshot too old," rediscovered verbatim. Read or watch from a revision older than the compaction point and etcd fails you: etcdserver: mvcc: required revision has been compacted. ORA-01555 again, surfacing in every Kubernetes cluster whose controllers resume a watch from a stale resourceVersion.
Exceed the default 2GB backend quota and etcd raises NOSPACE and refuses writes, including the Kubernetes control plane's, until a human compacts, defrags, and disarms the alarm. It's the wraparound outage's closest non-relational cousin: a safety limit that turns a storage-hygiene chore into an outage, with the babysitting outsourced to the operator.

etcd answered the four questions almost exactly the way PostgreSQL did: append versions, never modify, clean later, never cancel a reader early. It inherited the same operations calendar. The questions predate the relational model's dominance and will outlive it; there is no escaping the bill, only choosing who pays it.

Its distributed-KV neighbor FoundationDB answers the lunch question more brutally than anyone in this post: transactions have a hard five-second lifetime, full stop. Hold a transaction open over lunch and you don't hold back a vacuum, a purge, or a compactor, your transaction is simply dead. It's the one design here that made the idle-transaction problem the client's problem, by fiat.

Fixing PostgreSQL itself

If the undo design were clearly better, we should simply use an undo-based storage engine for PostgreSQL. People tried. zheap, announced in 2018, was exactly that: in-place updates with undo logs, targeting the bloat problem head-on. It led to years of work at EnterpriseDB on a general undo framework for core. Then, it found a second life at CYBERTEC, but now the code is mostly dormant. The point isn’t that Álvaro Herrera from EDB outlined Postgres for 2029 in his PGConf.DE 2026 keynote. The main request for in-core table access was "zheap (no need for vacuum)." Note the verb tense, though: could have, in ten years. The key lesson from zheap is its simple design is crucial. Once versions exist in undo, every subsystem—like recovery, replication, indexes, and hot standby—adopts new rules. This means you might need to rebuild half the engine just to change how the other half stores rows.

The attempt continues outside core. OrioleDB (the engine by Alexander Korotkov, now part of Supabase) blends undo-based versioning with index-organised tables and row-level WAL. It shows impressive benchmark results, proving that a fresh approach can be beneficial. However, it still needs patches to core PostgreSQL, which are being integrated upstream. The table access method API (PostgreSQL 12) allows alternatives to the heap. Also, 64-bit transaction IDs could prevent wraparound completely. They have a long-running patchset and are included in at least one commercial fork.

Meanwhile, core PostgreSQL has spent fifteen years shrinking the constants without changing the design. HOT updates arrived in 8.3. The visibility map that lets vacuum skip clean pages came in 8.4 and improved in 9.6. B-tree deduplication landed in 13, and bottom-up index deletion that fights version churn inside index pages came in 14. Version 17 updated the vacuum's TID store. It removed the memory limit that used to require multi-pass index vacuums. Version 19 added REPACK, consolidating VACUUM FULL and CLUSTER with a CONCURRENTLY mode that rebuilds without the exclusive lock. The four charges from the top of this post were all worse in 2010. The design held; the bill shrank.

The scorecard

Five storage designs (etcd maps onto the PostgreSQL column), the same four questions, and no column without a failure mode:

	PostgreSQL heap	Oracle / InnoDB undo	SQL Server + RCSI	WiredTiger (MongoDB)	LSM (CockroachDB etc.)
Old versions live	in the table	undo segments/logs	tempdb version store	in the cache, as delta chains (spill to history store)	in the LSM, timestamped keys
Update writes	full new row + all index entries (unless HOT)	in place + undo delta; unchanged secondary indexes untouched	in place + version copy to tempdb	in-memory delta; page rewritten at checkpoint; moved documents update all indexes	new key version
Secondary indexes point at	physical ctid; every index follows every moved version (HOT excepted)	logical PK (InnoDB), rowid (Oracle); stable under in-place update	clustering key / RID; rows gain a 14-byte version tag	RecordId; a moved document rewrites every index	logical keys; each version is a new key, timestamp-suffixed
Abort cost	~zero	proportional to work done	~zero with ADR (2019+); undo replay before	~zero (uncommitted deltas discarded)	~zero (intents removed lazily)
Long reader causes	bloat, vacuum starvation, held xmin	ORA-01555 or undo/history growth	tempdb growth, instance-wide risk (ADR moves versions into the user DB)	history store growth, cache pressure, eviction stalls	GC-window errors, tombstone debt
Cleanup	vacuum (yours to tune)	purge/undo recycling (theirs, still stalls)	version store cleanup	eviction + checkpoints (continuous, in RAM)	compaction (continuous)
Signature outage	wraparound, bloat	history list explosion, day-long rollback	tempdb full	node-wide latency smear; SnapshotTooOld	tombstone-choked scans

The costs of multi-versioning, keeping history, reading history, discarding history, are conserved. Each engine only chooses who pays, when, and how it fails: PostgreSQL charges the future and fails by bloating; the undo camp charges the writer and the reader-of-history, and fails by cancelling queries or ballooning undo; SQL Server charges tempdb and fails instance-wide; WiredTiger charges the cache and fails by slowing everyone at once; the LSM engines charge the compactor and fail on time-travel and churn.

PostgreSQL's particular choice, viewed from this side of the table, is the one that never blocks a reader, never cancels a query with "snapshot too old" (unless you opt in, and even that was removed in 17 for lack of love), never makes you wait for a rollback, and never hides its garbage where you can't inspect it. Every dead tuple is sitting in an 8KB page you can open with pageinspect. The price is that the garbage is yours: visible, measurable, and on your maintenance calendar.

So yes, PostgreSQL's MVCC is bad. So is everyone else's. Postgres chose operationally loud failure modes and manually tuned mitigations, which is why zheap was attempted and why OrioleDB exists. But if someone tells you a database has solved multi-versioning, ask them the four questions: where do old versions live, which way do chains point, what do indexes point at, and who cleans up. Then ask what happens when a transaction stays open over lunch.

UPDATED 2027-07-29: Added the Oracle row-migration caveat after consulting with a reader about what was worth noting.

UPDATED 2027-07-30: Added a callout clarifying that Oracle's default READ COMMITTED uses statement-level snapshots, not transaction-level snapshots.

UPDATED 2026-08-05: Fixed to HOT-update paragraph (failure conditions) and correct 64-bit SCN chronology.

VACUUM at the Page Level

2026-07-05T17:13:00+00:00

In HOT Updates in Postgres we covered page pruning clean up HOT chains, an elegant shortcut where PostgreSQL reclaims dead tuple space during ordinary reads. All that without waiting for any background process. But pruning is exactly that: a shortcut. It only works within a single page, and only for HOT-updated tuples. For everything else (cold updates that touch indexed columns, plain DELETEs, index entry cleanup, free space map registration, visibility map maintenance) we need VACUUM.

This article won't repeat what VACUUM does operationally. The DELETEs are difficult article covers autovacuum tuning, worker allocation, and the operational side of dead tuple cleanup. Here we are going to watch VACUUM work byte by byte. We'll snapshot a page before and after each phase, tracking exactly what changes in the page header, line pointers, tuple headers, free space map, and visibility map. Same tools as always: pageinspect, pg_visibility, and pg_freespacemap.

Setup

We need a table with enough rows to make the before-and-after comparison meaningful, plus indexes to demonstrate the full VACUUM cycle.

CREATE EXTENSION IF NOT EXISTS pageinspect;
CREATE EXTENSION IF NOT EXISTS pg_visibility;
CREATE EXTENSION IF NOT EXISTS pg_freespacemap;

CREATE TABLE vacuum_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    category text NOT NULL,
    payload text
);

INSERT INTO vacuum_demo (category, payload)
SELECT
    'cat_' || (i % 5),
    repeat('x', 100)
FROM generate_series(1, 50) AS i;

Fifty rows with a 100-byte payload each. The primary key gives us an index, which matters: VACUUM's behavior changes when indexes are involved. Run VACUUM once upfront so we start from a clean baseline:

VACUUM vacuum_demo;

Snapshot before any deletes

Record the baseline state of page 0. First the page header:

SELECT lower, upper, special, pagesize
FROM page_header(get_raw_page('vacuum_demo', 0));

 lower | upper | special | pagesize
-------+-------+---------+----------
   224 |  1392 |    8192 |     8192
(1 row)

pd_lower is at 224: that's the 24-byte page header plus 50 line pointers at 4 bytes each (24 + 200 = 224). pd_upper is at 1392, so our tuples occupy bytes 1392 through 8191. Free space is 1392 - 224 = 1168 bytes. Not much room left; those 100-byte payloads add up.

Now the line pointers and tuple headers:

SELECT lp, lp_flags, lp_off, lp_len, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('vacuum_demo', 0))
LIMIT 10;

 lp | lp_flags | lp_off | lp_len | t_xmin | t_xmax | t_ctid
----+----------+--------+--------+--------+--------+--------
  1 |        1 |   8056 |    135 |    746 |      0 | (0,1)
  2 |        1 |   7920 |    135 |    746 |      0 | (0,2)
  3 |        1 |   7784 |    135 |    746 |      0 | (0,3)
  4 |        1 |   7648 |    135 |    746 |      0 | (0,4)
  5 |        1 |   7512 |    135 |    746 |      0 | (0,5)
  6 |        1 |   7376 |    135 |    746 |      0 | (0,6)
  7 |        1 |   7240 |    135 |    746 |      0 | (0,7)
  8 |        1 |   7104 |    135 |    746 |      0 | (0,8)
  9 |        1 |   6968 |    135 |    746 |      0 | (0,9)
 10 |        1 |   6832 |    135 |    746 |      0 | (0,10)
(10 rows)

lp_len is 135, the tuple's actual byte length, but each tuple occupies MAXALIGN'd slot of 136 bytes on the page; notice the lp_off values step down by 136. That aligned stride is what the free-space arithmetic below uses.

Every line pointer is LP_NORMAL (lp_flags = 1). Every tuple has t_xmax = 0: nobody has touched these rows since they were inserted. Every t_ctid points to itself. This is a perfectly clean page.

Create dead tuples

Now make some rows dead:

DELETE FROM vacuum_demo WHERE id % 3 = 0;

DELETE 16

That deletes roughly every third row: IDs 3, 6, 9, 12, and so on. Sixteen rows are now dead. Look at the page before VACUUM runs:

SELECT lp, lp_flags, lp_off, lp_len, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('vacuum_demo', 0))
LIMIT 10;

 lp | lp_flags | lp_off | lp_len | t_xmin | t_xmax | t_ctid
----+----------+--------+--------+--------+--------+--------
  1 |        1 |   8056 |    135 |    746 |      0 | (0,1)
  2 |        1 |   7920 |    135 |    746 |      0 | (0,2)
  3 |        1 |   7784 |    135 |    746 |    747 | (0,3)
  4 |        1 |   7648 |    135 |    746 |      0 | (0,4)
  5 |        1 |   7512 |    135 |    746 |      0 | (0,5)
  6 |        1 |   7376 |    135 |    746 |    747 | (0,6)
  7 |        1 |   7240 |    135 |    746 |      0 | (0,7)
  8 |        1 |   7104 |    135 |    746 |      0 | (0,8)
  9 |        1 |   6968 |    135 |    746 |    747 | (0,9)
 10 |        1 |   6832 |    135 |    746 |      0 | (0,10)
(10 rows)

Look at rows 3, 6, and 9. Their t_xmax is now 747, the transaction ID of the DELETE statement. But everything else is unchanged. lp_flags is still 1 (LP_NORMAL). lp_off and lp_len are the same. The tuples are still physically sitting on the page, consuming space. The page header hasn't changed either:

SELECT lower, upper, special, pagesize
FROM page_header(get_raw_page('vacuum_demo', 0));

 lower | upper | special | pagesize
-------+-------+---------+----------
   224 |  1392 |    8192 |     8192
(1 row)

pd_lower and pd_upper are identical to before the DELETE. PostgreSQL marked the rows as dead (by stamping t_xmax) but did not reclaim a single byte. The dead tuples are bloat, and they will stay that way until VACUUM arrives.

How VACUUM processes a table

Before we run VACUUM and watch the page change, one thing about how it runs will explain everything the snapshots are about to show. VACUUM does its work in three passes, and the split between them is the whole reason a deleted tuple's storage disappears at one moment and its line pointer disappears at another. Keep an eye on that gap: it is what the rest of this article makes visible.

Phase 1: Heap scan - prune, freeze, collect dead TIDs

VACUUM scans every heap page sequentially, and this first pass does far more than look. For each page, it runs page pruning (the same heap_page_prune_and_freeze machinery that fires opportunistically during ordinary reads). Pruning is where the bytes actually come back: it removes the storage of dead tuples, defragments the page, and advances pd_upper. It also opportunistically freezes tuples that are old enough.

Prior to PostgreSQL 17, VACUUM stored dead TIDs in a flat array allocated from maintenance_work_mem. If the array filled up, VACUUM had to pause, perform index and heap cleanup for the collected batch, then resume scanning. Since PostgreSQL 17, VACUUM uses a radix tree-based TID store that is far more memory-efficient, making maintenance_work_mem much less likely to be a bottleneck.

But here is the subtlety that the rest of this article hinges on. Pruning cannot simply mark a deleted tuple's line pointer LP_UNUSED, because indexes still point at it by TID. So for a table with indexes, a dead tuple's line pointer is set to LP_DEAD: its storage is gone, but the 4-byte slot stays put, holding the TID's place until the index entries are removed. Those LP_DEAD TIDs are what VACUUM collects into its dead-TID store for the next phase.

Phase 2: Index cleanup

With the list of dead TIDs in hand, VACUUM scans each index on the table. For every index, it walks through all index entries and removes any that point to a dead TID. This is the expensive part. VACUUM must read every index page, even if only handful of entries need removal.

This is also why index bloat happens. If VACUUM cannot finish this phase, because a long-running transaction is holding back the visibility horizon or the table has many indexes and the dead TID list exceeds memory, index entries pointing to dead tuples accumulate. We covered the index bloat implications in detail in VACUUM Is a Lie.

Phase 3: Heap cleanup - freeing the line pointers

After the indexes are clean, no index entry references those dead TIDs anymore, so the reserved slots can finally be released. VACUUM revisits each page that had dead tuples and does what it couldn't do in phase 1:

Flips each collected LP_DEAD line pointer to LP_UNUSED (0), reclaiming the slot
Sets the visibility map bit if all remaining tuples are visible to everyone

A table with no indexes skips this split entirely: with no index entries to worry about, phase 1's prune sets dead line pointers straight to LP_UNUSED in a single heap pass, and there is no phase 3. The two-pass dance, LP_DEAD now and LP_UNUSED later, exists precisely because of the indexes.

Notice what is not on that list. The tuple data was already removed and the page already defragmented back in phase 1's prune; that is where pd_upper moved. Phase 3 reclaims the line-pointer slots, not the tuple bytes. The distinction is invisible if you only look before and after plain VACUUM, so let's make it visible.

Let's run VACUUM in two steps to catch the intermediate state. VACUUM (INDEX_CLEANUP OFF) performs phase 1's prune but skips index cleanup, and therefore skips the second heap pass too; it has no choice but to leave the dead line pointers as LP_DEAD.

After the prune: LP_DEAD, and space already back

Page header after pruning:

SELECT lower, upper, special, pagesize
FROM page_header(get_raw_page('vacuum_demo', 0));

 lower | upper | special | pagesize
-------+-------+---------+----------
   224 |  3568 |    8192 |     8192
(1 row)

pd_lower is still 224; the line pointer array hasn't shrunk. But pd_upper jumped from 1392 to 3568. That's 2176 bytes of reclaimed space (16 dead tuples at the 136-byte aligned stride = 2176). Free space on the page went from 1168 to 3344 bytes. And we have not touched an index yet: this all happened during pruning, in the first heap pass.

Now the line pointers:

SELECT lp, lp_flags, lp_off, lp_len, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('vacuum_demo', 0))
LIMIT 10;

 lp | lp_flags | lp_off | lp_len | t_xmin | t_xmax | t_ctid
----+----------+--------+--------+--------+--------+--------
  1 |        1 |   8056 |    135 |    746 |      0 | (0,1)
  2 |        1 |   7920 |    135 |    746 |      0 | (0,2)
  3 |        3 |      0 |      0 |        |        |
  4 |        1 |   7784 |    135 |    746 |      0 | (0,4)
  5 |        1 |   7648 |    135 |    746 |      0 | (0,5)
  6 |        3 |      0 |      0 |        |        |
  7 |        1 |   7512 |    135 |    746 |      0 | (0,7)
  8 |        1 |   7376 |    135 |    746 |      0 | (0,8)
  9 |        3 |      0 |      0 |        |        |
 10 |        1 |   7240 |    135 |    746 |      0 | (0,10)
(10 rows)

Line pointers 3, 6, and 9 are now lp_flags = 3 (LP_DEAD), not LP_UNUSED. Their storage is gone (lp_off and lp_len are 0), but the slots are still there, reserving those TIDs. They can't be freed yet, because the primary key still points at them:

SELECT live_items FROM bt_page_stats('vacuum_demo_pkey', 1);

 live_items
------------
         50
(1 row)

Fifty index entries, same as before the delete. The index cleanup we skipped is exactly what removes the sixteen stale ones, and until it runs, those LP_DEAD slots stay stuck.

After the full VACUUM: LP_UNUSED

Now run an ordinary VACUUM to finish the job:

VACUUM vacuum_demo;

SELECT lower, upper, special, pagesize
FROM page_header(get_raw_page('vacuum_demo', 0));

 lower | upper | special | pagesize
-------+-------+---------+----------
   224 |  3568 |    8192 |     8192
(1 row)

pd_upper is unchanged at 3568. That is the point: the second heap pass reclaims no tuple bytes; there are none left, pruning already took them. What it does is free the line-pointer slots, now that the index is clean:

SELECT lp, lp_flags, lp_off, lp_len, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('vacuum_demo', 0))
LIMIT 10;

 lp | lp_flags | lp_off | lp_len | t_xmin | t_xmax | t_ctid
----+----------+--------+--------+--------+--------+--------
  1 |        1 |   8056 |    135 |    746 |      0 | (0,1)
  2 |        1 |   7920 |    135 |    746 |      0 | (0,2)
  3 |        0 |      0 |      0 |        |        |
  4 |        1 |   7784 |    135 |    746 |      0 | (0,4)
  5 |        1 |   7648 |    135 |    746 |      0 | (0,5)
  6 |        0 |      0 |      0 |        |        |
  7 |        1 |   7512 |    135 |    746 |      0 | (0,7)
  8 |        1 |   7376 |    135 |    746 |      0 | (0,8)
  9 |        0 |      0 |      0 |        |        |
 10 |        1 |   7240 |    135 |    746 |      0 | (0,10)
(10 rows)

Line pointers 3, 6, and 9 have gone from LP_DEAD to lp_flags = 0 (LP_UNUSED). The slots are now empty and available for reuse by the next INSERT that lands on this page. And the index has dropped to 34 entries; the sixteen stale ones are gone:

SELECT live_items FROM bt_page_stats('vacuum_demo_pkey', 1);

 live_items
------------
         34
(1 row)

The surviving heap tuples were compacted back in the prune; notice the lp_off values shifted from their pre-VACUUM positions, VACUUM having moved surviving tuples to form a single contiguous block of free space between pd_lower and pd_upper.

pd_lower didn't change. Even though line pointers 3, 6, and 9 are now LP_UNUSED, they still occupy slots in the array. The next INSERT will reuse one of these slots rather than appending a new one at the end. PostgreSQL tracks reusable slots via the PD_HAS_FREE_LINES flag in the page header.

The line pointer lifecycle, precisely

Let's be exact about how line pointers transition between states. This matters because different code paths produce different transitions:

LP_UNUSED (0): the slot is empty. Either it was never used, or VACUUM has fully reclaimed it. The next INSERT that targets this page will grab this slot and assign it to a new tuple, transitioning it to LP_NORMAL.

LP_NORMAL (1): the slot points to a tuple on the page. That tuple might be live (t_xmax = 0 or t_xmax aborted), or it might be dead (t_xmax committed and invisible to all). The line pointer itself doesn't encode liveness; that's determined by the tuple header and the visibility rules we covered in PostgreSQL MVCC, Byte by Byte.

LP_REDIRECT (2): the slot points not to a tuple but to another line pointer. This is created by HOT pruning: when the head of a HOT chain is pruned, its line pointer becomes a redirect so that indexes (which still reference the original line pointer number) can follow the redirect to find the current version. We saw this in HOT Updates in Postgres.

LP_DEAD (3): the slot is known to contain a dead tuple whose storage has been reclaimed, but whose TID may still be referenced by index entries. Index scans that encounter it skip it immediately without a visibility check. On the heap it is set by page pruning, including the prune in VACUUM's own first heap pass. The index side carries the same LP_DEAD hint, and ordinary readers set it: when an index scan follows an entry to a heap tuple that turns out to be dead to everyone, the kill_prior_tuple optimization flags that index entry LP_DEAD so later scans skip it without a heap visit, cleanup work that plain SELECTs do long before VACUUM arrives. For a table with indexes, this is the state a plain deleted tuple sits in between pruning and index cleanup, exactly as we watched above.

The transitions look like this:

LP_UNUSED (0) ----INSERT----> LP_NORMAL (1)

-- table WITH indexes: two heap passes
LP_NORMAL (1) --prune (1st pass)--> LP_DEAD (3)
LP_DEAD (3) --index cleanup + 2nd pass--> LP_UNUSED (0)

-- table WITHOUT indexes: single heap pass
LP_NORMAL (1) --prune--> LP_UNUSED (0)

-- HOT
LP_NORMAL (1) --HOT prune (chain head)--> LP_REDIRECT (2)
LP_NORMAL (1) --HOT prune (heap-only chain member)--> LP_UNUSED (0)
LP_REDIRECT (2) --VACUUM (when target is also dead)--> LP_UNUSED (0)

Whether a deleted tuple passes through LP_DEAD depends on whether the table has indexes. With indexes, pruning can't free the slot (an index entry still references the TID), so it parks the line pointer at LP_DEAD, and only the second heap pass, after index cleanup, turns it into LP_UNUSED. Without indexes there is nothing to reference the TID, so pruning sets it straight to LP_UNUSED in a single pass. Either way the tuple's storage is reclaimed during the prune; LP_DEAD is only ever about the fate of the 4-byte slot.

Free space map

Before VACUUM, PostgreSQL's free space map (FSM) had no idea that our page contained reclaimable space. The dead tuples were invisible to the FSM; they still looked like occupied bytes. After VACUUM, the reclaimed space is registered:

SELECT blkno, avail
FROM pg_freespace('vacuum_demo');

 blkno | avail
-------+-------
     0 |  3328
(1 row)

Page 0 now reports 3,328 bytes of available space. This is slightly less than the raw pd_upper - pd_lower value of 3,344 because the FSM tracks free space in coarse categories of roughly 32 bytes and rounds down. But the critical point is that this page is now a candidate for new INSERTs.

Without this FSM update, PostgreSQL would skip past this page when inserting new rows and extend the table file with a fresh page instead. That's how tables grow even when they contain plenty of free space internally; the FSM wasn't updated because VACUUM never ran.

-- New inserts reuse the vacuumed space instead of extending the file
INSERT INTO vacuum_demo (category, payload)
SELECT 'cat_new', repeat('y', 100)
FROM generate_series(1, 10) AS i;

SELECT pg_relation_size('vacuum_demo') AS size;

 size
------
 8192
(1 row)

The table is still a single 8 KB page. The ten new rows landed in the space VACUUM freed on page 0; the FSM pointed the inserter straight at it, so no new page was appended. Delete-then-insert without a VACUUM in between would tell a different story: the FSM would still report the page as full, the inserter would skip it, and the file would grow. That is the mechanism behind a table that keeps expanding on disk while sitting half empty.

Visibility map

The visibility map tracks two bits per heap page. Those ten inserts we just did left fresh, not-yet-all-visible tuples on page 0, so the bit is currently clear. One more VACUUM settles the page, and now it's set:

VACUUM vacuum_demo;

SELECT blkno, all_visible, all_frozen
FROM pg_visibility('vacuum_demo');

 blkno | all_visible | all_frozen
-------+-------------+------------
     0 | t           | f
(1 row)

Page 0 is marked all_visible. This means every tuple currently on the page is visible to all transactions. The implications are significant:

Index-only scans can return results from the index without fetching this heap page at all. The visibility map confirms that whatever is on this page, it's visible, so the index's copy of the data is guaranteed to be correct.
Future VACUUM passes can potentially skip this page during the heap scan phase, since there's nothing dead to find.

The visibility map is stored as a separate fork file alongside the main heap file. For a table in base/16384/24601, the visibility map is base/16384/24601_vm. It's tiny: two bits per page means a 1 GB table (131,072 pages) needs only about 32 KB for its visibility map.

The all_frozen bit is still false. That flag indicates something stronger: not only are all tuples visible, but all of their xmin values have been frozen: they will survive XID wraparound without any further attention.

If we now delete a row from this page, the all_visible bit gets cleared:

DELETE FROM vacuum_demo WHERE id = 1;

SELECT blkno, all_visible, all_frozen
FROM pg_visibility('vacuum_demo');

 blkno | all_visible | all_frozen
-------+-------------+------------
     0 | f           | f
(1 row)

A single dead tuple on the page is enough to invalidate the all_visible flag. PostgreSQL clears this bit eagerly, at the moment of the DELETE, because it must be conservative. Index-only scans rely on this flag being correct.

Freezing

Let's bring the page to its final resting state. Running VACUUM FREEZE forces PostgreSQL to freeze every tuple on the table, regardless of age:

VACUUM FREEZE vacuum_demo;

Now inspect the tuples:

SELECT lp, t_xmin, t_infomask,
    CASE WHEN (t_infomask & 256) > 0 AND (t_infomask & 512) > 0 THEN 'FROZEN'
         WHEN (t_infomask & 256) > 0 THEN 'XMIN_COMMITTED'
         ELSE 'no hint bits' END AS freeze_status
FROM heap_page_items(get_raw_page('vacuum_demo', 0))
WHERE lp_flags = 1
LIMIT 8;

 lp | t_xmin | t_infomask | freeze_status
----+--------+------------+---------------
  2 |    746 |       2818 | FROZEN
  3 |    748 |       2818 | FROZEN
  4 |    746 |       2818 | FROZEN
  5 |    746 |       2818 | FROZEN
  6 |    748 |       2818 | FROZEN
  7 |    746 |       2818 | FROZEN
  8 |    746 |       2818 | FROZEN
  9 |    748 |       2818 | FROZEN
(8 rows)

In older PostgreSQL versions (before 9.4), freezing literally rewrote t_xmin to 2 (FrozenTransactionId). Modern PostgreSQL preserves the original xmin value and sets the infomask bits instead. This is better for debugging: you can still see which transaction originally created the row.

Line pointer 1 is gone: that was id = 1, which we just deleted, and VACUUM FREEZE pruned it away before freezing the rest. Line pointers 3, 6, and 9 now carry t_xmin = 748: those are the cat_new rows from the free space map section, which reused the very slots the earlier VACUUM had freed. Every surviving tuple is FROZEN. The t_infomask value of 2818 is 0x0B02, which is HEAP_XMIN_COMMITTED (0x0100) plus HEAP_XMIN_INVALID (0x0200) plus HEAP_XMAX_INVALID (0x0800), plus the HEAP_HASVARWIDTH attribute bit (0x0002) that every row here carries because of its text columns. As we covered in PostgreSQL MVCC, Byte by Byte, the combination of both XMIN_COMMITTED and XMIN_INVALID is the freeze marker. It means "this tuple's xmin is permanently in the past, regardless of what the XID counter does."

We can also verify the table-level freeze horizon:

SELECT relfrozenxid FROM pg_class WHERE relname = 'vacuum_demo';

 relfrozenxid
--------------
          750
(1 row)

This tells PostgreSQL: every tuple in vacuum_demo has an xmin that's either frozen or newer than XID 750. Autovacuum uses this value to decide when anti-wraparound freezing is needed. Since we just froze everything, this horizon advanced to the current transaction ID.

And the visibility map now shows both flags set:

SELECT blkno, all_visible, all_frozen
FROM pg_visibility('vacuum_demo');

 blkno | all_visible | all_frozen
-------+-------------+------------
     0 | t           | t
(1 row)

The all_frozen bit is now true. Future VACUUM passes, including anti-wraparound freezing passes, can skip this page entirely. There is nothing to clean and nothing to freeze. For large, mostly-static tables (lookup tables, historical data, archived partitions), this is a substantial performance benefit: VACUUM can skip millions of pages in seconds by checking two bits per page instead of reading every tuple.

VACUUM FULL vs regular VACUUM

Everything we've seen so far is regular (lazy) VACUUM. There is a persistent myth that it can never shrink a table's file on disk. That's not quite right: at the very end of its run, lazy VACUUM tries to truncate trailing pages that have become completely empty, handing that space back to the operating system. What it cannot do is compact free space stranded in the interior of the file: half-empty pages surrounded by full ones. That distinction is the whole story, and both halves are easy to see.

Start with a table that fills 18 pages:

CREATE TABLE vt (id int GENERATED ALWAYS AS IDENTITY PRIMARY KEY, category text, payload text);
INSERT INTO vt (category, payload)
SELECT 'cat_' || (i % 5), repeat('x', 100)
FROM generate_series(1, 1000) AS i;

SELECT pg_relation_size('vt') AS full_size;

 full_size
-----------
    147456
(1 row)

147,456 bytes is 18 pages. Now delete everything past the first fifty rows, which all live on page 0, and vacuum:

DELETE FROM vt WHERE id > 50;

VACUUM vt;
SELECT pg_relation_size('vt') AS after_regular_vacuum;

 after_regular_vacuum
----------------------
                 8192
(1 row)

Down to a single page. Every page after the first became completely empty, so those seventeen pages sat at the tail of the file with nothing live on them, and VACUUM's truncation phase returned them to the OS. Regular VACUUM did shrink the file, because the free space happened to be all at the end.

Now the case it can't help with. Rebuild the table and delete every other row, so no page ever fully empties:

DROP TABLE vt;
CREATE TABLE vt (id int GENERATED ALWAYS AS IDENTITY PRIMARY KEY, category text, payload text);
INSERT INTO vt (category, payload)
SELECT 'cat_' || (i % 5), repeat('x', 100)
FROM generate_series(1, 1000) AS i;

DELETE FROM vt WHERE id % 2 = 0;

VACUUM vt;
SELECT pg_relation_size('vt') AS after_regular_vacuum;

 after_regular_vacuum
----------------------
               147456
(1 row)

Still 18 pages. Half the rows are gone and VACUUM registered all that free space in the FSM, but every page still holds live tuples, so there are no trailing empty pages to truncate. The file cannot shrink; the free space is stranded in the interior. This is exactly the bloat people mean when they say "I deleted half the table and it's still the same size on disk."

That is what VACUUM FULL is for:

VACUUM FULL vt;
SELECT pg_relation_size('vt') AS after_vacuum_full;

 after_vacuum_full
-------------------
             73728
(1 row)

147 KB down to 73 KB, 18 pages to 9. VACUUM FULL rewrote the entire table into a fresh file, packing the 500 surviving rows tightly and deleting the old file. Regular VACUUM had already made that space reusable, but only VACUUM FULL could compact it back out of the file.

VACUUM FULL requires an AccessExclusiveLock: it blocks ALL reads and writes on the table for the entire duration. For a large table, that can mean minutes or hours of complete downtime for that table. In production, consider pg_repack or pg_squeeze as online alternatives that achieve the same compaction without exclusive locks.

The tradeoff is clear: regular VACUUM is lightweight (ShareUpdateExclusiveLock, doesn't block reads or writes) and makes space reusable, truncating the file only when trailing pages are empty. VACUUM FULL compacts interior free space out of the file but takes an AccessExclusiveLock and blocks everything. For most workloads, regular VACUUM is sufficient: new INSERTs reuse the freed space, and the table reaches a steady-state size. VACUUM FULL is a remediation tool for when bloat has gotten out of hand.

Putting it all together

Let's trace the complete lifecycle of a tuple from INSERT to VACUUM, with every state change visible:

INSERT

Line pointer allocated as LP_NORMAL (1), pointing to the new tuple
Tuple stamped with t_xmin = inserting XID, t_xmax = 0
pd_lower advances by 4 bytes (new line pointer); pd_upper decreases by the tuple size

DELETE

Tuple's t_xmax set to the deleting XID
Visibility map all_visible bit cleared for the page
No space reclaimed, no line pointer change, no pd_lower/pd_upper change

VACUUM phase 1 (heap scan + prune)

Each page pruned: dead tuple data removed, page defragmented
pd_upper increases; free space map updated
Dead line pointers set to LP_DEAD (indexed table) or LP_UNUSED (no indexes)
Dead TIDs collected; tuples opportunistically frozen

VACUUM phase 2 (index cleanup)

Index entries pointing to dead TIDs removed from every index
Heap pages untouched

VACUUM phase 3 (heap cleanup, indexed tables only)

LP_DEAD line pointers flipped to LP_UNUSED (0), slots reusable
No tuple bytes reclaimed here; pruning already did that in phase 1
Visibility map all_visible bit set (if all remaining tuples are visible)

VACUUM FREEZE

Tuple infomask bits set to XMIN_COMMITTED + XMIN_INVALID (frozen)
Visibility map all_frozen bit set
pg_class.relfrozenxid advanced

The line pointers move through the state machine we traced above (LP_NORMAL to LP_DEAD to LP_UNUSED on an indexed table, straight to LP_UNUSED without one), but the phase trace is where the why lives.

Every INSERT creates a line pointer. Every DELETE stamps t_xmax but changes nothing else on the page. And every VACUUM cycle (prune, index cleanup, heap cleanup) converts dead space back into reusable space. The page never grows beyond 8 KB, and VACUUM's job is to make sure as much of those 8,192 bytes as possible are available for live data.

TOAST: Where PostgreSQL hides big values

2026-05-24T15:10:00+00:00

In earlier posts in this series we established that every heap tuple lives inside a strict 8KB page. Everything else is built on top of that hard limit: MVCC, HOT updates, and indexes that point at (page, line_pointer). And yet this still works:

CREATE TABLE docs (id int PRIMARY KEY, body jsonb);
INSERT INTO docs VALUES (1, (SELECT jsonb_agg(g) FROM generate_series(1, 100000) g));

That body value is somewhere north of half a megabyte. The heap page is still 8KB. Both statements are true at the same time, and the mechanism that makes them coexist is TOAST: The Oversized-Attribute Storage Technique.

The 2KB threshold

The page is 8KB, but PostgreSQL starts shrinking tuples at around 2KB. The goal is to keep at least four tuples per page, so anything bigger triggers TOAST.

Two constants are at play. TOAST_TUPLE_THRESHOLD decides when the toaster runs; TOAST_TUPLE_TARGET is the size it tries to shrink the tuple down to. Both default to 2032 bytes on a standard 8KB build. The threshold is fixed at compile time; the target is per-table tunable via the toast_tuple_target storage parameter.

When an INSERT or UPDATE produces a tuple wider than the threshold, PostgreSQL doesn't wait to see whether the row would actually fail to fit on a page. It starts compressing and relocating attributes one at a time, and stops the moment the tuple drops below the target. A row only modestly over the line often gets away with one cheap compression pass; a much wider one walks the full procedure.

Only variable-length attributes are candidates. A bigint is always 8 bytes, so there is nothing to compress and nowhere to move it. A text, bytea, jsonb, numeric, or array, on the other hand, is a varlena: a length-prefixed structure PostgreSQL can freely rearrange.

The four storage strategies

Each variable-length column has a storage strategy: a flag in pg_attribute.attstorage that answers two yes/no questions for the toaster. Can this value be compressed? Can it be moved out-of-line? PostgreSQL picks a default based on the column's type, and you can change it with ALTER TABLE ... ALTER COLUMN ... SET STORAGE. There are four valid values:

Strategy	Compress?	Out-of-line?	Default for
`PLAIN`	no	no	fixed-width types
`EXTENDED`	yes	yes	most varlena types
`EXTERNAL`	no	yes	(opt-in)
`MAIN`	yes	last resort only	(opt-in)

EXTENDED is what you get for text, bytea, jsonb, and friends unless you ask for something else. The other three are levers for specific tradeoffs:

EXTERNAL skips compression entirely. Useful when the column is already compressed (images, gzipped blobs), or when you plan to use substring/byte-range reads, which work better on uncompressed out-of-line data.
MAIN tries hard to keep the value in the main heap tuple even after compression. It only moves out-of-line as a last resort.
PLAIN forbids both. Fixed-width types use it because they have no choice; you can set it on a varlena column too, but then any value that doesn't fit will fail the INSERT.

You can inspect and change strategies through \d+ or directly:

ALTER TABLE docs ALTER COLUMN body SET STORAGE EXTERNAL;

The shrinking procedure

Once the toaster is running, PostgreSQL walks the tuple's variable-length attributes in a fixed order, applying the cheapest transformations first:

Compress EXTENDED attributes, largest first. If any single attribute is large enough that it should clearly go out-of-line anyway, push it out immediately in the same pass.
If the tuple still doesn't fit, move remaining EXTENDED or EXTERNAL attributes out-of-line, largest first.
If still too big, compress MAIN attributes.
If still too big, move MAIN attributes out-of-line as a last resort.

Each pass stops as soon as the tuple drops below the target. A row with one chubby JSON column and a handful of small text fields usually only touches the JSON: a single compression pass typically does it.

The whole procedure lives in heap_toast_insert_or_update() in src/backend/access/heap/heaptoast.c, with the per-attribute helpers in src/backend/access/common/toast_helper.c. The four numbered loops in heap_toast_insert_or_update map one-to-one onto the four steps above. It's one of the more readable hot paths in the heap code if you want to see the policy in source form.

Watching it happen

To watch this happen we need a table with one wide column and the pageinspect extension we've been using throughout this series.

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE toast_demo (
    id    integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    label text,
    body  text
);

Insert three rows of very different sizes:

INSERT INTO toast_demo (label, body)
VALUES
('tiny',         'hello'),
('medium',       repeat('compressible-text-', 200)),
('large-random', (SELECT string_agg(md5(g::text), '')
                   FROM generate_series(1, 1000) g));

The middle row is highly compressible: the same 18-byte sequence repeated 200 times. The third is effectively random (MD5 output concatenated), so compression won't help. Both are well over 2KB before TOAST gets involved.

Every table with a toastable column gets a companion TOAST table created for it automatically:

SELECT reltoastrelid::regclass FROM pg_class WHERE relname = 'toast_demo';

       reltoastrelid
---------------------------
 pg_toast.pg_toast_1367271

That pg_toast_<oid> table is where out-of-line chunks live. It has a fixed shape, the same for every TOAST table in the database:

\d pg_toast.pg_toast_1367271

   Column   |  Type
------------+---------
 chunk_id   | oid
 chunk_seq  | integer
 chunk_data | bytea
Owning table: "public.toast_demo"
Indexes:
    "pg_toast_1367271_index" PRIMARY KEY, btree (chunk_id, chunk_seq)

Every out-of-line value is sliced into chunks of up to TOAST_MAX_CHUNK_SIZE (~2000 bytes), each row identified by (chunk_id, chunk_seq). The primary key on those two columns is how PostgreSQL stitches chunks back together in order, on demand.

What the heap tuple looks like

Look at the main heap page now:

SELECT lp, lp_off, lp_len, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('toast_demo', 0));

 lp | lp_off | lp_len | t_xmin | t_xmax | t_ctid
----+--------+--------+--------+--------+--------
  1 |   8152 |     39 | 414236 |      0 | (0,1)
  2 |   8040 |    108 | 414236 |      0 | (0,2)
  3 |   7976 |     59 | 414236 |      0 | (0,3)
(3 rows)

Three rows, three very different lengths on disk. The tiny row at lp 1 is 39 bytes: tuple header, the label, and 'hello' stored inline. The large-random row at lp 3 is 59 bytes despite carrying a 32KB body value, because the data went out-of-line and only an 18-byte TOAST pointer is left behind in the main heap tuple.

The interesting one is lp 2. 108 bytes is way too big for a TOAST pointer, and way too small for a 3.6KB string. That's because the medium value was compressed by pglz and stayed inline. The repeating 18-byte pattern crushed down to roughly 70 bytes; once the tuple was under the target, step 1 of the procedure was done and step 2 (move out-of-line) never ran. This is exactly the compress-first, relocate-only-if-needed policy in action: a highly compressible column can stay on the main heap and never produce a single TOAST chunk.

What sits at lp 3 in place of the value is an 18-byte TOAST pointer: 1-byte varlena header tagged 0x80, 1-byte kind discriminator (VARTAG_ONDISK), and then four fields, the uncompressed size, the on-disk size (with a compression-method tag baked in on PG14+), the chunk_id to look up, and the OID of the TOAST table to look in. The remaining bytes of the 59-byte tuple are the tuple header, the 'large-random' label, and the standard alignment padding.

If you go reading raw page hex, don't look for the usual 4-byte varlena length word in front of a TOAST pointer. There isn't one. External TOAST pointers always use the short 1-byte header form (0x80 marks them as "external, no inline length"), and the actual size is implied by the va_tag byte that follows. There's no alignment padding inside the pointer either; the 18 bytes are 1 + 1 + 16 contiguous.

The toast table itself has the real data:

SELECT chunk_id, chunk_seq, length(chunk_data)
FROM pg_toast.pg_toast_1367271
ORDER BY chunk_id, chunk_seq;

 chunk_id | chunk_seq | length
----------+-----------+--------
  1367279 |         0 |   1996
  1367279 |         1 |   1996
  1367279 |         2 |   1996
  1367279 |         3 |   1996
  1367279 |         4 |   1996
  1367279 |         5 |   1996
  1367279 |         6 |   1996
  1367279 |         7 |   1996
  1367279 |         8 |   1996
  1367279 |         9 |   1996
  1367279 |        10 |   1996
  1367279 |        11 |   1996
  1367279 |        12 |   1996
  1367279 |        13 |   1996
  1367279 |        14 |   1996
  1367279 |        15 |   1996
  1367279 |        16 |     64
(17 rows)

Only one chunk_id, because only the large-random value ended up out-of-line. The medium value never made it here at all: compression alone was enough, and it stayed in the main heap. large-random was effectively incompressible, so pglz sampled the start, gave up early, and the value was stored uncompressed out-of-line. The full 32KB got sliced into 16 chunks of ~2KB plus a 64-byte tail.

The TOAST table is a regular heap table. It has its own pages, its own MVCC, its own VACUUM accounting, and its own indexes. When you update a TOASTed column, the chunks for the old value remain dead until VACUUM removes them, which is why VACUUM (VERBOSE) on a wide table reports separate counts for the main relation and its TOAST relation.

Compression: pglz vs lz4

For years, PostgreSQL had exactly one compression algorithm for TOAST: pglz, a built-in LZ-family compressor. Since PostgreSQL 14, lz4 is also available. It's usually faster on both compress and decompress, with similar or slightly worse ratios on text-like data. The choice is per-column, with a cluster-wide default:

SHOW default_toast_compression;   -- 'pglz' or 'lz4'

ALTER TABLE docs ALTER COLUMN body SET COMPRESSION lz4;

A column's current setting is visible in pg_attribute.attcompression:

SELECT attname, attstorage, attcompression
FROM pg_attribute
WHERE attrelid = 'toast_demo'::regclass AND attnum > 0;

 attname | attstorage | attcompression
---------+------------+----------------
 id      | p          |
 label   | x          |
 body    | x          |

attstorage mirrors the four strategies (p=PLAIN, x=EXTENDED, e=EXTERNAL, m=MAIN). attcompression is a single char: p for pglz, l for lz4, and blank when the column uses the cluster default. Changing the setting only affects values written after the change. Existing rows keep whatever they were compressed with, because the compression method is recorded inside the TOAST pointer itself.

If you flip SET COMPRESSION on a table full of pglz data, nothing changes on disk until those rows are updated or rewritten by something like VACUUM FULL or CLUSTER. Mixed-compression columns are completely normal and supported.

Because the compression method is recorded inside each TOAST pointer, you can ask any individual row how it was compressed without trusting the column-level setting:

SELECT id,
       pg_column_compression(body) AS method,
       pg_column_size(body)        AS on_disk_bytes,
       octet_length(body)          AS logical_bytes
FROM toast_demo;

 id | method | on_disk_bytes | logical_bytes
----+--------+---------------+---------------
  1 |        |             6 |             5
  2 | pglz   |            72 |          3600
  3 |        |         32000 |         32000

pg_column_compression returns the compression method (pglz or lz4) when the on-disk bytes are actually compressed, and NULL otherwise. pg_column_size reads the on-disk size out of the TOAST pointer (or just reports the inline size). Both are cheap, neither has to fetch a chunk or decompress anything. octet_length (or length on a text value) is the expensive one: it has to detoast and decompress the value to count.

The three rows show three different outcomes:

Row 1: value is inline, never compressed (tiny string). method is NULL.
Row 2: value is inline and compressed. method is pglz; on_disk_bytes is far smaller than logical_bytes.
Row 3: value is out-of-line and uncompressed. method is NULL because pglz sampled the data, saw it wasn't compressing, and gave up. on_disk_bytes equals logical_bytes; the raw bytes live in the TOAST relation untouched.

NULL in method means "the on-disk bytes are not compressed" and tells you nothing about whether the value is inline or out-of-line. To distinguish those, compare on_disk_bytes to logical_bytes, or use PG17's pg_column_toast_chunk_id() which returns NULL for inline values and the actual chunk_id for out-of-line ones.

What it costs you

TOAST is transparent in the same way virtual memory is: a seamless abstraction until performance characteristics force you to look at the underlying plumbing.

A SELECT body FROM docs WHERE id = 1 for a 1MB out-of-line value does a lot more than read one heap tuple. PostgreSQL fetches the tuple, follows the TOAST pointer, looks up chunk_id in the TOAST index, reads every chunk in order from the TOAST relation, reassembles them, decompresses the result, and returns it. For a 1MB value with default chunk size, that's around 500 toast-table tuples plus their index lookups, on top of the main heap read.

The implications are practical:

SELECT * is more expensive than it looks. Every wide column gets detoasted even if your application throws the value away. Asking for only the columns you need is not just a style preference; on TOAST-heavy tables it's an I/O decision.
pg_column_size(body) is cheap; length(body) is not. pg_column_size reads the on-disk size right out of the TOAST pointer without fetching a single chunk. length and most other functions have to detoast and decompress the whole value first. substring(body, n, m) is a useful middle case: when the value is stored out-of-line and uncompressed (EXTERNAL), PostgreSQL can fetch only the chunks the substring actually touches.
WHERE body LIKE '%foo%' reads everything. A sequential scan that filters on a TOASTed column has to detoast every row to evaluate the predicate. This is one of the strongest arguments for an expression index, a GIN trigram index, or just storing a separate searchable summary column.
Updates to a TOASTed value rewrite the chunks. PostgreSQL does not patch individual chunks. Update one key inside a 1MB jsonb, and you get a brand-new value, a brand-new chunk_id, and a full new set of chunks. The old chunks become dead and wait for VACUUM. This is the operational cost behind the common advice "don't store frequently-updated big documents in a single column."

Limits

The varlena length field gives every variable-length value a 1GB ceiling per column per row. That's the hard limit. Anything larger has to live outside the database, typically in object storage, with the database keeping a reference. There is no setting that raises this limit.

The toast pointer itself is small (18 bytes) and adds almost nothing to the main tuple. So while a TOASTed row is logically huge, its footprint on the main heap page is tiny, which is the whole point. A row whose wide column was pushed out-of-line is around 60 bytes on the heap, the same density as any narrow table. The expense only materializes when you actually ask for the wide column.

Following a tuple through the toaster

Every wide tuple takes the same path, but exits at a different door. The diagram below traces it end to end: threshold check, compress EXTENDED, move EXTENDED/EXTERNAL out-of-line, compress MAIN, move MAIN out-of-line. Each step is followed by a size check that lets the tuple leave early.

Our demo rows mapped neatly onto this flow. The medium value compressed under the target on the first step and never left the main heap. The large-random value failed the first size check, got pushed out-of-line by step two, and only the small TOAST pointer made it back to the heap page.

Keeping the page invariant safe

Whenever a heap tuple would exceed about 2KB, PostgreSQL shrinks it: first by compressing variable-length columns, then by relocating them, largest first, into the table's companion TOAST relation. The main-heap tuple is left holding a small TOAST pointer in place of the value. The out-of-line bytes live as (chunk_id, chunk_seq, chunk_data) rows, indexed for ordered reassembly. Reads transparently glue the chunks back together and decompress on the fly. The page-level invariants, so one tuple per page slot, indexes pointing at (page, lp), MVCC and HOT operating on heap tuples, they all hold, because as far as the page is concerned, every value is small. TOAST is the trick that makes "every tuple fits in one page" still true in a world of megabyte JSON documents.

HOT Updates in Postgres

2026-04-28T22:23:00+02:00

In the previous article we watched every UPDATE leave dead tuple behind. The same copy-on-write behaviour shows up from the operational angle in DELETEs are difficult. That's the tradeoff of MVCC and on the heap alone it's tolerable. The problem is the indexes.

Every UPDATE in PostgreSQL potentially writes to every index on the table, even when the indexed columns didn't change. Five indexes, one updated column? Five extra index writes, five new entries to vacuum, five times the WAL traffic. At thousands of updates per second this becomes the dominant cost of running a write-heavy table.

Heap-Only Tuple (HOT) updates are PostgreSQL's escape hatch from this problem. They are, in my opinion, the single cleverest optimization in the storage engine. Let's trace exactly how they work.

Cost of a normal UPDATE

Without HOT, index maintenance scales poorly. Here's a table with multiple indexes:

pageinspect ships with the contrib modules and is available on most installations. It exposes raw page contents, useful for understanding storage, but never expose it to application users.

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE hot_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    name text NOT NULL,
    status text NOT NULL DEFAULT 'active',
    score numeric(10,2)
);

CREATE INDEX idx_hot_name ON hot_demo (name);
CREATE INDEX idx_hot_status ON hot_demo (status);

INSERT INTO hot_demo (name, status, score) VALUES
    ('alice', 'active', 95.00),
    ('bob',   'active', 82.50),
    ('carol', 'active', 77.25);

Updating the indexed name column requires substantial background work.

UPDATE hot_demo SET name = 'BOB' WHERE id = 2;

PostgreSQL has to:

Set t_xmax on the old tuple to the current transaction ID, marking it dead
Create a new tuple
Insert a primary key index entry pointing to the new tuple's ctid
Insert a new entry into idx_hot_name pointing to the new tuple's ctid
Insert a new entry into idx_hot_status pointing to the new tuple's ctid

The id and status values remain identical, yet their respective indexes require new entries because the physical tuple location changed. The old index pointers become dead weight, pending removal by VACUUM.

Bypassing index updates

Index entries map key values to a physical tuple location (the ctid). If the new tuple is reachable through the old ctid, the index references remain valid and require no updates.

A ctid is a tuple's physical address: (page_number, line_pointer). Indexes store ctids, not row contents.

PostgreSQL uses a HOT update when two specific conditions are met. The new tuple must fit on the same page as the old tuple. And none of the updated columns are indexed.

If either condition fails, it's cold update with the full index dance. If both hold, the update goes HOT.

Demonstrating a HOT update

Let's see it. We update score, a column that isn't in any index:

UPDATE hot_demo SET score = 99.00 WHERE name = 'alice';

The pageinspect extension shows the HOT flags which are stored in t_infomask2.

SELECT
    lp, lp_off, lp_flags, lp_len,
    t_xmin, t_xmax, t_ctid,
    CASE WHEN (t_infomask2 & x'4000'::int) > 0 THEN 'HOT_UPDATED' END AS hot_old,
    CASE WHEN (t_infomask2 & x'8000'::int) > 0 THEN 'HEAP_ONLY' END AS hot_new
FROM heap_page_items(get_raw_page('hot_demo', 0));

The page also still carries the leftovers from the earlier name = 'BOB' cold update (dead bob at lp 2, new BOB at lp 4). Focus on lp 1 and lp 5, the alice update we just ran.

 lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_ctid |   hot_old   |  hot_new
----+--------+----------+--------+--------+--------+--------+-------------+-----------
  1 |   8144 |        1 |     46 | 411759 | 411761 | (0,5)  | HOT_UPDATED |
  2 |   8096 |        1 |     46 | 411759 | 411760 | (0,4)  |             |
  3 |   8048 |        1 |     48 | 411759 |      0 | (0,3)  |             |
  4 |   8000 |        1 |     46 | 411760 |      0 | (0,4)  |             |
  5 |   7952 |        1 |     46 | 411761 |      0 | (0,5)  |             | HEAP_ONLY
(5 rows)

Line pointer 1 marks the old tuple as dead with t_xmax = 411761. Its t_ctid points forward to (0,5). The HOT_UPDATED flag confirms the successor is a heap-only tuple.

Line pointer 5 holds the new tuple. The HEAP_ONLY flag means no index entries point directly to this tuple. It is only reachable by following the chain from line pointer 1.

The indexes remain untouched. They still point to (0,1). During an index scan, PostgreSQL reads the HOT_UPDATED flag at line pointer 1, follows t_ctid to (0,5), and returns the current version.

The ctid chain

Successive HOT updates form confined chains in a single page.

UPDATE hot_demo SET score = 98.00 WHERE name = 'alice';
UPDATE hot_demo SET score = 97.00 WHERE name = 'alice';

 lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_ctid |   hot_old   |  hot_new
----+--------+----------+--------+--------+--------+--------+-------------+-----------
  1 |   8144 |        1 |     46 | 411759 | 411761 | (0,5)  | HOT_UPDATED |
  2 |   8096 |        1 |     46 | 411759 | 411760 | (0,4)  |             |
  3 |   8048 |        1 |     48 | 411759 |      0 | (0,3)  |             |
  4 |   8000 |        1 |     46 | 411760 |      0 | (0,4)  |             |
  5 |   7952 |        1 |     46 | 411761 | 411762 | (0,6)  | HOT_UPDATED | HEAP_ONLY
  6 |   7904 |        1 |     46 | 411762 | 411763 | (0,7)  | HOT_UPDATED | HEAP_ONLY
  7 |   7856 |        1 |     46 | 411763 |      0 | (0,7)  |             | HEAP_ONLY
(7 rows)

The chain routes through lp 1 -> lp 5 -> lp 6 -> lp 7. Intermediate versions carry both HOT_UPDATED and HEAP_ONLY flags. They are dead links in the middle of the chain.

And here's the part that makes this work: the indexes still have exactly one entry for alice, pointing to (0,1). No matter how many times we HOT-update her score, the indexes never grow. PostgreSQL walks the chain from (0,1) through the dead intermediates until it lands on the live tuple at (0,7).

Long HOT chains have cost. Every index scan must walk the chain from start to finish. If you HOT-update the same row 50 times between vacuums, that's 50 hops. The chain is confined to a single page so it's all in-memory work, but it adds up on high-throughput workloads. Page pruning, which we'll see next, is how PostgreSQL shortens these chains without waiting for VACUUM.

Page pruning

When PostgreSQL accesses a page with dead HOT chain members, it can clean up opportunistically. No VACUUM, no background worker, no special scheduling. Just a regular query that happens to touch the page and decides there's work worth doing.

But "can" is doing real work in that sentence. Pruning is gated by two checks, not one. The first is the visibility check: pd_prune_xid < RecentGlobalXmin, meaning the dead tuples are invisible to every running transaction. The second is a cost check: the page has to be full enough that compacting it is worth the cycles. The threshold is roughly 10% free space. Above that, PostgreSQL leaves the page alone even when there's pruneable garbage on it.

Our demo page has seven tiny tuples and most of the page is empty. A plain SELECT against it does nothing. The flags don't move, t_xmax stays set on the dead intermediates, the chain stays long. On a busy table where pages actually fill up, pruning fires constantly during regular SELECTs and UPDATEs. On a quiet demo table, you can stare at it all day.

When pruning does fire, a plain SELECT can dirty a page and trigger a disk write. That's true here (pruning) and also for hint bits, covered in the MVCC article. "Read-only query" at the SQL level doesn't always mean "read-only" at the storage level.

To force the issue on our demo table, run VACUUM. VACUUM always prunes, regardless of free space:

VACUUM hot_demo;

Re-inspecting the page reveals the structural changes.

 lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_ctid | hot_old |  hot_new
----+--------+----------+--------+--------+--------+--------+---------+-----------
  1 |      7 |        2 |      0 |        |        |        |         |
  2 |      0 |        0 |      0 |        |        |        |         |
  3 |   8144 |        1 |     48 | 411759 |      0 | (0,3)  |         |
  4 |   8096 |        1 |     46 | 411760 |      0 | (0,4)  |         |
  5 |      0 |        0 |      0 |        |        |        |         |
  6 |      0 |        0 |      0 |        |        |        |         |
  7 |   8048 |        1 |     46 | 411763 |      0 | (0,7)  |         | HEAP_ONLY
(7 rows)

Line pointer 1 transitions to LP_REDIRECT (lp_flags = 2). It abandons its tuple data and points directly to line pointer 7.
Line pointers 2, 5, and 6 become LP_UNUSED (lp_flags = 0). Their tuple space is reclaimed for future inserts. (Line pointer 2 was the old bob tuple from the cold update earlier, finally cleaned up by VACUUM as well.)
Line pointer 7 is compacted. The page is defragmented, packing surviving tuples tightly at the end of the page.

LP_REDIRECT

PostgreSQL's four line pointer states (LP_UNUSED, LP_NORMAL, LP_REDIRECT, LP_DEAD) were introduced in Inside the 8KB Page. HOT pruning is where LP_REDIRECT earns its keep.

PostgreSQL retains the redirect at line pointer 1 because the indexes still reference (0,1). Freeing it would leave dangling pointers. The redirect only occupies the line pointer slot itself (four bytes), with no tuple data attached, and persists until a future VACUUM rewrites the index entries to point at (0,7) directly. Only then can line pointer 1 finally become LP_UNUSED.

When HOT fails

It helps to see the opposite case. Let's update name, which is indexed:

UPDATE hot_demo SET name = 'ALICE' WHERE name = 'alice';

SELECT
    lp, lp_flags, t_xmin, t_xmax, t_ctid,
    CASE WHEN (t_infomask2 & x'4000'::int) > 0 THEN 'HOT_UPDATED' END AS hot_old,
    CASE WHEN (t_infomask2 & x'8000'::int) > 0 THEN 'HEAP_ONLY' END AS hot_new
FROM heap_page_items(get_raw_page('hot_demo', 0));

 lp | lp_flags | t_xmin | t_xmax | t_ctid | hot_old |  hot_new
----+----------+--------+--------+--------+---------+-----------
  1 |        2 |        |        |        |         |
  2 |        1 | 411764 |      0 | (0,2)  |         |
  3 |        1 | 411759 |      0 | (0,3)  |         |
  4 |        1 | 411760 |      0 | (0,4)  |         |
  5 |        0 |        |        |        |         |
  6 |        0 |        |        |        |         |
  7 |        1 | 411763 | 411764 | (0,2)  |         | HEAP_ONLY
(7 rows)

No HOT_UPDATED on lp 7, no HEAP_ONLY on lp 2. This is cold update. The new tuple landed at line pointer 2 (the slot freed by VACUUM a moment ago) and lp 7 was stamped dead with t_ctid pointing forward. PostgreSQL also had to insert a new entry into idx_hot_name (the indexed value changed), and because it's cold, it added new entries into idx_hot_status and the primary key index even though neither of those values changed.

The second common failure mode is space. Even if you only touch unindexed columns, completely full page leaves PostgreSQL no choice but to put the new tuple on a different page. And a cross-page update always means index maintenance, because the ctid changes.

fillfactor: making room for HOT

HOT updates need free space on the actual page. By default, PostgreSQL writes pages with a fillfactor of 100, packing them full during inserts. The first UPDATE on a full page has to allocate the new tuple elsewhere, and HOT is off the table.

Lowering the fillfactor reserves space specifically for updates.

CREATE TABLE hot_ff_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    val text,
    counter integer DEFAULT 0
) WITH (fillfactor = 80);

A fillfactor of 80 leaves 20% of the page empty during inserts, giving HOT chains room to grow.

The right fillfactor depends on your workload. Read-heavy or insert-only tables: leave it at 100. Wasting 20% of storage for updates that never happen is pointless. Frequent updates on non-indexed columns: 80-90 is a common choice. Wide rows with heavy churn: 50-70. The trade-off is storage. A lower fillfactor means more pages, which means more I/O for sequential scans.

You can measure HOT success rates through pg_stat_user_tables.

SELECT
    relname,
    n_tup_upd,
    n_tup_hot_upd,
    round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE n_tup_upd > 0
ORDER BY n_tup_upd DESC;

A dropping hot_pct on an update-heavy table is signal worth investigating. The usual suspects:

Pages are full. Lower the fillfactor or make sure VACUUM runs frequently enough to reclaim space before updates need it.
Indexed columns are being updated. ORMs love to send UPDATE users SET name = 'alice', email = 'alice@example.com', updated_at = now() when only updated_at changed. If name or email are indexed, that's a cold update every time. Look for "dirty tracking" features in your ORM.
Too many indexes. Every index adds a column to the "no indexed columns can change" condition. Review whether all of them are actually used. An unused index doesn't just waste space, it prevents HOT updates on every column it covers.

pd_prune_xid revisited

Back to the page header from Inside the 8KB Page. After a HOT update, pd_prune_xid gets stamped with the t_xmax of the dead tuple:

UPDATE hot_demo SET name = 'Alice' WHERE name = 'ALICE';

SELECT prune_xid FROM page_header(get_raw_page('hot_demo', 0));

 prune_xid
-----------
    411764
(1 row)

The name update here is cold because name is indexed. Cold updates seed pd_prune_xid too, since dead tuples need pruning either way. Once set, the field tracks the oldest unpruned t_xmax on the page, so newer dead tuples don't bump it forward.

That prune_xid of 411764 is the t_xmax of the oldest dead tuple still waiting to be pruned (the ALICE tuple at lp 7, killed by this latest update). The next time any backend touches this page, it compares pd_prune_xid against RecentGlobalXmin. If the dead tuples are guaranteed invisible to everyone and the page is full enough to be worth compacting, pruning fires.

RecentGlobalXmin is the oldest snapshot xmin across every active transaction, replication slot, and prepared transaction on the cluster. It's the horizon below which "definitely invisible to everyone" is safe to assume.

This is why pruning is described as "opportunistic" or "lazy". PostgreSQL doesn't schedule it, doesn't run it in the background, and doesn't bother with pages that have plenty of room. It checks two cheap conditions on every page access and cleans up on the spot when both hold. On a busy table, pages get pruned constantly during regular SELECTs and UPDATEs. On a half-empty page, dead tuples linger until either the page fills up or VACUUM runs.

Long-running transactions are the enemy of page pruning. If a transaction started an hour ago and is still open, RecentGlobalXmin stays pinned to its snapshot. No tuples created after that point can be pruned, no matter how dead they are. Same root cause that blocks VACUUM, and one of the reasons long-running transactions in PostgreSQL are so dangerous.

HOT in one paragraph

When PostgreSQL gets an UPDATE, it checks two things: is there room on the same page, and are all the changed columns unindexed? If yes to both, the update goes HOT:

The new tuple lands on the same page with the HEAP_ONLY flag.
The old version is stamped HOT_UPDATED, with t_ctid pointing forward to the new one.
pd_prune_xid is set, flagging the page as having dead versions to clean up.
No index entry is created or touched.

Later, when any backend reads the page, it checks two things: are the dead versions invisible to everyone, and is the page full enough to be worth compacting? When both hold, pruning fires: intermediate chain members become LP_UNUSED, the original line pointer turns into an LP_REDIRECT that still satisfies every index reference, and the page is defragmented. The reclaimed space is ready for the next HOT update.

With the right fillfactor, this loop runs forever, and VACUUM stays largely out of the picture.

What comes next

Page pruning is lightweight cleanup that happens during normal reads and writes. It handles dead tuples in HOT chains and reclaims their space on the page. But it has limits. It cannot touch index entries. It can flag dead tuples from cold updates but can't fully reclaim them, since the index entries still need to come out. It cannot set visibility map bits or update the free space map. And it cannot reach across pages.

For all of that, you need VACUUM. And as we covered in VACUUM Is a Lie (About Your Indexes), even VACUUM has its own blind spots once indexes enter the picture.

PostgreSQL MVCC, Byte by Byte

2026-04-17T14:45:00+00:00

You run SELECT * FROM orders in one psql session and see 50 million rows. A colleague in another session runs the same query at the same moment and sees 49,999,999. Neither of you is wrong, and neither is seeing stale data. You are both reading the same 8KB heap pages, the same bytes on disk.

This is the promise of PostgreSQL's MVCC (Multi-Version Concurrency Control), and it's the reason readers never block writers and writers never block readers. It is also one of the most misunderstood pieces of the storage engine. People know "there are multiple versions of a row" and leave it at that.

The answer lives in eight bytes on every single tuple.

xmin and xmax: the only two XIDs that matter

If you've read Inside the 8KB Page, you know that every tuple starts with 23-byte header. The first eight bytes of that header are two 32-bit transaction IDs: t_xmin (the transaction that inserted this version) and t_xmax (the transaction that deleted or updated it, or 0 if it's still live).

That's the core of MVCC at the storage level. PostgreSQL does not keep a separate "current version" table. It does not mark rows as latest. Every tuple carries its own two-field timestamp, and when your query reads a page, PostgreSQL has to decide, tuple by tuple, whether your transaction is allowed to see it.

A minimal demo:

CREATE TABLE mvcc_demo (id int, val text);
INSERT INTO mvcc_demo VALUES (1, 'alpha'), (2, 'beta');

Peek at the raw page with pageinspect:

SELECT lp, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('mvcc_demo', 0));

 lp | t_xmin | t_xmax | t_ctid
----+--------+--------+--------
  1 |    100 |      0 | (0,1)
  2 |    100 |      0 | (0,2)
(2 rows)

Two tuples. Both stamped with t_xmin = 100 (the transaction that ran the INSERT) and t_xmax = 0 (nobody has deleted them). At this moment, every session on the database will see these rows, because everyone's snapshot agrees that transaction 100 has committed.

Now open two concurrent sessions. Session A runs an UPDATE without committing:

-- session A
BEGIN;
UPDATE mvcc_demo SET val = 'alpha-new' WHERE id = 1;
-- do not commit yet

Peek at the page again:

SELECT lp, t_xmin, t_xmax, t_ctid
FROM heap_page_items(get_raw_page('mvcc_demo', 0));

 lp | t_xmin | t_xmax | t_ctid
----+--------+--------+--------
  1 |    100 |    101 | (0,3)
  2 |    100 |      0 | (0,2)
  3 |    101 |      0 | (0,3)
(3 rows)

One UPDATE, three tuples. The old version of id=1 is still there at line pointer 1 with t_xmax = 101 stamped on it, and a new version lives at line pointer 3 with t_xmin = 101.

Session A hasn't committed. Transaction 101 is still in flight. Session B, running SELECT * FROM mvcc_demo right now, still sees the original alpha, not alpha-new. The three tuples live on the page, but Session B's snapshot says XID 101 is in flight and ignores anything it did. The visibility decision happens on the fly, every time tuple is touched.

This is the part that makes MVCC counterintuitive: the bytes on disk don't change depending on who's asking. What changes is the visibility verdict the planner applies when reading them.

Snapshots

pg_current_snapshot() is the cleanest way to see what your session is actually holding.

SELECT pg_current_snapshot();

 pg_current_snapshot
---------------------
 101:103:101
(1 row)

That's xmin:xmax:xip_list, and it's the whole snapshot:

xmin: the lowest XID that might still be in flight. Everything below has already resolved (committed or aborted). You can trust its t_xmin/t_xmax stamps without further checks.
xmax: the first not-yet-assigned XID. Anything at or above this value doesn't exist yet. Tuples stamped with it must be ignored.
xip_list: the XIDs between xmin and xmax that are still running. These are the "in-flight" transactions whose writes must not be visible to you.

PostgreSQL applies this test tuple by tuple. If your snapshot considers t_xmin aborted or still in flight, the tuple doesn't exist for you and PostgreSQL skips it. If t_xmin is committed, t_xmax decides: zero means the tuple is live, a committed t_xmax means someone has deleted it and you can't see it, and an in-flight or aborted t_xmax means the deletion hasn't reached your snapshot yet.

Same page. Same bytes. Different sessions have different snapshots, so they land on different outcomes for the same tuple.

READ COMMITTED vs REPEATABLE READ

The difference between PostgreSQL's two most-used isolation levels boils down to one question: when is the snapshot captured?

READ COMMITTED (the default) captures a fresh snapshot at the start of every statement. If another session commits between your first and second SELECT, your second SELECT sees the change. The world moves forward under your transaction, statement by statement.

REPEATABLE READ captures a single snapshot when the transaction begins and reuses it for every subsequent statement. The world is frozen from your transaction's perspective. Other sessions can commit thousand changes; your query keeps returning what was visible at BEGIN.

The bytes on the page are identical in both cases. The only difference is which snapshot your transaction is walking around with.

-- session A, READ COMMITTED (default)
BEGIN;
SELECT val FROM mvcc_demo WHERE id = 1;  -- 'alpha'

-- session B, in another terminal:
UPDATE mvcc_demo SET val = 'alpha-new' WHERE id = 1;
-- (auto-commits)

-- back in session A:
SELECT val FROM mvcc_demo WHERE id = 1;  -- 'alpha-new' new statement, new snapshot
COMMIT;

Now repeat with REPEATABLE READ:

-- session A, REPEATABLE READ
BEGIN ISOLATION LEVEL REPEATABLE READ;
SELECT val FROM mvcc_demo WHERE id = 1;  -- 'alpha-new'

-- session B:
UPDATE mvcc_demo SET val = 'alpha-newer' WHERE id = 1;
-- (auto-commits)

-- Back in session A:
SELECT val FROM mvcc_demo WHERE id = 1;  -- still 'alpha-new'  same snapshot as BEGIN
COMMIT;

The visualizer exposes this directly: there's an isolation-level selector on each session. Under REPEATABLE READ, the snapshot is captured at BEGIN and persists. Under READ COMMITTED, it refreshes every time you run SELECT. Watch how the visibility badges on each tuple flip accordingly.

Every UPDATE leaves a dead tuple behind

Every UPDATE in PostgreSQL creates a new tuple version. The old version doesn't vanish. It gets stamped with t_xmax and sits on the page, taking up space, until VACUUM comes through and reclaims it.

On a busy table with lots of updates, dead tuples can accumulate faster than VACUUM clears them. This is "bloat," and it's one of the most common reasons teams think Postgres needs re-tuning. The MVCC contract ("never block, always give a consistent view") is paid for in disk space.

You can see the dead tuples accumulate with pgstattuple:

CREATE EXTENSION IF NOT EXISTS pgstattuple;

-- After lots of updates
SELECT table_len, tuple_count, dead_tuple_count, dead_tuple_percent
FROM pgstattuple('mvcc_demo');

 table_len | tuple_count | dead_tuple_count | dead_tuple_percent
-----------+-------------+------------------+--------------------
      8192 |           2 |                3 |              42.15
(1 row)

Three dead tuples, two live ones, 42% of the page wasted. That 42% stays wasted until VACUUM runs, or until page-level pruning kicks in during the next query that touches this page and notices the dead space.

The xmin horizon

VACUUM can only reclaim a dead tuple when no running transaction might still need to see it. If Session B started a REPEATABLE READ transaction five minutes ago and has been sitting idle, its snapshot still considers the pre-update version of id=1 to be the live one. VACUUM cannot touch it without breaking that session.

So VACUUM finds the oldest active transaction on the system and refuses to clean anything newer. A long-running REPEATABLE READ transaction (say, an analytics query that takes an hour) effectively pins every tuple version produced during that hour. The table keeps bloating. Autovacuum runs, finds nothing it's allowed to clean, and goes home.

The long-running-transaction problem isn't an MVCC bug. It's MVCC working as designed. The price of "readers never block" is that readers can block cleanup. If you've ever checked pg_stat_activity on a misbehaving production database and spotted a 14-hour-old idle in transaction, you know this shape.

The visualizer shows this cleanly: start a REPEATABLE READ transaction in Session B, have Session A run a bunch of UPDATEs and COMMITs, then hit VACUUM. The reclaim count won't include tuple versions Session B can still see.

Hint bits: why SELECT can dirty a page

The first SELECT that touches a page after new writes can cause the page to be written back to disk. Not because the SELECT modified any data, but because it set hint bits.

When PostgreSQL encounters a tuple with t_xmin = 101 and needs to know whether 101 committed, it doesn't magically know. It has to look up 101 in pg_xact (formerly pg_clog), the commit log. Once it finds the answer, it caches that answer in the tuple's t_infomask bits (HEAP_XMIN_COMMITTED or HEAP_XMIN_INVALID). Future readers skip the pg_xact lookup entirely.

Setting those bits is a write. The page becomes dirty. Eventually it gets flushed. Your innocent SELECT ends up triggering I/O.

This is why EXPLAIN (ANALYZE, BUFFERS) on a cold table sometimes shows dirtied buffers even when the plan contains nothing but reads. It's also why the "first query after bulk load" pattern has that mysterious slow run: you're paying the one-time cost of setting hint bits acros thousands of freshly written pages. See Understanding EXPLAIN Buffers for more on how these counters show up.

The MVCC contract in one paragraph

Every tuple carries t_xmin and t_xmax. Every transaction carries a snapshot of (xmin, xmax, xip_list). Visibility is a two-stage lookup that compares the two. UPDATE and DELETE don't modify bytes in place. They stamp t_xmax on the old version and append a new one. VACUUM cleans up dead versions, but only the ones no live transaction could still need. Long-running transactions block VACUUM. Every SELECT can dirty a page the first time it sees new data, because it caches commit status in hint bits.

Eight bytes of XIDs per tuple, plus a three-number snapshot per transaction, plus one visibility function. That's the whole mechanism, but the consequences spread through every piece of PostgreSQL operations, from bloat monitoring to replication to autovacuum tuning.

For the full byte-level tour (hint-bit encodings, the visibility map, freezing, XID wraparound), the storage series covers these in detail. The next stop is HOT updates and page pruning -- how PostgreSQL stops every UPDATE from also touching every index. If you've never watched MVCC happen, the visualizer is the fastest way to build the intuition. Run the two sessions against each other, flip the isolation level, then come back to this post.

Inside PostgreSQL's 8KB Page

2026-02-19T21:52:00+00:00

If you read previous post about buffers, you already know PostgreSQL might not necessarily care about your rows. You might be inserting a user profile, or retrieving payment details, but all that Postgres works with are blocks of data. 8KB blocks, to be precise. You want to retrieve one tiny row? PostgreSQL hauls an entire 8,192-byte page off the disk just to give it to you. You update a single boolean flag? Same thing. The 8KB page is THE atomic unit of I/O.

But knowing those pages exist isn't enough. To understand why the database behaves the way it does, you need to understand how it works. Every time you execute INSERT, PostgreSQL needs to figure out how to fit it into one of those 8,192-byte pages.

The buffer pool caches them, Write-Ahead Log (WAL) protects them, and VACUUM cleans them. The deep dive into the PostgreSQL storage internals starts by understanding what happens inside those 8KB pages. Pages that are used by PostgreSQL to organize all data - tables, indexes, sequences, TOAST relations.

The 8KB

In case of Oracle, the default block size is set at database creation (DB_BLOCK_SIZE), though tablespaces with non-standard block sizes can be created separately.

Before looking inside we actually need to discuss why 8KB in first place? The answer might be surprising - it's a setting that survived for over 40 years and nobody found a good reason to change it. Plus PostgreSQL isn't the only one who thinks that way. Oracle and SQL Server use the same exact number.

The 8KB page size can be traced down to original Berkley POSTGRES project created in mid-1980s. In those times Unix systems typically used 4KB or 8KB virtual memory pages, and disk sectors were 512 bytes. Choosing 8KB meant a single database page mapped cleanly to OS memory pages and aligned well with filesystem I/O.

And the math still works today. Modern Linux kernels manage memory in 4KB virtual memory pages. SSDs now use 4KB physical sectors instead of 512 bytes. The default filesystem block size on ext4 and XFS is 4KB. PostgreSQL's 8KB page still maps to two OS pages, two disk sectors, two filesystem blocks. The hardware changed underneath, but the alignment is still there.

But the choice isn't just about hardware alignment. It's a tradeoff between two opposing forces. Make the page size too small and you going to increase the overhead (page metadata being good example). Make it too large and you waste space and increase I/O requirements when you need a single narrow row.

The different situation is outside OLTP world. DuckDB, designed for analytical columnar workloads, uses 256KB blocks. When you're scanning millions of rows sequentially, the overhead-per-page penalty barely matters - you want big chunks to maximize throughput.

Is PostgreSQL 8KB page size fixed? Technically no. PostgreSQL supports 1, 2, 4, 8, 16, or 32KB pages via --with-blocksize at compile time. Some analytical workloads with wide rows benefit from 16KB or 32KB pages. But you'll need to rebuild everything from scratch, and unless you have a very specific reason and understand the downstream implications you almost certainly don't want to. The default works.

You can confirm the page size on any running instance.

SHOW block_size;

 block_size
------------
 8192
(1 row)

pageinspect

PostgreSQL comes with an extension called pageinspect that lets you read raw page contents from SQL. It is part of the contrib modules and available on most installations. Let's enable it and create a small test table:

CREATE EXTENSION IF NOT EXISTS pageinspect;

CREATE TABLE page_demo (
    id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    title text NOT NULL,
    created_at TIMESTAMPTZ DEFAULT CURRENT_TIMESTAMP,
    value numeric(10,2)
);

INSERT INTO page_demo (title, created_at, value)
VALUES
    ('Ergonomic standing desk, oak finish', '2026-01-15 09:23:45+00', 549.99),
    ('Wireless mechanical keyboard', '2026-01-15 10:05:12+00', 189.00),
    ('Desk lamp', '2026-01-16 14:31:22+00', 45.00);

We have three rows. PostgreSQL has written them into a single heap page. Let's look inside.

The page_header function takes a raw page and returns the parsed header fields:

SELECT * FROM page_header(get_raw_page('page_demo', 0));

-[ RECORD 1 ]---------
lsn       | F/1F9ED410
checksum  | 0
flags     | 0
lower     | 36
upper     | 7976
special   | 8192
pagesize  | 8192
version   | 4
prune_xid | 0

Every one of those fields lives in the first 24 bytes of the page. Together they form the page header, and they tell PostgreSQL everything it needs to know about the page before touching any actual data.

The first two fields are about safety, making sure the page survives crashes and silent corruption.

pd_lsn (8 bytes) the Log Sequence Number of the last WAL record that modified this page. During crash recovery, PostgreSQL compares this LSN against the WAL stream. If the WAL record's LSN is less than or equal to the page's LSN, the page is already up to date and the record is skipped. If it is greater, the record must be replayed. This single field is what makes crash recovery work.

pd_checksum (2 bytes) a checksum of the page contents. This is only active if the cluster was initialized with initdb --data-checksums (or checksums were enabled later with pg_checksums). When enabled, PostgreSQL verifies the checksum every time it reads a page from disk. A mismatch means silent data corruption, the kind that would otherwise go undetected until your data is wrong in ways nobody can explain.

Are checksums enabled on your cluster? Before PostgreSQL 17, they were off by default. Check with SHOW data_checksums;. If you see off on a production database, that's worth fixing. You can enable them retroactively with pg_checksums, though it requires a full shutdown.

Magic of PD_ALL_VISIBLE

Indexes know what your data is, but not who is allowed to see it (in case a row was recently deleted or updated). Postgres has to do extra work to double-check the main table (the "heap") to ensure a row is visible.

When a page is marked PD_ALL_VISIBLE, it guarantees every row on that page is old enough to be seen by everyone. This lets Postgres skip the expensive heap check entirely and serve data straight from the index. A speed boost you may know as an Index-Only Scan.

The next fields are the page's spatial ma. They tell PostgreSQL where things are and where there's room for more.

pd_flags (2 bytes) bit flags that describe the page state. The important ones are: PD_HAS_FREE_LINES (are there any unused line pointers?), PD_PAGE_FULL (not enough free space for new tuple) and PD_ALL_VISIBLE (all tuples on page are visible to everyone).

pd_lower and pd_upper (2 bytes each) define the free space gap. pd_lower marks where the line pointer array ends -- in our output it is 36, that is 24 bytes of header plus 3 line pointers at 4 bytes each: 24 + (3 x 4) = 36. pd_upper marks where tuple data begins -- our value is 7976, meaning the three tuples together occupy bytes 7976 through 8191. Everything between these two offsets is free space. As you insert rows, these two values creep toward each other until there's no room left.

pd_special (2 bytes) the byte offset to the "special space" at the end of the page. For heap (table) pages, this equals the page size (8192), meaning there is no special space. For index pages, this area contains index-specific metadata.

pd_pagesize_version (2 bytes) encodes both the page size and the layout version number. The version is currently 4 for all modern PostgreSQL releases.

Finally, one field dedicated to cleanup.

The Mini-Cleanup (Page-Level Pruning)

Instead of waiting for a heavy VACUUM, Postgres can do garbage collection on the fly. When a normal query reads a page, it checks `pd_prune_xid`. If that transaction ID is older than all currently running transactions, the query instantly reclaims the dead space itself.

This is one of the cases where SELECT might modify data pages.

pd_prune_xid (4 bytes) the oldest transaction ID whose dead tuples have not yet been pruned from this page. When a new transaction accesses the page and finds that prune_xid is older than the global horizon, it triggers page-level pruning. A lightweight cleanup that reclaims space without a full VACUUM.

What is a line pointer?

If you're paying attention, we just mentioned line pointers in pd_lower description. If the page header discussed above is the general metadata, the line pointers (internally represented as ItemIdData) are the page table of contents.

Every time you insert a row, PostgreSQL doesn't just drop the raw data into the page. It actually splits the job into two steps.

It puts the bulky, unpredictable row data (the tuple) at the very bottom of the page (more on this in the next section).
It adds a tiny, fixed 4-byte line pointer right after the page header.

This pointer acts as a direct map. It holds exact byte offset and length of the tuple it points to. It allows PostgreSQL to only look at the offset and read the required number of bytes to get the tuple.

The line pointer array starts immediately after the 24-byte header and grows downward. Let's look at ours.

SELECT lp, lp_off, lp_flags, lp_len
FROM heap_page_items(get_raw_page('page_demo', 0));

 lp | lp_off | lp_flags | lp_len
----+--------+----------+--------
  1 |   8112 |        1 |     79
  2 |   8032 |        1 |     77
  3 |   7976 |        1 |     53

Each line pointer is 4 bytes and contains three pieces of information:

lp the ordinal position in the array (1-based). This, combined with the page number, forms the ctid - the physical address of a tuple. Row 1 on page 0 has ctid (0,1).
lp_off the byte offset within the page where the actual tuple data begins. Notice the offsets decrease: 8112, 8032, 7976. Tuples are packed from the bottom up.

Line pointers are the reason PostgreSQL can move tuples around within a page (during defragmentation) or perform HOT (Heap-Only Tuple) updates without invalidating index entries. An index stores a ctid like (0, 2), which means "page 0, line pointer 2". Because the index points to the pointer rather than the physical byte offset, Postgres can shuffle data or redirect pointers under the hood while the index reference remains perfectly valid.

lp_flags the state of the pointer. The values are: 0 = LP_UNUSED (available for reuse), 1 = LP_NORMAL (points to a live tuple), 2 = LP_REDIRECT (points to another line pointer, used after HOT pruning), 3 = LP_DEAD (the tuple has been determined dead but the pointer persists until cleanup).
lp_len the total length of the tuple in bytes, including the 23-byte tuple header, null bitmap, alignment padding, and actual column data.

The anatomy of a page

Here is how all these pieces fit together inside the 8KB block.

We already mentioned it above, but if you review the image carefully you can confirm the important insight. While line pointers are allocated downward from the header (pd_lower increases) the tuple data is stored upwards from the bottom of the page (pd_upper decreases). Free space is the gap between them.

This is called Slotted page layout and it's easy way how to prevent unnecessary data fragmentation. By storing predictable line pointers at top, they don't mix together with bulky and unpredicable tuples.

Free space: the gap in the middle

The free space on a page is the region between pd_lower and pd_upper. For our page:

SELECT lower, upper, upper - lower AS free_space
FROM page_header(get_raw_page('page_demo', 0));

 lower | upper | free_space
-------+-------+------------
    36 |  7976 |       7940
(1 row)

With only three small rows, we have 7,940 bytes of free space. Almost the entire page is still available. Let's see what happens when we add more data:

INSERT INTO page_demo (title, created_at, value)
SELECT 
    'Generated item ' || i, 
    '2026-01-15 00:00:00+00'::timestamptz + (i || ' hours')::interval, 
    (i * 11.11)::numeric(10,2)
FROM generate_series(1, 10) AS i;

SELECT lower, upper, upper - lower AS free_space
FROM page_header(get_raw_page('page_demo', 0));

 lower | upper | free_space
-------+-------+------------
    76 |  7336 |       7260
(1 row)

You can observe

pd_lower moved from 36 to 76. Why? With 10 new line pointers at 4 bytes each we moved by 40 bytes.
pd_upper dropped from 7,976 to 7,336. The 10 new tuples consumed 640 bytes of the data space.
And the free space shrank to 7,260 bytes.

How fast does a page fill up?

If we going to think about the page capacity, let's consider each row in this page costs:

4 bytes for the line pointer
23 bytes for the tuple header (transaction metadata, null bitmap offset, info mask)
Alignment padding to the nearest 8-byte boundary after the header (1 byte of padding, bringing the header to 24 bytes)
Actual column data: 4 bytes for the integer, variable bytes for the text, variable bytes for the numeric

For our new schema, each tuple is a bit larger than before. We saw earlier that our first 3 tuples took up exactly 216 bytes, which averages exactly 72 bytes per tuple. Add the 4-byte line pointer, and each row costs about 76 bytes of page space.

With 8,192 bytes in a page minus the 24-byte header, we have 8,168 bytes of usable space. At roughly 76 bytes per row, we can theoretically fit approximately 107 rows on a single page.

Let's test it. We already have 13 rows (3 original plus 10 generated). Let's insert more and check:

INSERT INTO page_demo (title, created_at, value)
SELECT
    'Generated item ' || i,
    '2026-01-15 00:00:00+00'::timestamptz + (i || ' hours')::interval,
    (i * 11.11)::numeric(10,2)
FROM generate_series(11, 150) AS i;

SELECT count(*) AS tuples_on_page_0
FROM heap_page_items(get_raw_page('page_demo', 0));

Total of 119 tuples fit on page 0 before PostgreSQL had to start using page 1. A bit more than our estimate of 107 -- the shorter generated titles take less space than "Ergonomic standing desk, oak finish".

The exact number always depends on column values, but this gives you a practical sense of capacity. Because this schema includes a variable-length text column, the row size flexes, but we still hit a very predictable ceiling right around 110-120 rows per 8KB block.

If you want to estimate how many pages a table will need, the rough formula is: rows * average_row_size / 8192. But remember that "average_row_size" includes the 23-byte tuple header, alignment padding, and the 4-byte line pointer.

PostgreSQL's pg_column_size() function can help you measure actual row sizes.

Spanning multiple pages

With 160 rows in our table, we have started storing the tuples into a second page. Let's verify using the system catalog:

SELECT relpages, reltuples
FROM pg_class
WHERE relname = 'page_demo';

The relpages and reltuples values are estimates updated by ANALYZE and autovacuum. After bulk inserts, run ANALYZE page_demo; to refresh them.

relpages | reltuples
----------+-----------
        2 |       160
(1 row)

Two pages. Let's insert substantially more data to see a real multi-page table:

INSERT INTO page_demo (title, created_at, value)
SELECT 
    'Generated row ' || i, 
    CURRENT_TIMESTAMP + (i || ' minutes')::interval,
    (random() * 1000)::numeric(10,2)
FROM generate_series(1, 500) AS i;

ANALYZE page_demo;

SELECT relpages, reltuples
FROM pg_class
WHERE relname = 'page_demo';

 relpages | reltuples
----------+-----------
        6 |       660
(1 row)

Six pages now. Each page is an independent 8KB block with its own header. Let's confirm by reading the headers of the first two:

SELECT 0 AS page, lower, upper, upper - lower AS free_space
FROM page_header(get_raw_page('page_demo', 0))
UNION ALL
SELECT 1, lower, upper, upper - lower
FROM page_header(get_raw_page('page_demo', 1));

 page | lower | upper | free_space
------+-------+-------+------------
    0 |   500 |   552 |         52
    1 |   504 |   512 |          8
(2 rows)

Both pages are nearly full, with only 52/8 bytes of free space remaining. Which is too little for another row to fit there.

The space that isn't there

You may have noticed that pd_special equals 8192 for our heap pages. The same size as the entire page size. I.e. there's no special space allocated.

This applies to the heap pages. But not all pages are heap pages. Index pages use the special space at the end of the page to store index-specific metadata. For a B-tree index, it includes pointers to sibling pages (for range scans), the tree level, and flags about the page type (leaf, internal, root, deleted).

Let's peek at the primary key index that was automatically created for our table:

SELECT type, live_items, dead_items, avg_item_size,
       page_size, free_size
FROM bt_page_stats('page_demo_pkey', 1);

 type | live_items | dead_items | avg_item_size | page_size | free_size
------+------------+------------+---------------+-----------+-----------
 l    |        367 |          0 |            16 |      8192 |       808
(1 row)

The type is l for leaf page. And if we check its page header:

SELECT special FROM page_header(get_raw_page('page_demo_pkey', 1));

 special
---------
    8176
(1 row)

The special space starts at byte 8176, giving us 16 bytes (8192 - 8176) of B-tree metadata at the end of the page. Heap pages have none; index pages rely on it.

Putting it all together

Let's close with a complete view of our page. We know the header, the line pointers, the free space, and the tuple data regions. Here is a summary query that shows all of it at once:

SELECT
    'header'       AS region,
    0              AS start_byte,
    23             AS end_byte,
    24             AS size_bytes
UNION ALL
SELECT
    'line pointers',
    24,
    lower - 1,
    lower - 24
FROM page_header(get_raw_page('page_demo', 0))
UNION ALL
SELECT
    'free space',
    lower,
    upper - 1,
    upper - lower
FROM page_header(get_raw_page('page_demo', 0))
UNION ALL
SELECT
    'tuple data',
    upper,
    special - 1,
    special - upper
FROM page_header(get_raw_page('page_demo', 0))
UNION ALL
SELECT
    'special space',
    special,
    8191,
    8192 - special
FROM page_header(get_raw_page('page_demo', 0));

    region     | start_byte | end_byte | size_bytes
---------------+------------+----------+------------
 header        |          0 |       23 |         24
 line pointers |         24 |      499 |        476
 free space    |        500 |      551 |         52
 tuple data    |        552 |     8191 |       7640
 special space |       8192 |     8191 |          0
(5 rows)

476 bytes of line pointers means 119 entries (476 / 4). 7,640 bytes of tuple data. 52 bytes of free space. And zero bytes of special space (it's heap page).

That is the entire PostgreSQL 8KB page. Twenty-four bytes of header tell PostgreSQL where everything is. Line pointers provide table of contents.

Introduction to Buffers in PostgreSQL

2026-01-24T16:57:00+00:00

The work around RegreSQL led me to focus a lot on buffers. If you are a casual PostgreSQL user, you have probably heard about adjusting shared_buffers and followed the good old advice to set it to 1/4 of available RAM. But after we went a little bit too enthusiastic about them on a recent Postgres FM episode I've been asked what that's all about.

Buffers are one of those topics that easily gets forgotten. And while they are a foundation block of PostgreSQL's performance architecture, most of us treat them as a black box.

The 8KB page

PostgreSQL stores and retrieves data in 8KB blocks. When reading or writing a single row, the engine operates on the entire 8KB page that contains it. You retrieve or write one row, and the surrounding page comes along with it.

-- you can check the block size (which should be almost always 8192 bytes)
show block_size;

 block_size
------------
 8192
(1 row)

Every table and index is a collection of these pages. A row might span multiple pages if it's large enough, but the page remains the atomic unit of I/O.

Want to look inside?
If you want to see exactly how PostgreSQL packs your data, headers, and line pointers into these 8,192 bytes, check out the byte-level tour in Inside PostgreSQL's 8KB Page.

PostgreSQL vs OS

Why maintain a dedicated buffer cache when the operating system already caches disk pages?

PostgreSQL understands the semantics of the data it touches. The operating system only sees files, offsets, and bytes; PostgreSQL sees tables, indexes, and execution plans. When a query runs a large sequential scan, the OS happily caches every page it encounters, but PostgreSQL knows the scan is a transient bulk read and diverts it into a small ring buffer to protect the main pool from eviction.

The second requirement is ACID durability: PostgreSQL must guarantee that write-ahead log (WAL) records reach disk before a modified data page is written. The kernel page cache does not enforce that ordering.

shared_buffers

The shared_buffers parameter controls the size of the shared memory pool accessible to all backends. When a backend needs a page, it first checks shared buffers. If the page is present, it is a cache hit and requires no disk I/O. On a miss, PostgreSQL loads the page from disk (or the OS cache) into shared buffers for subsequent queries.

WAL has its own buffer area (wal_buffers). This separation exists because WAL writes are sequential and must be persisted before the corresponding data change can be considered committed. The default is 3% of shared_buffers, capped at 16MB (one WAL segment).

show shared_buffers;

 shared_buffers
----------------
 128MB
(1 row)

The default value 128MB is very conservative and is part of making sure the PostgreSQL default installation will work pretty much on any system (including those with limited RAM). But in your regular environment this value should typically be much higher.

The value of 128MB there refers to the actual buffered content. If you consider a single page being 8KB, you can imagine this as 16,384 individual slots for storing the data.

Interactive Visualizer

Explore how PostgreSQL's buffer pool actually works. Watch the clock sweep algorithm evict cold pages, see buffers transition between clean, dirty, and pinned states, and understand how the hash table enables O(1) lookups.

Open Visualizer

Beyond the data pages, PostgreSQL tracks metadata and maintains lookup indices in shared memory across three components:

buffer blocks: the actual 8KB pages where data lives.
buffer descriptors: an array of ~64-byte structures, one per buffer slot.
buffer mapping hash table: maps page identifiers to buffer slot indices.

Each descriptor tracks which page occupies the slot (the buffer tag), state flags (dirty, valid, I/O in progress), and two counters: pin count and usage count.

O(1) constant time, regardless of buffer pool size.

When a backend requests page N of a table, it hashes the page tag and jumps directly to the hash bucket, avoiding a scan across every slot in the pool.

Pin and usage counts

What happens to each buffer slot is ruled by two counters: pin count and usage count.

Pin count tracks active references. When a backend is actively reading or modifying a page, it pins the buffer so it cannot be evicted. When the backend finishes, it unpins the buffer.

Usage count tracks how recently and frequently a buffer was accessed. Each access increments the count (capped at 5). During eviction, the clock sweep decrements this value. Buffers with higher counts survive longer, while those at 0 get evicted.

The usage counter prevents a single large read from wiping out frequently accessed pages. A sequential scan over a cold table touches each page once, leaving its usage count at 1, while actively queried pages accumulate counts up to 5 and survive longer under eviction pressure.

The clock sweep algorithm

When PostgreSQL needs to load a page and every buffer slot is occupied, it runs the clock sweep to find a candidate for eviction.

A traditional LRU linked list would require locks on every cache hit, creating severe lock contention under concurrent read workloads.

The clock sweep avoids that bottleneck by sweeping circularly through the array of buffer descriptors:

If the buffer is pinned, skip it.
If usagecount > 0, decrement it by 1 and advance to the next descriptor.
If usagecount == 0 and the buffer is unpinned, choose this buffer for eviction.

Cold pages with a usage count of 1 are reclaimed on the first or second pass. Frequently queried pages with higher counts survive multiple sweeps.

Dirty buffers and the background writer

Every modification is logged to WAL before the buffer in shared memory is dirtied.

When a backend modifies a page, the buffer becomes dirty. Writing every modification to disk immediately would be slow, especially when the same page is updated repeatedly in rapid succession.

Instead, dirty buffers accumulate in shared memory and are flushed through three paths:

Checkpoints: a periodic background process writes all dirty buffers to disk and fsyncs them, creating a point where data files are consistent with WAL.
The background writer: continuously scans for dirty buffers and writes them out ahead of eviction. When a backend needs a buffer slot, it finds already-clean buffers and avoids waiting on write I/O.
Backend allocation: if the clock sweep selects a dirty buffer that neither checkpoint nor background writer has flushed, the backend must write the page out synchronously before reusing the slot.

The goal is keeping enough clean buffers available so backends never stall on synchronous writes during eviction.

Watching the buffer pool work

The pg_buffercache extension exposes one row per buffer slot, including page identity, usage count, and dirty status. On PostgreSQL 17 with the default 128MB shared_buffers:

CREATE EXTENSION pg_buffercache;

-- 500 rows, small enough that every page fits in the buffer pool
CREATE TABLE demo_orders (id int, payload text)
    WITH (autovacuum_enabled = off);

INSERT INTO demo_orders
SELECT i, repeat('x', 100) FROM generate_series(1, 500) i;

-- the state of the table's pages inside shared buffers
SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | t       |     9
(1 row)

All 9 heap pages are resident and marked dirty. Pinning during the insert drove each page's usage count to the maximum of 5. The relforknumber = 0 filter restricts the query to the heap fork; omitting it also pulls in the table's free space map.

CHECKPOINT flushes the dirty pages to disk while leaving them cached:

CHECKPOINT;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | f       |     9
(1 row)

The dirty flag flips to false. A read-only query marks them dirty again:

SELECT count(*) FROM demo_orders;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | t       |     9
(1 row)

The scan set commit-status hint bits on the new tuples. Writing hint bits modifies the page header in shared memory, which is why EXPLAIN (ANALYZE, BUFFERS) can report dirtied buffers on pure SELECT queries.

Loading a table significantly larger than shared_buffers forces the clock sweep through multiple passes, decrementing usage counts to zero and reclaiming the buffers:

CREATE TABLE filler (id int, payload text)
    WITH (autovacuum_enabled = off);

INSERT INTO filler
SELECT i, repeat('z', 200) FROM generate_series(1, 3600000) i;
-- 827MB insert through a 128MB pool

SELECT count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0;

 count
-------
     0
(1 row)

-- reading the table loads pages back from disk, usagecount starts at 1
SELECT count(*) FROM demo_orders;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          1 | f       |     9
(1 row)

demo_orders was evicted once the clock sweep decremented its usage count to zero. Reading the table again loads the pages with usagecount = 1. Because the eviction flush had already written the hint bits to disk, this read leaves the buffers clean.

Ring buffers

As mentioned above there are also special types of buffers. We already touched the scenario when a query scans a large table.

In a naive implementation, the sequential scan would load all data into shared buffers, evicting everything else along the way. Your warmed cache would vanish and subsequent queries would suffer for minutes to come.

PostgreSQL's solution for this case is ring buffers. Small, private buffer pools for bulk operations. Instead of using the shared buffer pool, certain operations get their own limited ring.

The individual cases are:

Sequential scans on large tables over 1/4 of shared_buffers use a dedicated 256 KB ring buffer. Pages cycle through this tiny ring and never touch the main cache.

-- perform a sequential scan over a large table
EXPLAIN (ANALYZE, BUFFERS) SELECT count(1) FROM ring_buffer_test;

Which would show you Buffers: shared read=127285 instead of hit. The follow-up article Reading Buffer Statistics in EXPLAIN Output goes through those counters line by line.

Bulk writes (COPY, CREATE TABLE AS) use a 16MB capped ring buffer, large enough for efficient batching, yet small enough not to pollute the shared buffer pool.

As VACUUM touches every page and shouldn't evict the hot data, it uses its own dedicated ring buffer. Historically it was set to 256KB, but since PostgreSQL 16 it can be configured using vacuum_buffer_usage_limit (the previous two sizes are fixed).

Local buffers

The second kind of exception to shared buffer pool, is the session based temporary tables. In this case as the concurrency is out of question, each backend has its own local buffer pool, controlled by temp_buffers (8MB default).

Local buffers are faster than shared buffers because they have simpler locking. There is no need for the heavy cross-process coordination required in the main cache.

While this might seem like a minor implementation detail, it can translate into a powerful optimisation strategy. Many developers tend to default to complex CTE logic for intermediate data, but using temporary tables offers a distinct advantage in the form of lower I/O as the changes to temporary tables are not WAL logged and by their nature reduce pollution of shared buffer pools.

If your workload involves large temporary tables, increasing temp_buffers can help keep those operations purely in RAM. However, remember that this memory is per-connection, so it multiplies across all backends.

The OS Page Cache

PostgreSQL does not bypass the kernel. Every read and write goes through the operating system page cache, creating double buffering: the same 8KB page can exist in both PostgreSQL's shared buffers and kernel memory simultaneously.

The OS cache acts as a second-tier cache; evicted clean pages often remain in kernel memory.

When PostgreSQL evicts a clean page, that page usually remains in the OS page cache. If another query requests it shortly after, the kernel serves it from RAM without disk I/O. The kernel also provides read-ahead, prefetching sequential pages before PostgreSQL asks for them.

This coordination is why standard guidance recommends setting shared_buffers to roughly 25% of RAM on dedicated servers, leaving the remainder for the OS page cache and per-backend memory.

-- typical starting point for a dedicated database server
ALTER SYSTEM SET shared_buffers = '8GB';

This parameter allocates no memory; it is purely a cost hint for the planner.

The query planner needs an estimate of total available cache (shared buffers plus OS page cache) when calculating plan costs. That estimate is set by effective_cache_size:

SHOW effective_cache_size;

A higher value encourages index scans over sequential scans by assuming requested pages are likely already cached in RAM.

Where to go next

The same buffer mechanics appear at the query level. EXPLAIN (ANALYZE, BUFFERS) reports exact counts for pages hit in shared memory, read from disk or the OS cache, and dirtied by writes. How to inspect and tune those query-level counters is covered in Reading Buffer Statistics in EXPLAIN Output.

boringSQL | Supercharge your SQL & PostgreSQL powers - internals

How TABLESAMPLE picks rows

Setup

Every row has an address

Pages come as bundles

REPEATABLE remembers addresses

Hashing the key instead

The join doesn't shrink

Which to reach for

Postgres relation files, filed under base/

Setup

From table name to file name

The files you didn't create

Forks

The same bytes, two ways

OID versus relfilenode

Tablespaces

Segment files

Where PostgreSQL stores row locks

Setup

What FOR UPDATE writes

The stamp outlives the lock

Four modes, four bit patterns

Two lockers: the MultiXactId

Foreign keys take FOR KEY SHARE

Multixacts on disk

PostgreSQL's MVCC is bad. So is everyone else's.

The charges against PostgreSQL

Charge 1: write amplification

Charge 2: your table is an accident scene

Charge 3: one idle transaction poisons the well

Charge 4: the 32-bit debt

The undo camp: Oracle and InnoDB

SQL Server: versioning as an optional extra

MongoDB: versions in the cache

The LSM generation: garbage collection as a lifestyle

Outside the RDBMS entirely: etcd

Fixing PostgreSQL itself

The scorecard

VACUUM at the Page Level

Setup

Snapshot before any deletes

Create dead tuples

How VACUUM processes a table

Phase 1: Heap scan - prune, freeze, collect dead TIDs

Phase 2: Index cleanup

Phase 3: Heap cleanup - freeing the line pointers

After the prune: LP_DEAD, and space already back

After the full VACUUM: LP_UNUSED

The line pointer lifecycle, precisely

Free space map

Visibility map

Freezing

VACUUM FULL vs regular VACUUM

Putting it all together

TOAST: Where PostgreSQL hides big values

The 2KB threshold

The four storage strategies

The shrinking procedure

Watching it happen

What the heap tuple looks like

Compression: pglz vs lz4

What it costs you

Limits

Following a tuple through the toaster

Keeping the page invariant safe

HOT Updates in Postgres

Cost of a normal UPDATE

Bypassing index updates

Demonstrating a HOT update

The ctid chain

Page pruning

LP_REDIRECT

When HOT fails

fillfactor: making room for HOT

pd_prune_xid revisited

HOT in one paragraph

What comes next

PostgreSQL MVCC, Byte by Byte

xmin and xmax: the only two XIDs that matter