Introduction to Buffers in PostgreSQL

The work around RegreSQL led me to focus a lot on buffers. If you are a casual PostgreSQL user, you have probably heard about adjusting shared_buffers and followed the good old advice to set it to 1/4 of available RAM. But after we went a little bit too enthusiastic about them on a recent Postgres FM episode I've been asked what that's all about.

Buffers are one of those topics that easily gets forgotten. And while they are a foundation block of PostgreSQL's performance architecture, most of us treat them as a black box.

The 8KB page

PostgreSQL stores and retrieves data in 8KB blocks. When reading or writing a single row, the engine operates on the entire 8KB page that contains it. You retrieve or write one row, and the surrounding page comes along with it.

-- you can check the block size (which should be almost always 8192 bytes)
show block_size;

 block_size
------------
 8192
(1 row)

Every table and index is a collection of these pages. A row might span multiple pages if it's large enough, but the page remains the atomic unit of I/O.

PostgreSQL vs OS

Why maintain a dedicated buffer cache when the operating system already caches disk pages?

PostgreSQL understands the semantics of the data it touches. The operating system only sees files, offsets, and bytes; PostgreSQL sees tables, indexes, and execution plans. When a query runs a large sequential scan, the OS happily caches every page it encounters, but PostgreSQL knows the scan is a transient bulk read and diverts it into a small ring buffer to protect the main pool from eviction.

The second requirement is ACID durability: PostgreSQL must guarantee that write-ahead log (WAL) records reach disk before a modified data page is written. The kernel page cache does not enforce that ordering.

shared_buffers

The shared_buffers parameter controls the size of the shared memory pool accessible to all backends. When a backend needs a page, it first checks shared buffers. If the page is present, it is a cache hit and requires no disk I/O. On a miss, PostgreSQL loads the page from disk (or the OS cache) into shared buffers for subsequent queries.

show shared_buffers;

 shared_buffers
----------------
 128MB
(1 row)

The default value 128MB is very conservative and is part of making sure the PostgreSQL default installation will work pretty much on any system (including those with limited RAM). But in your regular environment this value should typically be much higher.

The value of 128MB there refers to the actual buffered content. If you consider a single page being 8KB, you can imagine this as 16,384 individual slots for storing the data.

Beyond the data pages, PostgreSQL tracks metadata and maintains lookup indices in shared memory across three components:

• buffer blocks: the actual 8KB pages where data lives.
• buffer descriptors: an array of ~64-byte structures, one per buffer slot.
• buffer mapping hash table: maps page identifiers to buffer slot indices.

Each descriptor tracks which page occupies the slot (the buffer tag), state flags (dirty, valid, I/O in progress), and two counters: pin count and usage count.

When a backend requests page N of a table, it hashes the page tag and jumps directly to the hash bucket, avoiding a scan across every slot in the pool.

Pin and usage counts

What happens to each buffer slot is ruled by two counters: pin count and usage count.

Pin count tracks active references. When a backend is actively reading or modifying a page, it pins the buffer so it cannot be evicted. When the backend finishes, it unpins the buffer.

Usage count tracks how recently and frequently a buffer was accessed. Each access increments the count (capped at 5). During eviction, the clock sweep decrements this value. Buffers with higher counts survive longer, while those at 0 get evicted.

The usage counter prevents a single large read from wiping out frequently accessed pages. A sequential scan over a cold table touches each page once, leaving its usage count at 1, while actively queried pages accumulate counts up to 5 and survive longer under eviction pressure.

The clock sweep algorithm

When PostgreSQL needs to load a page and every buffer slot is occupied, it runs the clock sweep to find a candidate for eviction.

The clock sweep avoids that bottleneck by sweeping circularly through the array of buffer descriptors:

1. If the buffer is pinned, skip it.
2. If usagecount > 0, decrement it by 1 and advance to the next descriptor.
3. If usagecount == 0 and the buffer is unpinned, choose this buffer for eviction.

Cold pages with a usage count of 1 are reclaimed on the first or second pass. Frequently queried pages with higher counts survive multiple sweeps.

Dirty buffers and the background writer

When a backend modifies a page, the buffer becomes dirty. Writing every modification to disk immediately would be slow, especially when the same page is updated repeatedly in rapid succession.

Instead, dirty buffers accumulate in shared memory and are flushed through three paths:

• Checkpoints: a periodic background process writes all dirty buffers to disk and fsyncs them, creating a point where data files are consistent with WAL.
• The background writer: continuously scans for dirty buffers and writes them out ahead of eviction. When a backend needs a buffer slot, it finds already-clean buffers and avoids waiting on write I/O.
• Backend allocation: if the clock sweep selects a dirty buffer that neither checkpoint nor background writer has flushed, the backend must write the page out synchronously before reusing the slot.

The goal is keeping enough clean buffers available so backends never stall on synchronous writes during eviction.

Watching the buffer pool work

The pg_buffercache extension exposes one row per buffer slot, including page identity, usage count, and dirty status. On PostgreSQL 17 with the default 128MB shared_buffers:

CREATE EXTENSION pg_buffercache;

-- 500 rows, small enough that every page fits in the buffer pool
CREATE TABLE demo_orders (id int, payload text)
    WITH (autovacuum_enabled = off);

INSERT INTO demo_orders
SELECT i, repeat('x', 100) FROM generate_series(1, 500) i;

-- the state of the table's pages inside shared buffers
SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | t       |     9
(1 row)

All 9 heap pages are resident and marked dirty. Pinning during the insert drove each page's usage count to the maximum of 5. The relforknumber = 0 filter restricts the query to the heap fork; omitting it also pulls in the table's free space map.

CHECKPOINT flushes the dirty pages to disk while leaving them cached:

CHECKPOINT;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | f       |     9
(1 row)

The dirty flag flips to false. A read-only query marks them dirty again:

SELECT count(*) FROM demo_orders;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          5 | t       |     9
(1 row)

The scan set commit-status hint bits on the new tuples. Writing hint bits modifies the page header in shared memory, which is why EXPLAIN (ANALYZE, BUFFERS) can report dirtied buffers on pure SELECT queries.

Loading a table significantly larger than shared_buffers forces the clock sweep through multiple passes, decrementing usage counts to zero and reclaiming the buffers:

CREATE TABLE filler (id int, payload text)
    WITH (autovacuum_enabled = off);

INSERT INTO filler
SELECT i, repeat('z', 200) FROM generate_series(1, 3600000) i;
-- 827MB insert through a 128MB pool

SELECT count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0;

 count
-------
     0
(1 row)

-- reading the table loads pages back from disk, usagecount starts at 1
SELECT count(*) FROM demo_orders;

SELECT usagecount, isdirty, count(*)
FROM pg_buffercache
WHERE relfilenode = 'demo_orders'::regclass::oid
  AND relforknumber = 0
GROUP BY 1, 2
ORDER BY 1, 2;

 usagecount | isdirty | count
------------+---------+-------
          1 | f       |     9
(1 row)

demo_orders was evicted once the clock sweep decremented its usage count to zero. Reading the table again loads the pages with usagecount = 1. Because the eviction flush had already written the hint bits to disk, this read leaves the buffers clean.

Ring buffers

As mentioned above there are also special types of buffers. We already touched the scenario when a query scans a large table.

In a naive implementation, the sequential scan would load all data into shared buffers, evicting everything else along the way. Your warmed cache would vanish and subsequent queries would suffer for minutes to come.

PostgreSQL's solution for this case is ring buffers. Small, private buffer pools for bulk operations. Instead of using the shared buffer pool, certain operations get their own limited ring.

The individual cases are:

Sequential scans on large tables over 1/4 of shared_buffers use a dedicated 256 KB ring buffer. Pages cycle through this tiny ring and never touch the main cache.

-- perform a sequential scan over a large table
EXPLAIN (ANALYZE, BUFFERS) SELECT count(1) FROM ring_buffer_test;

Which would show you Buffers: shared read=127285 instead of hit. The follow-up article Reading Buffer Statistics in EXPLAIN Output goes through those counters line by line.

Bulk writes (COPY, CREATE TABLE AS) use a 16MB capped ring buffer, large enough for efficient batching, yet small enough not to pollute the shared buffer pool.

As VACUUM touches every page and shouldn't evict the hot data, it uses its own dedicated ring buffer. Historically it was set to 256KB, but since PostgreSQL 16 it can be configured using vacuum_buffer_usage_limit (the previous two sizes are fixed).

Local buffers

The second kind of exception to shared buffer pool, is the session based temporary tables. In this case as the concurrency is out of question, each backend has its own local buffer pool, controlled by temp_buffers (8MB default).

Local buffers are faster than shared buffers because they have simpler locking. There is no need for the heavy cross-process coordination required in the main cache.

While this might seem like a minor implementation detail, it can translate into a powerful optimisation strategy. Many developers tend to default to complex CTE logic for intermediate data, but using temporary tables offers a distinct advantage in the form of lower I/O as the changes to temporary tables are not WAL logged and by their nature reduce pollution of shared buffer pools.

If your workload involves large temporary tables, increasing temp_buffers can help keep those operations purely in RAM. However, remember that this memory is per-connection, so it multiplies across all backends.

The OS Page Cache

PostgreSQL does not bypass the kernel. Every read and write goes through the operating system page cache, creating double buffering: the same 8KB page can exist in both PostgreSQL's shared buffers and kernel memory simultaneously.

When PostgreSQL evicts a clean page, that page usually remains in the OS page cache. If another query requests it shortly after, the kernel serves it from RAM without disk I/O. The kernel also provides read-ahead, prefetching sequential pages before PostgreSQL asks for them.

This coordination is why standard guidance recommends setting shared_buffers to roughly 25% of RAM on dedicated servers, leaving the remainder for the OS page cache and per-backend memory.

-- typical starting point for a dedicated database server
ALTER SYSTEM SET shared_buffers = '8GB';

The query planner needs an estimate of total available cache (shared buffers plus OS page cache) when calculating plan costs. That estimate is set by effective_cache_size:

SHOW effective_cache_size;

A higher value encourages index scans over sequential scans by assuming requested pages are likely already cached in RAM.

Where to go next

The same buffer mechanics appear at the query level. EXPLAIN (ANALYZE, BUFFERS) reports exact counts for pages hit in shared memory, read from disk or the OS cache, and dirtied by writes. How to inspect and tune those query-level counters is covered in Reading Buffer Statistics in EXPLAIN Output.