Flag</th>	Effect</th></tr></thead>
`--statistics</code></td>`	dump the statistics (you have to request it explicitely)</td></tr>
`--statistics-only</code></td>`	dump only the statistics, not schema or data</td></tr>
`--no-statistics</code></td>`	do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database</p> pg_dump --statistics-only -d production_db > stats.sql</span></span> </span></code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above.</p> The full workflow to turn your production data into testable plans might look like this:</p> # 1. dump schema from production</span></span> pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span> </span> # 2. dump statistics from production</span></span> pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span> </span> # 3. create test database with schema</span></span> createdb</span> test_db</span></span> psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span> </span> # 4. load fixture data (optional; masked, minimal)</span></span> psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span> </span> # 5. inject production statistics</span></span> psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span> </span> # 6. query plans now match production</span></span> psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</span></span></code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started.</p> To prevent this, disable autovacuum analyze on the tables you've injected.</p> -- disable autovacuum</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_enabled </span>=</span> false);</span></span> </span> -- or set analyze threshold so high it nevers kicks-in</span></span> ALTER TABLE</span> test_orders </span>SET</span> (autovacuum_analyze_threshold </span>=</span> 2147483647</span>);</span></span></code></pre> Be careful here.</strong> If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. </p> For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run.</p> And please, never ever do this in production!</p> </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups).</p> It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>.</p> Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>.</p> The easiest way to grant it for automation:</p> GRANT</span> pg_maintain </span>TO</span> ci_service_account;</span></span></code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser.</p> Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment?</p> When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden.</p> You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run</p> CREATE DATABASE dbname;</span></span></code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use</p> CREATE DATABASE dbname TEMPLATE template1;</span></span></code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>.</p> In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine.</p> CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic.</p> Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time.</p> The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options.</p> FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files.</p> When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>.</p> With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space.</p> Quick setup checklist:</strong></p> Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database.</p> CREATE DATABASE</span> source_db</span>;</span></span> \c source_db</span></span> </span> CREATE TABLE</span> boring_data</span> (</span></span> id </span>serial PRIMARY KEY</span>,</span></span> payload </span>text</span></span> );</span></span> </span> -- generate 50m rows</span></span> INSERT INTO</span> boring_data (payload)</span></span> SELECT</span> md5(random()::</span>text</span>) </span>\|\|</span> md5(random()::</span>text</span>)</span></span> FROM</span> generate_series</span>(</span>1</span>, </span>50000000</span>);</span></span> </span> -- force a checkpoint</span></span> CHECKPOINT</span>;</span></span></code></pre> You can verify the database now has roughly 6GB of data.</p> Name \| source_db</span></span> Owner \| postgres</span></span> Encoding \| UTF8</span></span> Locale Provider \| libc</span></span> Collate \| en_US.UTF-8</span></span> Ctype \| en_US.UTF-8</span></span> Locale \|</span></span> ICU Rules \|</span></span> Access privileges \|</span></span> Size \| 6289 MB</span></span> Tablespace \| pg_default</span></span> Description \|</span></span></code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get</p> CREATE DATABASE slow_copy TEMPLATE source_db;</span></span> CREATE DATABASE</span></span> Time: 67000.615 ms (01:07.001)</span></span></code></pre> Now, let's verify our configuration is set for speed:</p> show file_copy_method;</span></span> file_copy_method</span></span> ------------------</span></span> clone</span></span> (1 row)</span></span></code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time.</p> CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY;</span></span> CREATE DATABASE</span></span> Time: 212.053 ms</span></span></code></pre> That's a quite an improvement, isn't it?</p> Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes?</p> When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage.</p> This can create some initial confusion. If you ask PostgreSQL for the size:</p> SELECT</span> pg_database_size(</span>'source_db'</span>) </span>as</span> source,</span></span> pg_database_size(</span>'fast_clone'</span>) </span>as</span> clone;</span></span></code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size.</p> -[ RECORD 1 ]------</span></span> source \| 6594041535</span></span> clone \| 6594041535</span></span></code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page.</p> A single UPDATE will therefore trigger copy-on-write on multiple pages:</p> the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage.</p> XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql#</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 2031: 10471550.. 10473581: 2032: shared</span></span> 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</span></span></code></pre> All it takes is to update some rows using</p> update</span> boring_data </span>set</span> payload </span>=</span> 'new value'</span> \|\|</span> id </span>where</span> id </span>IN</span> (</span>select</span> id </span>from</span> boring_data </span>limit</span> 20</span>);</span></span></code></pre> and the situation will start to change.</p> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10471550.. 10471589: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16402/16404: 7 extents found</span></span> root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404</span></span> Filesystem type is: 58465342</span></span> File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes)</span></span> ext: logical_offset: physical_offset: length: expected: flags:</span></span> 0: 0.. 39: 10297326.. 10297365: 40:</span></span> 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared</span></span> 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared</span></span> 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared</span></span> 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared</span></span> 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared</span></span> 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared</span></span> 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof</span></span> /var/lib/postgresql/18/main/base/16418/16404: 8 extents found</span></span> root@clone-demo:/var/lib/postgresql#</span></span></code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared.</p> Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections.</p> Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy.</p> Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it.</p> Custom PostgreSQL extensions with Rust 2024-05-24T00:00:00+00:00 This article explores the pgrx framework, which simplifies the creation of custom PostgreSQL extensions to bring more logic closer to your database. Traditionally, writing such extensions required familiarity with C and a deep understanding of PostgreSQL internals, which could be quite challenging. pgrx</code> lowers the barrier and allows developers to use Rust, known for its safety and performance, making the process of creating efficient and safe database extensions much more accessible.</p> pg_sysload</a> </h2> When working with large datasets and migrations (as discussed in How Not to Change PostgreSQL Column Type</a>), or during resource-intensive maintenance tasks, you'll want to optimise speed and minimise disruption to other processes. One way to control the pace of batch operations is to consider the load on the underlying system.</p> Many Unix-based systems (we will focus on Linux) provide a valuable metric called the system load average</strong>. This average consists of three values: the 1-minute, 5-minute, and 15-minute load averages. The load average is not normalised for the number of CPU cores, so a load average of 1 on a single-core system means full utilisation, while on a quad-core system, it indicates 25% utilisation.</p> In many cases, the system load average is also an excellent indicator of how ongoing operations are impacting a busy database cluster. In this article, we will create a PostgreSQL extension with a function called sys_loadavg()</code> that retrieves this load information. We will use the /proc/loadavg</code> file (part of the proc filesystem</strong>), which exposes underlying system details.</p> Getting Started with pgrx</code></a> </h2> Before we start, ensure you have:</p> Rust installed (Get Started with Rust</a>)</li> System dependencies</a> installed for pgrx</code></li> </ul> With these prerequisites in place, you can install pgrx</code> itself and create a new extension skeleton:</p> cargo</span> install</span> --locked</span> cargo-pgrx</span></span> cargo</span> pgrx new pg_sysload</span></span> cd</span> pg_sysload</span></span></code></pre> This gives you a complete environment for developing your own PostgreSQL extensions in Rust. While Rust might seem daunting at first (especially with its async and other features), the language itself is quite powerful. Its ease of extension creation makes it worth a second look.</p> Setting Up the Extension</a> </h2> While the pgrx new</code> command sets up a basic template, we will create a more sophisticated extension. Its primary logic involves parsing the /proc/loadavg</code> file, extracting its values, and returning them to the user.</p> The logic itself is straightforward:</p> #[pg_extern]</span></span> fn</span> sys_loadavg</span>()</span> -></span> Option</span><</span>Vec</span><</span>f64</span>>> {</span></span> // Read the contents of the /proc/loadavg</span></span> let mut</span> file</span> = match</span> fs</span>::</span>File</span>::</span>open</span>(</span>"/proc/loadavg"</span>) {</span></span> Ok</span>(file)</span> =></span> file,</span></span> Err</span>(err)</span> =></span> {</span></span> pgrx</span>::</span>error!</span>(</span>"Error reading /proc/loadavg: {}"</span>, err);</span></span> }</span></span> };</span></span> let mut</span> contents</span> =</span> String</span>::</span>new</span>();</span></span> if let</span> Err</span>(err)</span> =</span> file</span>.</span>read_to_string</span>(</span>&mut</span> contents) {</span></span> pgrx</span>::</span>error!</span>(</span>"Error reading /proc/loadavg: {}"</span>, err);</span></span> }</span></span> </span> // Extract the load average fields</span></span> let</span> fields</span> =</span> contents</span>.</span>split_whitespace</span>()</span>.</span>collect</span>::</span><</span>Vec</span><_>>();</span></span> if</span> fields</span>.</span>len</span>()</span> >=</span> 3</span> {</span></span> Some</span>(</span></span> fields[</span>..</span>3</span>]</span></span> .</span>iter</span>()</span></span> .</span>filter_map</span>(</span>\|</span>s</span>\|</span> f64</span>::</span>from_str</span>(s)</span>.</span>ok</span>())</span></span> .</span>collect</span>(),</span></span> )</span></span> }</span> else</span> {</span></span> pgrx</span>::</span>error!</span>(</span>"Invalid format in /proc/loadavg"</span>);</span></span> }</span></span> }</span></span></code></pre> To limit usage to systems supporting the proc filesystem</strong>, we check for the presence of the file when the extension is loaded:</p> #[pg_guard]</span></span> fn</span> _PG_init</span>() {</span></span> INIT</span>.</span>call_once</span>(</span>\|\|</span> {</span></span> let</span> loadavg_available</span> =</span> fs</span>::</span>metadata</span>(</span>"/proc/loadavg"</span>)</span>.</span>is_ok</span>();</span></span> if !</span>loadavg_available {</span></span> pgrx</span>::</span>error!</span>(</span>"/proc/loadavg not found. Extension cannot load."</span>);</span></span> }</span></span> });</span></span> }</span></span></code></pre> This basic check prevents users from loading the extension on incompatible systems. With minor details omitted, that's almost all there is to it.</p> You can find the complete lib.rs</code> file in the accompanying GitHub repository</a>.</p> Running the Extension</a> </h2> Where pgrx</code> truly shines is in all the heavy lifting it does for you. It leverages the cargo</code> command. To initialise the extension, run:</p> cargo</span> pgrx init</span></span></code></pre> This downloads and prepares PostgreSQL source code for versions 12 to 16 (at the time of writing). After a bit of waiting (hopefully successfully, if you have all the system dependencies), run the extension:</p> cargo</span> pgrx run</span></span></code></pre> You'll get a psql</code> prompt for a dedicated PostgreSQL instance. Create and use the extension:</p> CREATE</span> EXTENSION pg_sysload;</span></span></code></pre> Now you can try the newly exposed function:</p> SELECT</span> sys_loadavg();</span></span> sys_loadavg</span></span> ------------------</span></span> {</span>1</span>.</span>17</span>, </span>0</span>.</span>57</span>, </span>0</span>.</span>32</span>}</span></span> (</span>1</span> row</span>)</span></span></code></pre> And that's all. Your very first PostgreSQL extension is working.</p> Throttling Batch Processing</a> </h2> With the extension ready, let's revisit our original goal: throttling long-running batch data processing. The data from sys_loadavg</code> can be used in various ways:</p> Dynamic Sleep Times:</strong> Insert calculated sleep intervals based on the system load.</li> Dynamic Batch Sizes:</strong> Adjust the number of rows processed per batch based on available resources.</li> </ul> Here's an example:</p> -- Sleep for 5 seconds multiplied by the 1-minute load of the system</span></span> SELECT</span> pg_sleep((sys_loadavg())[1]</span> </span> 5</span>);</span></span> </span> -- Assume 20 cores and for each one available, add 100 rows to process</span></span> SELECT</span> ... </span>LIMIT</span> (</span>SELECT</span> (</span>20</span> -</span> (sys_loadavg())[1])::</span>int </span> 100</span>);</span></span></code></pre> This allows you to fine-tune processing for unsupervised operation, automatically adapting to the current system load.</p> Conclusion</a> </h2> And there you have it! You've just built your first PostgreSQL extension using Rust and the pgrx</code> framework. I have always found the prospect of writing an extension quite daunting (probably because I'm long gone from the C-ecosystem), but this wasn't so bad. You've now got a handy tool for monitoring system load, perfect for keeping tabs on how your database processes long-running migrations.</p> This is just scratching the surface. We haven't even touched on the really fun stuff – extending PostgreSQL's internals with custom data types, operators, or indexes. You could build extensions that transform or aggregate data, hook into external APIs, or create background workers, bringing various business logic directly into the database engine. And yes, tapping into the database's core can be a bit risky, and it is always important to assess risks and recovery options. But whatever you choose to create, remember, with pgrx</code> at your side, you've got the power of Rust to keep things safe and sound.</p> The Bloat Busters: pg_repack vs pg_squeeze 2024-04-27T00:00:00+00:00 As the database size increases and the number of transactions per second rise, you'll inevitably face the challenge of the table bloat. Although PostgreSQL assists as much as possible with its auto-vacuum feature</a>, there will come a time when you will compel whether to run VACUUM FULL</code>. Unless you have option of longish downtime windows, this is not an easy decision.</p> Thankfully, the rich ecosystem of PostgreSQL offers more than one solution how to make it simpler. Ignoring older tools like pg_reorg</code>, two contenders worth considering are pg_repack</strong> and pg_squeeze</strong>. This article dives into their strengths and weaknesses to help you decide which one is better for your specific use case.</p> Cases of Heavy Duty Maintenance</a> </h2> In scenarios with a continuous and predictable transaction pattern, you can usually rely on auto-vacuum. However, there are use cases where this won't be sufficient. Such examples typically involve "bulk" operations—whether it's bulk imports, deletions, or a combination of both. Imagine scenarios where:</p> You migrate one or more column data types</a> over a longer period (and no, using ALTER COLUMN name TYPE new_type</code> is not the best option).</li> You drop a column that has been moved to a different table.</li> You need to modify a large amount of data, using soft-deletes and later actual DELETEs</a>.</li> Due to changes in compliance requirements, you need to DELETE a massive amount of data.</li> </ul> In all these cases, you might end up with a huge table and the traditional solution would be VACUUM FULL</code>, which is associated with significant downtime due to table locking.</p> Alternatives to VACUUM FULL</a> </h2> Due to the limitations of VACUUM FULL</code>, several alternatives have emerged. The first contender in this space was pg_reorg</code> (which this comparison will not cover), later superseded by pg_repack</code>. The relatively new kid on the block is pg_squeeze</code>.</p> Each solution comes with different architectures, deployment methods, and sets of pros and cons.</p> As an added benefit, both tools can effectively serve as alternatives to CLUSTER</code> (which also requires the exclusive table lock similar to full vacuum), making them effective in optimising data storage based on the given order.</p> Both tools also share common behaviour. They won’t magically remove the bloat—you need to have at least the same amount of space available as the new table will require. Therefore, deploying both tools needs to be considered well before you reach critical levels of available disk space. Both will also generate a significant volume of WAL files — affecting every 8KB page</a> along the way — which need to be stored, processed, backed up, etc.</p> While not directly comparable to these tools, another method that can be used manually is table partitioning. Although it won’t be included in this comparison, if planned in advance, it can simplify certain maintenance tasks, improve performance, and make routine vacuuming faster and more efficient. It's only fair to say - it all depends on the specific scenarios and usage.</p> pg_repack</a> </h2> At the time of writing this article, pg_repack</code> can be considered the most well-known solution for combating table bloat in the PostgreSQL ecosystem. Built as an extension, it's easy to install (either from package repositories or via a self-compiled artifact) and its setup does not require a cluster restart.</p> pg_repack</code> works by creating a new copy of the table being processed, setting up triggers to replicate new data while it fills up with the existing data. An exclusive full table lock is required both at the start and finish of the process when swapping the old and new tables.</p> The maintenance is initiated from the CLI and allows you to specify a number of arguments to fine-tune the repacking of individual tables to match the requirements. pg_repack</code> allows re-clustering of data based on columns only, i.e., it does not explicitly require an index and can therefore overcome some limitations of pg_squeeze</code> in this regard.</p> pg_repack</code> is very effective in reclaiming space and can work with all types of bloat. It's available out of the box both on Amazon RDS and Google Cloud SQL.</p> The drawback you might experience when terminating the process (which may be necessary for various reasons, such as impact on the running environment) is that it won’t clean up all the fragments as it won't remain connected to the target database.</p> pg_squeeze</a> </h2> Compared to the previous solution, pg_squeeze</code> is built differently, relying on logical decoding instead of triggers. Its main benefit is a lower impact on the host system during table rebuilding, improving availability and stability.</p> Like pg_repack</code>, pg_squeeze</code> creates a new table and copies the existing data from the bloated table. Logical replication is involved in streaming changes from the original table to the newly created one in real-time. This allows the new table to stay up-to-date during the process without unnecessary impact on the regular operations performed on the bloated table. Thus, pg_squeeze</code> significantly reduces the need for locking. The exclusive lock is needed only during the final phase of the operation, when the old table is swapped out for the new, optimized table. The duration of the exclusive lock can also be configured.</p> While pg_squeeze</code> is also available as an extension, its deployment necessitates configuration changes involving wal_level</code>, max_replication_slots</code>, and shared_preload_libraries</code> — the same settings used for logical replication</a>. Due to this, a restart of the cluster is required.</p> On the other hand, pg_squeeze</code> is designed for regular, rather than ad-hoc processing only. You can register a table for regular processing, and whenever the table meets the criteria to be "squeezed," a task will be added to a queue, where it will be sequentially processed in the order they were created. The automated processing offers basic options usable in most scenarios, but you might find it limited if you need to work around other operational constraints of the cluster/environment. Having said that, the maintenance of the table can also be triggered manually.</p> While pg_squeeze</code> might be considered superior to pg_repack</code> in terms of maintenance operations and impact, it comes with a significant caveat when reclaiming space—compared to pg_repack</code>, it copies the full rows as they are. This behaviour renders it ineffective at removing bloat created due to dropped columns (behaviour still present at the time of the writing of this article at the end of April 2024).</p> As already mentioned, pg_squeeze</code> can use a specified index for clustering when needed. The limitation you might find is that clustering cannot be performed on a partial index. Compared to pg_repack</code>, it always seems to clean up all the artefacts accordingly (thanks to the always-running worker process).</p> Unfortunately at the moment of writing the article pg_squeeze</code> is not available for Amazon RDS, only Google Cloud SQL.</p> Conclusion</a> </h2> It's remarkable to have two mature and production-tested tools at your disposal. Deciding between them comes down to the specific requirements, the use cases, and the operational specifics of the business services.</p> The very opinionated difference between the tools can be made, using pg_squeeze</code> for automated, continuous cleaning of specific tables, and pg_repack</code> as the heavyweight champion of controlled setups.</p> (Edit) Added availability on Amazon RDS and Google Cloud SQL.</p>

boringSQL | Supercharge your SQL & PostgreSQL powers - expert

pg_regresql: truly portable PostgreSQL statistics

2026-03-21T14:32:00+00:00

The previous article</a> showed that PostgreSQL 18 makes optimizer statistics portable, but left one gap open:

It's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportionally. </blockquote>

The planner doesn't trust pg_class.relpages</code>. It calls smgrnblocks()</code> to read the actual number of 8KB pages from disk. Your table is 74 pages on disk but pg_class.relpages</code> says 123,513? The planner uses the ratio to scale reltuples</code> down to match the actual file size. The selectivity ratios stay correct, plan shapes mostly survive, but the absolute cost estimates are off.

For debugging a single query, that's usually fine. For automated regression testing where you compare EXPLAIN</code> costs across runs, it breaks things. A cost threshold of 2× means something different when the baseline was computed from fake-scaled numbers.

As part of my work on RegreSQL</a> I'm happy to announce pg_regresql</a> extension which fixes this by hooking directly into the planner.

Why the planner ignores relpages</a>
</h2>
When PostgreSQL's planner calls get_relation_info()</code> in plancat.c</code>, it delegates to estimate_rel_size()</code> which ends up in table_block_relation_estimate_size()</code> in tableam.c</code>. There, the actual page count comes from the storage manager:</p>
curpages </span>=</span> RelationGetNumberOfBlocks</span>(rel);</span></span></code></pre>
The function then computes a tuple density from pg_class</code> (reltuples / relpages</code>) and multiplies it by curpages</code> to estimate tuples. So pg_class.reltuples</code> isn't ignored, it's scaled to match the real file size. The reasoning is sound for normal operation: the catalog might be stale, but the file system is always current.</p>
The same applies to indexes. The planner reads their actual sizes from disk too.</p>
What pg_regresql does</a>
</h2>
The extension hooks into get_relation_info_hook</code>, a planner callback that runs after PostgreSQL reads the physical file stats. The hook replaces the file-based numbers with the values stored in pg_class</code>:</p>

rel->pages</code> ← pg_class.relpages</code></li>
rel->tuples</code> ← pg_class.reltuples</code></li>
rel->allvisfrac</code> ← pg_class.relallvisible / pg_class.relpages</code></li>
</ul>
It does the same for every index in rel->indexlist</code>. Pages and tuples for each index are overridden from the index's own pg_class</code> entry.</p>
The guard conditions are simple: skip the override if relpages == 0</code> (empty or never analyzed) or reltuples == -1</code> (never analyzed). The hook only activates for tables that have been ANALYZE</code>d or had statistics injected.</p>
Installation</a>
</h2>
Build from source using PGXS:</p>
cd</span> pg_ext</span></span>
make</span></span>
make</span> install</span></span></code></pre>
It requires no GUCs, background workers, or shared memory.</p>
Usage</a>
</h2>
Load the extension in your session:</p>
LOAD</span> 'pg_regresql'</span>;</span></span></code></pre>
That's it. Every EXPLAIN</code> in this session will now use catalog statistics instead of file sizes. There are no functions to call, no tables to configure.</p>
You can also load it per-database by adding it to session_preload_libraries</code> in postgresql.conf</code> or via ALTER DATABASE</code>:</p>
ALTER DATABASE</span> test_db </span>SET</span> session_preload_libraries </span>=</span> 'pg_regresql'</span>;</span></span></code></pre>The difference it makes</a>
</h2>
Using the same test_orders</code> example from the previous article</a>: 10,000 actual rows, injected with production statistics claiming 50 million rows across 123,513 pages.</p>
Without pg_regresql</code>:</p>
EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre>                                             QUERY PLAN</span></span>
----------------------------------------------------------------------------------------------------</span></span>
 Index Scan using test_orders_created_at_idx on test_orders  (cost=0.29..153.21 rows=6340 width=26)</span></span>
   Index Cond: (created_at > '2024-06-01'::date)</span></span></code></pre>
The plan shape is correct (index scan thanks to the histogram), but the row estimate is 6,340. For a 50-million-row table where the filter covers roughly 10% of the histogram range, the expected estimate should be in the millions. The planner saw 74 real pages on disk, scaled reltuples</code> down to ~30,000, then applied selectivity. The ratio is preserved but the absolute number is wrong.</p>
With pg_regresql</code>:</p>
LOAD</span> 'pg_regresql'</span>;</span></span>
EXPLAIN </span>SELECT * FROM</span> test_orders </span>WHERE</span> created_at </span>></span> '2024-06-01'</span>;</span></span></code></pre>                                                QUERY PLAN</span></span>
-----------------------------------------------------------------------------------------------------------</span></span>
 Index Scan using test_orders_created_at_idx on test_orders  (cost=0.29..153212.27 rows=10791836 width=27)</span></span>
   Index Cond: (created_at > '2024-06-01'::date)</span></span>
(2 rows)</span></span></code></pre>
Cost numbers now reflect the full 50 million rows.</p>
Where this matters</a>
</h2>
Cost-based regression testing.</strong> If you're comparing EXPLAIN</code> costs between schema versions (which is what RegreSQL</a> does), you need the absolute numbers to be stable and realistic. With the scaling behavior, your baseline costs are proportional to your test database size, not production. A migration that doubles a cost in production might show a 1.3× increase in CI because the scaled-down numbers compress the range.</p>
Reproducing production plans on a laptop.</strong> Sometimes the plan shape itself changes depending on the absolute numbers. A query with multiple joins might get a different join order when the planner sees 50 million rows vs. 30,000 rows, because the cost crossover between hash join and nested loop depends on the absolute row count, not just the ratio.</p>
Index-only scans.</strong> The allvisfrac</code> (fraction of all-visible pages) matters for index-only scan costing. Without the hook, allvisfrac</code> is computed from the real relallvisible</code> catalog value divided by the real page count. With injected stats, relallvisible</code> might be 120,000 but the real page count is 74, so the fraction clamps to 1.0 and the planner overestimates how cheap index-only scans are. The hook fixes this by using the injected relpages</code> as the denominator.</p>
What it doesn't do</a>
</h2>
Column-level statistics and ANALYZE</code> behavior are unchanged. The extension only affects how the planner reads table and index sizes. One thing worth noting: EXPLAIN ANALYZE</code> will still show actual row counts from the real (small) data. The extension changes the planner's cost estimates, not query execution.</p>
The full workflow</a>
</h2>
Combining PostgreSQL 18's portable statistics with pg_regresql</code>, the full workflow looks like this:</p>
# 1. dump schema and statistics from production</span></span>
pg_dump</span> --schema-only -d</span> production_db</span> ></span> schema.sql</span></span>
pg_dump</span> --statistics-only -d</span> production_db</span> ></span> stats.sql</span></span>
</span>
# 2. create test database</span></span>
createdb</span> test_db</span></span>
psql</span> -d</span> test_db</span> -f</span> schema.sql</span></span>
</span>
# 3. load minimal fixture data (optional)</span></span>
psql</span> -d</span> test_db</span> -f</span> fixtures.sql</span></span>
</span>
# 4. inject production statistics</span></span>
psql</span> -d</span> test_db</span> -f</span> stats.sql</span></span>
</span>
# 5. install pg_regresql and prevent stats from being overwritten</span></span>
psql</span> -d</span> test_db</span> <<</span>SQL</span></span>
ALTER DATABASE test_db SET session_preload_libraries = 'pg_regresql';</span></span>
ALTER TABLE orders SET (autovacuum_enabled = false);</span></span>
-- repeat for other tables</span></span>
SQL</span></span>
</span>
# 6. reconnect and verify (plans now match production)</span></span>
psql</span> -d</span> test_db</span> -c</span> "EXPLAIN SELECT * FROM orders WHERE status = 'pending'"</span></span></code></pre>Compatibility</a>
</h2>
The extension works with PostgreSQL 13 through 18. The portable statistics functions (pg_restore_relation_stats</code>, pg_restore_attribute_stats</code>) require PostgreSQL 18, but pg_regresql</code> works with any method of writing to pg_class</code>, including direct catalog updates on older versions.</p>
PostgreSQL 19 will need a small update: the get_relation_info_hook</code> used by pg_regresql has been replaced</a> with build_simple_rel_hook</code>. The new hook runs slightly later with different arguments, but the override logic stays the same.</p>

Don't use this in production.</strong> The extension makes the planner ignore reality. That's exactly what you want for testing with injected statistics. In production, you want the planner to see the actual file sizes so it can adapt to data growth, bloat, and vacuum activity. Keep pg_regresql</code> in your dev/test and CI databases.
</div>

Production query plans without production data 2026-03-08T22:05:00+00:00 In the previous article</a> we covered how the PostgreSQL planner reads pg_class</code> and pg_statistic</code> to estimate row counts, choose join strategies, and decide whether an index scan is worth it. The message was clear: when statistics are wrong, everything else goes with it. Streaming replication provides bit-to-bit replication, so all replicas share the same statistics with primary server.</div> But there was one thing we didn't talk about. Statistics are specific to the database cluster that generated them. The primary way to populate them is `ANALYZE` which requires the actual data. PostgreSQL 18 changed that. Two new functions: pg_restore_relation_stats</code> and pg_restore_attribute_stats</code> write numbers directly into the catalog tables. Combined with pg_dump --statistics-only</code>, you can treat optimizer statistics as a deployable artifact. Compact, portable, plain SQL. The feature was driven by the upgrade use case</a>. In the past, major version upgrades used to leave pg_statistic</code> empty, forcing you to run ANALYZE</code>. Which might take hours on large clusters. With PostgreSQL 18 upgrades now transfer statistics automatically. But that's just the beginning. The same logic lets you export statistics from production and inject them anywhere - test database, local debugging, or as part of CI pipelines. The problem</a> </h2> Your CI database has 1,000 rows. Production has 50 million. The planner makes completely different decisions for each. Running EXPLAIN</code> in CI tells you nothing about the production plan. This is the core premise behind RegreSQL</a>. Catching query plan regressions in CI is far more reliable when the planner sees production-scale statistics. Same applies to debugging. A query is slow in production and you want to reproduce the plan locally, but your database has different statistics, and planner chooses the predictable path. Porting production stats can provide you that snapshot of thinking planner has to do in production, without actually going to production. pg_restore_relation_stats</a> </h2> The first of function behind portable PostgreSQL statistics is pg_restore_relation_stats</code>. It writes table-level data directly into pg_class</code> in form of variadic name/value pairs. SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer, 'relallfrozen', 120000::integer );</code></pre> But that's just an example. Let's modify some real statistics to see the full value. We will create a small table, inject fake production-like statistics and watch the planner to change its mind. CREATE TABLE test_orders ( id integer GENERATED ALWAYS AS IDENTITY PRIMARY KEY, customer_id integer NOT NULL, amount numeric(10,2) NOT NULL, status text NOT NULL DEFAULT 'pending', created_at date NOT NULL DEFAULT CURRENT_DATE ); INSERT INTO test_orders (customer_id, amount, status, created_at) SELECT (random() * 9999 + 1)::int, (random() * 5000 + 5)::numeric(10,2), (ARRAY['pending','shipped','delivered','cancelled'])[floor(random()*4+1)::int], '2024-01-01'::date + (random() * 365)::int FROM generate_series(1, 10000); CREATE INDEX ON test_orders (created_at); CREATE INDEX ON test_orders (status); ANALYZE test_orders;</code></pre> When you check the current statistics, it has predictable data. SELECT relname, relpages, reltuples FROM pg_class WHERE relname = 'test_orders';</code></pre> relname | relpages | reltuples -------------+----------+----------- test_orders | 74 | 10000 (1 row)</code></pre> With 10,000 rows across 74 pages, the planner picks a sequential scan. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ----------------------------------------------------------------- Seq Scan on test_orders (cost=0.00..199.00 rows=5891 width=26) Filter: (created_at > '2024-06-01'::date) (2 rows)</code></pre> Now inject production-scale table stats: SELECT pg_restore_relation_stats( 'schemaname', 'public', 'relname', 'test_orders', 'relpages', 123513::integer, 'reltuples', 50000000::real, 'relallvisible', 123513::integer );</code></pre> And you might be surprised by the result. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=17649 width=26) Filter: (created_at > '2024-06-01'::date)</code></pre> The planner is still using the sequential plan. Only the estimated number of rows has changed. Why? If you remember from previous article, it's where column level statistics come into play. Histogram bounds still match the original 10,000 rows we inserted. pg_restore_attribute_stats</a> </h2> This function writes column-level statistics into pg_statistic</code> the same catalog that ANALYZE populates</a> with MCVs, histograms, and correlation</a>. In previous section, we left the planner stuck on a sequential scan despite believing the table has 50 million rows. The missing piece is column-level statistics. Let's pick up where we left off and inject histogram bounds for created_at</code>. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'created_at', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 4::integer, 'n_distinct', -0.05::real, 'histogram_bounds', '{2019-01-01,2019-07-01,2020-01-01,2020-07-01,2021-01-01,2021-07-01,2022-01-01,2022-07-01,2023-01-01,2023-07-01,2024-01-01}'::text, 'correlation', 0.98::real );</code></pre> Now the planner knows the data spans 5 years. A query filtering on the last 6 months of 2024 covers a narrow slice. EXPLAIN SELECT * FROM test_orders WHERE created_at > '2024-06-01';</code></pre> QUERY PLAN ---------------------------------------------------------------------------------------------------- Index Scan using test_orders_created_at_idx on test_orders (cost=0.29..153.21 rows=6340 width=26) Index Cond: (created_at > '2024-06-01'::date)</code></pre> Histogram bounds divide the non-MCV portion of the data into equal-population buckets. If most_common_vals</code> accounts for most of the data, the histogram covers only the remaining tail. The number of buckets is controlled by default_statistics_target</code> (default 100, meaning 101 bounds). </div> And that's a plan flip! The histogram tells the planner the data spans 2019–2024, so > '2024-06-01'</code> matches a narrow tail. A small fraction of 50 million rows. The index scan that was ignored before is now the obvious choice. Table-level stats set the scale, column-level stats shaped the selectivity, and together they changed the plan. The correlation</code> statistic tells the planner how closely the physical row order matches the column's sort order. A value near 1.0 means sequential access patterns - making index scan cheaper</a> because the next row is likely on the same or adjacent page. For time-series data like created_at</code> where rows are inserted chronologically, correlation is typically very high. </div> Injecting a skewed distribution</a> </h2> The same function handles MCV lists</a>. In production, your status</code> column isn't uniform, 95% of orders are delivered, 1.5% are pending. SELECT pg_restore_attribute_stats( 'schemaname', 'public', 'relname', 'test_orders', 'attname', 'status', 'inherited', false::boolean, 'null_frac', 0.0::real, 'avg_width', 9::integer, 'n_distinct', 5::real, 'most_common_vals', '{delivered,shipped,cancelled,pending,returned}'::text, 'most_common_freqs', '{0.95,0.015,0.015,0.015,0.005}'::real[] );</code></pre> You can see EXPLAIN SELECT * FROM test_orders WHERE status = 'pending';</code></pre> QUERY PLAN --------------------------------------------------------------------------------------- Bitmap Heap Scan on test_orders (cost=8.93..90.42 rows=599 width=27) Recheck Cond: (status = 'pending'::text) -> Bitmap Index Scan on test_orders_status_idx (cost=0.00..8.78 rows=599 width=0) Index Cond: (status = 'pending'::text) (4 rows)</code></pre> and compare it with EXPLAIN SELECT * FROM test_orders WHERE status = 'delivered';</code></pre> QUERY PLAN ------------------------------------------------------------------ Seq Scan on test_orders (cost=0.00..448.45 rows=28458 width=27) Filter: (status = 'delivered'::text) (2 rows)</code></pre> Same column, same operator, different plans. The planner uses a bitmap index scan for pending</code> (1.5% rare enough to justify the index) and a sequential scan for delivered</code> (95% being most of the table). The selectivity ratios from the MCV list drive the plan choice. You might have noticed the row estimates (599 and 28,458) are lower than you'd expect for a 50-million-row table. The planner checks the actual physical file size. Our table is only 74 pages on disk, not the 123,513 we injected. Hence the planner scales reltuples</code> and relpages</code> down proportionally. The absolute numbers shrink, but the ratios between them stay correct, and it's the ratios that determine plan shape. When you use pg_dump --statistics-only</code> in practice, you're typically restoring into a database with comparable data volume, so the estimates align naturally. </div> </div> </div> </div> </div> pg_regresql extension The pg_regresql</code> extension fixes this scaling problem. It hooks into the planner to trust the injected relpages</code> value instead of reading the physical file size, so cost estimates match production even when your test database is tiny. Read more</a> </div> </div> pg_dump</a> </h2> The functions we covered are the mechanics. For operational use pg_dump</code> provides everything you need. PostgreSQL 18 added three flags. Flag</th> Effect</th></tr></thead> --statistics</code></td> dump the statistics (you have to request it explicitely)</td></tr> --statistics-only</code></td> dump only the statistics, not schema or data</td></tr> --no-statistics</code></td> do not dump statistics</td></tr> </tbody></table> When you export the statistics for your production database pg_dump --statistics-only -d production_db > stats.sql </code></pre> you will see the output is series of SELECT pg_restore_relation_stats(...)</code> and SELECT pg_restore_attribute_stats(...)</code> calls. Exactly as we explained above. The full workflow to turn your production data into testable plans might look like this: # 1. dump schema from production pg_dump --schema-only -d production_db > schema.sql # 2. dump statistics from production pg_dump --statistics-only -d production_db > stats.sql # 3. create test database with schema createdb test_db psql -d test_db -f schema.sql # 4. load fixture data (optional; masked, minimal) psql -d test_db -f fixtures.sql # 5. inject production statistics psql -d test_db -f stats.sql # 6. query plans now match production psql -d test_db -c "EXPLAIN SELECT * FROM test_orders WHERE status = 'pending'"</code></pre> Statistics dumps are tiny. A database with hundreds of tables and thousands of columns produces a statistics dump under 1MB. The production data might be hundreds of GB. The statistics that describe it fit in a text file. </div> Keeping injected statistics alive</a> </h2> Now you might ask yourself, where's the catch? And there's a big one, the autovacuum will eventually kick in and run ANALYZE</code>. Which will overwrite your injected statistics with real numbers and you are back where you started. To prevent this, disable autovacuum analyze on the tables you've injected. -- disable autovacuum ALTER TABLE test_orders SET (autovacuum_enabled = false); -- or set analyze threshold so high it nevers kicks-in ALTER TABLE test_orders SET (autovacuum_analyze_threshold = 2147483647);</code></pre> Be careful here. If you're also writing data to these tables in dev: running migrations, loading fixtures, testing inserts, the injected statistics will drift further from reality with every write. The planner will plan based on a production distribution that no longer reflects the local data. For read-only query plan testing this is exactly what you want. For integration tests that modify data, you may need to re-inject statistics after each test run. And please, never ever do this in production! </div> What's not covered?</a> </h2> As we have seen earlier, it's not worth trying to inject relpages</code> as the planner checks the actual file size and scales it proportationally. This limits the number of absolute rows planner might estimate. I.e. to get comparable numbers to production environment you still would have to create comparable data volume (which isn't a problem when talking about the primary use case of this feature - restoring backups). It's also worth to note that CREATE STATISTICS</code> used for multivariate correlations, distinct counts across column groups and MCV lists for column combinations</a> are not covered within PostgreSQL 18. Those still require ANALYZE</code> after restore. PostgreSQL 19 will close this gap with pg_restore_extended_stats()</code>. Security</a> </h2> The restore functions require the MAINTAIN</code> privilege on the target table. This is the same privilege needed for ANALYZE</code>, VACUUM</code>, REINDEX</code>, and CLUSTER</code> as it was introduced in PostgreSQL 17</a>. The easiest way to grant it for automation: GRANT pg_maintain TO ci_service_account;</code></pre> This grants MAINTAIN</code> on all tables in the database. Enough for a CI pipeline to inject statistics without needing superuser. Instant database clones with PostgreSQL 18 2025-12-22T23:53:16+00:00 Have you ever watched a long running migration script</a>, wondering if it's about to wreck your data? Or wish you can "just" spin a fresh copy of database for each test run? Or wanted to have reproducible snapshots to reset between runs of your test suite, (and yes, because you are reading boringSQL) needed to reset the learning environment? When your database is a few megabytes, pg_dump</code> and restore works fine. But what happens when you're dealing with hundreds of megabytes/gigabytes - or more? Suddenly "just make a copy" becomes a burden. You've probably noticed that PostgreSQL connects to template1</code> by default. What you might have missed is that there's a whole templating system hiding in plain sight. Every time you run CREATE DATABASE dbname;</code></pre> PostgreSQL quietly clones standard system database template1</code> behind the scenes. Making it same as if you would use CREATE DATABASE dbname TEMPLATE template1;</code></pre> The real power comes from the fact that you can replace template1</code> with any database. You can find more at Template Database documentation</a>. In this article, we will cover a few tweaks that turn this templating system into an instant, zero-copy database cloning machine. CREATE DATABASE ... STRATEGY</a> </h2> Before PostgreSQL 15, when you created a new database from a template, it operated strictly on the file level. This was effective, but to make it reliable, Postgres had to flush all pending operations to disk (using CHECKPOINT</code>) before taking a consistent snapshot. This created a massive I/O spike - a "Checkpoint Storm" - that could stall your production traffic. Version 15 of PostgreSQL introduced new parameter CREATE DATABASE ... STRATEGY = [strategy]</code> and at the same time changed the default behaviour how the new databases are created from templates. The new default become WAL_LOG</code> which copies block-by-block via the Write-Ahead Log (WAL), making I/O sequential (and much smoother) — operations that also show up in EXPLAIN buffer statistics</a> — and support for concurrency without facing latency spike. This prevented the need to CHECKPOINT but made the database cloning operation potentially significantly slower. For an empty template1</code>, you won't notice the difference. But if you try to clone a 500GB database using WAL_LOG, you are going to be waiting a long time. The STRATEGY</code> parameter allows us to switch back to the original method FILE_COPY</code> to keep the behaviour, and speed. And since PostgreSQL 18, this opens the whole new set of options. FILE_COPY</a> </h2> Because the FILE_COPY</code> strategy is a proxy to operating system file operations, we can change how the OS handles those files. When using standard file system (like ext4</code>), PostgreSQL reads every byte of the source file and writes it to a new location. It's a physical copy. However starting with PostgreSQL 18 - file_copy_method</code> gives you options to switch that logic; while default option remains copy</code>. With modern filesystems (like ZFS, XFS with reflinks, APFS, etc.) you can switch it to clone</code> and leverage CLONE</code> (FICLONE</code> on Linux) operation for almost instant operation. And it won't take any additional space. Quick setup checklist: Linux with XFS or ZFS, macOS with APFS, or FreeBSD with ZFS</li> PostgreSQL 18+ cluster on that filesystem</li> Set file_copy_method = clone</code> in your config</li> Reload configuration</li> </ul> </div> The benchmark</a> </h2> We need some dummy data to copy. This is the only part of the tutorial where you have to wait. Let's generate a ~6GB database. CREATE DATABASE source_db; \c source_db CREATE TABLE boring_data ( id serial PRIMARY KEY, payload text ); -- generate 50m rows INSERT INTO boring_data (payload) SELECT md5(random()::text) || md5(random()::text) FROM generate_series(1, 50000000); -- force a checkpoint CHECKPOINT;</code></pre> You can verify the database now has roughly 6GB of data. Name | source_db Owner | postgres Encoding | UTF8 Locale Provider | libc Collate | en_US.UTF-8 Ctype | en_US.UTF-8 Locale | ICU Rules | Access privileges | Size | 6289 MB Tablespace | pg_default Description |</code></pre> While enabling \timing</code> you can test the default (WAL_LOG) strategy. And on my test volume (relatively slow storage) I get CREATE DATABASE slow_copy TEMPLATE source_db; CREATE DATABASE Time: 67000.615 ms (01:07.001)</code></pre> Now, let's verify our configuration is set for speed: show file_copy_method; file_copy_method ------------------ clone (1 row)</code></pre> Let's request the semi-instant clone of the same database, without taking extra disk space at the same time. CREATE DATABASE fast_clone TEMPLATE source_db STRATEGY=FILE_COPY; CREATE DATABASE Time: 212.053 ms</code></pre> That's a quite an improvement, isn't it? Working with cloned data</a> </h2> That was the simple part. But what is happening behind the scenes? When you clone a database with file_copy_method = clone</code>, PostgreSQL doesn't duplicate any data. The filesystem creates new metadata entries that point to the same physical 8KB pages</a>. Both databases share identical storage. This can create some initial confusion. If you ask PostgreSQL for the size: SELECT pg_database_size('source_db') as source, pg_database_size('fast_clone') as clone;</code></pre> PostgreSQL reports both as ~6GB because that's the logical size - how much data each database "contains" - i.e. logical size. -[ RECORD 1 ]------ source | 6594041535 clone | 6594041535</code></pre> The interesting part happens when you start writing. PostgreSQL doesn't update tuples in place. When you UPDATE a row, it writes a new tuple version somewhere (often a different page entirely) and marks the old one as dead. The filesystem doesn't care about PostgreSQL internals - it just sees writes to 8KB pages</a>. Any write to a shared page triggers a copy of that entire page. A single UPDATE will therefore trigger copy-on-write on multiple pages: the page holding the old tuple</li> the page receiving the new tuple</li> index pages if any indexed columns changed</li> FSM and visibility map pages as PostgreSQL tracks free space</li> </ul> And later, VACUUM touches even more pages while cleaning up dead tuples</a>. In this case diverging quickly from the linked storage. XFS proof</a> </h2> Using the database OID and relfilenode we can verify the both databases are now sharing physical blocks. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 2031: 10471550.. 10473581: 2032: shared 1: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 2: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 3: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 4: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 5: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 6: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 7 extents found</code></pre> All it takes is to update some rows using update boring_data set payload = 'new value' || id where id IN (select id from boring_data limit 20);</code></pre> and the situation will start to change. root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16402/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16402/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10471550.. 10471589: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16402/16404: 7 extents found root@clone-demo:/var/lib/postgresql# sudo filefrag -v /var/lib/postgresql/18/main/base/16418/16404 Filesystem type is: 58465342 File size of /var/lib/postgresql/18/main/base/16418/16404 is 1073741824 (262144 blocks of 4096 bytes) ext: logical_offset: physical_offset: length: expected: flags: 0: 0.. 39: 10297326.. 10297365: 40: 1: 40.. 2031: 10471590.. 10473581: 1992: 10297366: shared 2: 2032.. 16367: 10474098.. 10488433: 14336: 10473582: shared 3: 16368.. 32751: 10497006.. 10513389: 16384: 10488434: shared 4: 32752.. 65519: 10522066.. 10554833: 32768: 10513390: shared 5: 65520.. 129695: 10571218.. 10635393: 64176: 10554834: shared 6: 129696.. 195231: 10635426.. 10700961: 65536: 10635394: shared 7: 195232.. 262143: 10733730.. 10800641: 66912: 10700962: last,shared,eof /var/lib/postgresql/18/main/base/16418/16404: 8 extents found root@clone-demo:/var/lib/postgresql#</code></pre> In this case extent 0 no longer has shared flag, first 40 blocks size (with default size 4KB) now diverge, making it total of 160KB. Each database now has its own copy at different physical address. The remaining extents are still shared. Things to be aware of</a> </h2> Cloning is tempting but there's one serious limitation you need to be aware if you ever attempt to do it in production. The source database can't have any active connections during cloning. This is a PostgreSQL limitation, not a filesystem one. For production use, this usually means you create a dedicated template database rather than cloning your live database directly. Or given the relatively short time the operation takes you have to schedule the cloning in times where you can temporary block/terminate all connections. Other limitation is that the cloning only works within a single filesystem. If your databases spans multiple table spaces on different mount points, cloning will fall back to regular physical copy. Finally, in most managed cloud environments (AWS RDS, Google Cloud SQL), you will not have access to the underlying filesystem to configure this. You are stuck with their proprietary (and often billed) functionality. But for your own VMs or bare metal? Go ahead and try it. Custom PostgreSQL extensions with Rust 2024-05-24T00:00:00+00:00 This article explores the pgrx framework, which simplifies the creation of custom PostgreSQL extensions to bring more logic closer to your database. Traditionally, writing such extensions required familiarity with C and a deep understanding of PostgreSQL internals, which could be quite challenging. pgrx</code> lowers the barrier and allows developers to use Rust, known for its safety and performance, making the process of creating efficient and safe database extensions much more accessible. pg_sysload</a> </h2> When working with large datasets and migrations (as discussed in How Not to Change PostgreSQL Column Type</a>), or during resource-intensive maintenance tasks, you'll want to optimise speed and minimise disruption to other processes. One way to control the pace of batch operations is to consider the load on the underlying system. Many Unix-based systems (we will focus on Linux) provide a valuable metric called the system load average. This average consists of three values: the 1-minute, 5-minute, and 15-minute load averages. The load average is not normalised for the number of CPU cores, so a load average of 1 on a single-core system means full utilisation, while on a quad-core system, it indicates 25% utilisation. In many cases, the system load average is also an excellent indicator of how ongoing operations are impacting a busy database cluster. In this article, we will create a PostgreSQL extension with a function called sys_loadavg()</code> that retrieves this load information. We will use the /proc/loadavg</code> file (part of the proc filesystem), which exposes underlying system details. Getting Started with pgrx</code></a> </h2> Before we start, ensure you have: Rust installed (Get Started with Rust</a>)</li> System dependencies</a> installed for pgrx</code></li> </ul> With these prerequisites in place, you can install pgrx</code> itself and create a new extension skeleton: cargo install --locked cargo-pgrx cargo pgrx new pg_sysload cd pg_sysload</code></pre> This gives you a complete environment for developing your own PostgreSQL extensions in Rust. While Rust might seem daunting at first (especially with its async and other features), the language itself is quite powerful. Its ease of extension creation makes it worth a second look. Setting Up the Extension</a> </h2> While the pgrx new</code> command sets up a basic template, we will create a more sophisticated extension. Its primary logic involves parsing the /proc/loadavg</code> file, extracting its values, and returning them to the user. The logic itself is straightforward: #[pg_extern] fn sys_loadavg() -> Option<Vec<f64>> { // Read the contents of the /proc/loadavg let mut file = match fs::File::open("/proc/loadavg") { Ok(file) => file, Err(err) => { pgrx::error!("Error reading /proc/loadavg: {}", err); } }; let mut contents = String::new(); if let Err(err) = file.read_to_string(&mut contents) { pgrx::error!("Error reading /proc/loadavg: {}", err); } // Extract the load average fields let fields = contents.split_whitespace().collect::<Vec<_>>(); if fields.len() >= 3 { Some( fields[..3] .iter() .filter_map(|s| f64::from_str(s).ok()) .collect(), ) } else { pgrx::error!("Invalid format in /proc/loadavg"); } }</code></pre> To limit usage to systems supporting the proc filesystem, we check for the presence of the file when the extension is loaded: #[pg_guard] fn _PG_init() { INIT.call_once(|| { let loadavg_available = fs::metadata("/proc/loadavg").is_ok(); if !loadavg_available { pgrx::error!("/proc/loadavg not found. Extension cannot load."); } }); }</code></pre> This basic check prevents users from loading the extension on incompatible systems. With minor details omitted, that's almost all there is to it. You can find the complete lib.rs</code> file in the accompanying GitHub repository</a>. Running the Extension</a> </h2> Where pgrx</code> truly shines is in all the heavy lifting it does for you. It leverages the cargo</code> command. To initialise the extension, run: cargo pgrx init</code></pre> This downloads and prepares PostgreSQL source code for versions 12 to 16 (at the time of writing). After a bit of waiting (hopefully successfully, if you have all the system dependencies), run the extension: cargo pgrx run</code></pre> You'll get a psql</code> prompt for a dedicated PostgreSQL instance. Create and use the extension: CREATE EXTENSION pg_sysload;</code></pre> Now you can try the newly exposed function: SELECT sys_loadavg(); sys_loadavg ------------------ {1.17, 0.57, 0.32} (1 row)</code></pre> And that's all. Your very first PostgreSQL extension is working. Throttling Batch Processing</a> </h2> With the extension ready, let's revisit our original goal: throttling long-running batch data processing. The data from sys_loadavg</code> can be used in various ways: Dynamic Sleep Times: Insert calculated sleep intervals based on the system load.</li> Dynamic Batch Sizes: Adjust the number of rows processed per batch based on available resources.</li> </ul> Here's an example: -- Sleep for 5 seconds multiplied by the 1-minute load of the system SELECT pg_sleep((sys_loadavg())[1] * 5); -- Assume 20 cores and for each one available, add 100 rows to process SELECT ... LIMIT (SELECT (20 - (sys_loadavg())[1])::int * 100);</code></pre> This allows you to fine-tune processing for unsupervised operation, automatically adapting to the current system load. Conclusion</a> </h2> And there you have it! You've just built your first PostgreSQL extension using Rust and the pgrx</code> framework. I have always found the prospect of writing an extension quite daunting (probably because I'm long gone from the C-ecosystem), but this wasn't so bad. You've now got a handy tool for monitoring system load, perfect for keeping tabs on how your database processes long-running migrations. This is just scratching the surface. We haven't even touched on the really fun stuff – extending PostgreSQL's internals with custom data types, operators, or indexes. You could build extensions that transform or aggregate data, hook into external APIs, or create background workers, bringing various business logic directly into the database engine. And yes, tapping into the database's core can be a bit risky, and it is always important to assess risks and recovery options. But whatever you choose to create, remember, with pgrx</code> at your side, you've got the power of Rust to keep things safe and sound. The Bloat Busters: pg_repack vs pg_squeeze 2024-04-27T00:00:00+00:00 As the database size increases and the number of transactions per second rise, you'll inevitably face the challenge of the table bloat. Although PostgreSQL assists as much as possible with its auto-vacuum feature</a>, there will come a time when you will compel whether to run VACUUM FULL</code>. Unless you have option of longish downtime windows, this is not an easy decision. Thankfully, the rich ecosystem of PostgreSQL offers more than one solution how to make it simpler. Ignoring older tools like pg_reorg</code>, two contenders worth considering are pg_repack and pg_squeeze. This article dives into their strengths and weaknesses to help you decide which one is better for your specific use case. Cases of Heavy Duty Maintenance</a> </h2> In scenarios with a continuous and predictable transaction pattern, you can usually rely on auto-vacuum. However, there are use cases where this won't be sufficient. Such examples typically involve "bulk" operations—whether it's bulk imports, deletions, or a combination of both. Imagine scenarios where: You migrate one or more column data types</a> over a longer period (and no, using ALTER COLUMN name TYPE new_type</code> is not the best option).</li> You drop a column that has been moved to a different table.</li> You need to modify a large amount of data, using soft-deletes and later actual DELETEs</a>.</li> Due to changes in compliance requirements, you need to DELETE a massive amount of data.</li> </ul> In all these cases, you might end up with a huge table and the traditional solution would be VACUUM FULL</code>, which is associated with significant downtime due to table locking. Alternatives to VACUUM FULL</a> </h2> Due to the limitations of VACUUM FULL</code>, several alternatives have emerged. The first contender in this space was pg_reorg</code> (which this comparison will not cover), later superseded by pg_repack</code>. The relatively new kid on the block is pg_squeeze</code>. Each solution comes with different architectures, deployment methods, and sets of pros and cons. As an added benefit, both tools can effectively serve as alternatives to CLUSTER</code> (which also requires the exclusive table lock similar to full vacuum), making them effective in optimising data storage based on the given order. Both tools also share common behaviour. They won’t magically remove the bloat—you need to have at least the same amount of space available as the new table will require. Therefore, deploying both tools needs to be considered well before you reach critical levels of available disk space. Both will also generate a significant volume of WAL files — affecting every 8KB page</a> along the way — which need to be stored, processed, backed up, etc. While not directly comparable to these tools, another method that can be used manually is table partitioning. Although it won’t be included in this comparison, if planned in advance, it can simplify certain maintenance tasks, improve performance, and make routine vacuuming faster and more efficient. It's only fair to say - it all depends on the specific scenarios and usage. pg_repack</a> </h2> At the time of writing this article, pg_repack</code> can be considered the most well-known solution for combating table bloat in the PostgreSQL ecosystem. Built as an extension, it's easy to install (either from package repositories or via a self-compiled artifact) and its setup does not require a cluster restart. pg_repack</code> works by creating a new copy of the table being processed, setting up triggers to replicate new data while it fills up with the existing data. An exclusive full table lock is required both at the start and finish of the process when swapping the old and new tables. The maintenance is initiated from the CLI and allows you to specify a number of arguments to fine-tune the repacking of individual tables to match the requirements. pg_repack</code> allows re-clustering of data based on columns only, i.e., it does not explicitly require an index and can therefore overcome some limitations of pg_squeeze</code> in this regard. pg_repack</code> is very effective in reclaiming space and can work with all types of bloat. It's available out of the box both on Amazon RDS and Google Cloud SQL. The drawback you might experience when terminating the process (which may be necessary for various reasons, such as impact on the running environment) is that it won’t clean up all the fragments as it won't remain connected to the target database. pg_squeeze</a> </h2> Compared to the previous solution, pg_squeeze</code> is built differently, relying on logical decoding instead of triggers. Its main benefit is a lower impact on the host system during table rebuilding, improving availability and stability. Like pg_repack</code>, pg_squeeze</code> creates a new table and copies the existing data from the bloated table. Logical replication is involved in streaming changes from the original table to the newly created one in real-time. This allows the new table to stay up-to-date during the process without unnecessary impact on the regular operations performed on the bloated table. Thus, pg_squeeze</code> significantly reduces the need for locking. The exclusive lock is needed only during the final phase of the operation, when the old table is swapped out for the new, optimized table. The duration of the exclusive lock can also be configured. While pg_squeeze</code> is also available as an extension, its deployment necessitates configuration changes involving wal_level</code>, max_replication_slots</code>, and shared_preload_libraries</code> — the same settings used for logical replication</a>. Due to this, a restart of the cluster is required. On the other hand, pg_squeeze</code> is designed for regular, rather than ad-hoc processing only. You can register a table for regular processing, and whenever the table meets the criteria to be "squeezed," a task will be added to a queue, where it will be sequentially processed in the order they were created. The automated processing offers basic options usable in most scenarios, but you might find it limited if you need to work around other operational constraints of the cluster/environment. Having said that, the maintenance of the table can also be triggered manually. While pg_squeeze</code> might be considered superior to pg_repack</code> in terms of maintenance operations and impact, it comes with a significant caveat when reclaiming space—compared to pg_repack</code>, it copies the full rows as they are. This behaviour renders it ineffective at removing bloat created due to dropped columns (behaviour still present at the time of the writing of this article at the end of April 2024). As already mentioned, pg_squeeze</code> can use a specified index for clustering when needed. The limitation you might find is that clustering cannot be performed on a partial index. Compared to pg_repack</code>, it always seems to clean up all the artefacts accordingly (thanks to the always-running worker process). Unfortunately at the moment of writing the article pg_squeeze</code> is not available for Amazon RDS, only Google Cloud SQL. Conclusion</a> </h2> It's remarkable to have two mature and production-tested tools at your disposal. Deciding between them comes down to the specific requirements, the use cases, and the operational specifics of the business services. The very opinionated difference between the tools can be made, using pg_squeeze</code> for automated, continuous cleaning of specific tables, and pg_repack</code> as the heavyweight champion of controlled setups. (Edit) Added availability on Amazon RDS and Google Cloud SQL.