boringSQL | Supercharge your SQL & PostgreSQL powers - boringSQL

The tests passed. The plan didn't.

2026-07-12T20:15:00+00:00

TL;DR - RegreSQL 1.0 tested that your queries return the right rows. 2.0 tests that they return them the right way, and it does the checking against production's real statistics instead of your empty dev database, which lies.

A migration cleanup dropped an index nobody thought was that important. Every test passed: same rows, same order, green. Three days later the API started timing out on a query that hadn't changed a character, because the planner had quietly switched it from an index scan to a sequential scan over a table that had kept growing.

The first version of RegreSQL would have passed that change too. It tests what your queries return: run them, diff the rows against a committed expected file, go red when the output changes. That catches the query that now returns the wrong rows. It says nothing about the query that returns the right rows the wrong way, which is most of what takes a database down.

Version 2.0 tests that.

Test the plan, not just the rows

Here is that failure on a laptop. An orders table, an index on customer_id, and a query that reads one customer's orders:

-- orders-by-customer.sql
select id, total
  from orders
 where customer_id = :cid
order by id;

Baseline it and run the tests. Green:

$ regresql test
  ✓ 2 passing

Now drop the index the way that migration did, and run the same tests again:

$ regresql test
  ✓ 1 passing
  ✗ 1 failing

FAILING:
  orders-by-customer.1.buffers (3898 > 109 * 102%, +3476.1%)
  Expected buffers: 109
  Actual buffers:   3898 (+3476.1%)

  Cost (info):      7962.41 (baseline: 421.03)

  ⚠️ Table 'orders': Bitmap Heap Scan → Seq Scan

Here is the whole loop, start to failure, as it actually runs:

The output check still passes; the rows didn't change. The plan check catches what the output check can't: the same query, returning the same result, now runs a sequential scan instead of the bitmap index scan it used before, reading thirty-five times the buffers. That failure blocks the merge. And the diff names the table and the exact change, in the same shape as the row diffs you already read in code review. Plan drift stops being something you find in production and becomes something you review in a pull request.

An empty database is a liar

That demo caught the regression because the table was big enough for the index to matter. On your dev database, it usually isn't. Run EXPLAIN there and it says every query is fine, and it isn't lying on purpose: at a few hundred rows the cheapest plan really is a sequential scan, and it picks it. Production's planner reads different statistics and picks a different plan. A green suite on small data proves your queries are correct on small data, and nothing about the plan production will run. That gap, and why code review can't see it, is a story of its own.

You can't copy production data to your laptop. You don't need to. You need what the planner reads: the row counts, the histograms, the most-common-values. PostgreSQL 18 dumps and loads those on their own (pg_dump --statistics-only), and RegreSQL 2.0 injects them so EXPLAIN, against ten rows on your laptop, sees the real table's distribution:

regresql test --stats production-stats.sql

Now the plan you baseline is the plan production would pick, and a plan regression fails on your laptop before it reaches a server. That is the difference between testing your query and testing it against reality.

Gate on what happened, not what was guessed

Cost estimates are the planner's opinion. Opinions drift, and they drift most exactly when a plan is about to go wrong, because a bad plan is usually a bad estimate first. Gating a test on estimated cost means trusting the number that breaks first to tell you when it broke.

So 2.0 gates on measured behavior instead. regresql baseline --analyze runs the query for real and records what happened: buffers actually read, whether a sort or hash spilled to disk, how many rows actually flowed through each node, and the cardinality error, the ratio between what the planner predicted and what it got (the "q-error"). A query whose estimates were off by 2x and are now off by 200x has a plan that is one data change away from collapse, and the row counts say so long before the cost does. The baseline stores actuals; the test compares actuals; the estimate is never on the witness stand.

Wall-clock time is the actual everyone wants and the one that lies most, because it moves with cache state, whatever else is on the box, and luck. 2.0 measures it and treats it as evidence, not a verdict: interleaved runs, a per-query median, and a noise threshold derived by permuting the samples, so a query counts as slower only when it beats its own measurement noise. Anything that can't clear the threshold goes in an unstable bucket instead of failing the build. Timing informs the report, but only the measured actuals (buffers, spills, cardinality error) can fail the build.

Knowing when you can't tell

Here is the part that took the longest to get right, and the part I am most sure of.

Point RegreSQL at the same corpus on two builds of PostgreSQL and it will show you every query that plans differently between them. Early on, one query looked like a clean regression: the newer build read nearly three times the buffers, deterministically, run after run. It had every mark of a real find.

It was not one. Both builds had ANALYZE'd their own copy of the data, and the query sat on a knife-edge join-order decision that a few unlucky sampled rows tip one way or the other. Inject one build's exact statistics into the other and they plan identically. Re-ANALYZE and the "regression" moves to the other build. It was sampling noise, and a tool with less doubt would have reported it as a regression.

So 2.0 will not make a cross-version claim on statistics it does not trust. It injects identical statistics into both builds, so any remaining difference is code and not luck, and it re-ANALYZEs the baseline several times to throw out queries whose plan is a coin-flip to begin with. A difference is reported only when it survives both. On that investigation the honest output was zero regressions, and zero was the correct answer.

A tool that only ever reports "found something" loses credibility over time, so RegreSQL will also report "I can't tell you that from this data" when the evidence doesn't support a verdict - that's a valid result here, not an apology.

Testing the planner itself

Chase these problems far enough and you end up at PostgreSQL's own test suite. pg_regress, the harness that has guarded the database for decades, checks one thing: does the query return the expected text. It says nothing about the plan, the cost, or the cardinality error, because it folds all of them into a single .out text file that either matches or does not. It is correctness-only by construction, and nothing in the standard toolchain measures whether a planner change made real queries better or worse.

Everything above, separating results from plan-shape from measured actuals from cardinality error into layers that can each be judged on their own, is what you would need to answer that question. Which is why 2.0 also has the pieces that only make sense pointed at PostgreSQL rather than at your application:

Cross-version scoreboards with the trust filter above, so a planner patch can be measured against a corpus without drowning in ANALYZE noise.
A metamorphic check: flip an optimization that is supposed to preserve results (eager aggregation, memoize, incremental sort) and confirm the rows do not move. When they move, you have found a wrong-results bug in the optimizer, with no second database to compare against.
Determinism admission, which keeps only the queries whose result is stable across different plans, so an ambiguous LIMIT over ties never masquerades as a regression.

This is a longer road, and an open one. But the same machinery that tells you your application query got slower can tell a PostgreSQL developer whether their patch made a thousand real queries slower, and that is a thing the project cannot currently see per-change. RegreSQL started as a way to test the queries you write. It turns out the honest way to do that is most of the way to a harness for the planner underneath them.

Leave evidence

None of this is useful if it only runs on your laptop. regresql test exits non-zero on a failure, emits JUnit, GitHub Actions, or JSON, and reads DATABASE_URL so the same project runs against a CI database without editing committed config. A plan flip fails the build the way a wrong-rows change does, and the run leaves a machine-readable record of what was checked against which snapshot. The check that advised you in the editor is the same check that blocks the merge.

You can go further and name the tables where a sequential scan is never acceptable, and have that fail the build outright. But which tables those are, and which of these signals is a release blocker versus a shrug, is a judgment the planner doesn't have and never will. That part is a longer story, and a separate one.

What "verified" means

RegreSQL 2.0 is careful about the word. It does not promise your database is safe. It verifies what it can measure and says so plainly when it can't, which on the day it mattered meant reporting nothing at all. A green RegreSQL run is not "nothing will go wrong." It is "these queries return what they returned, plan the way they planned, against the statistics that actually apply." That is a smaller promise than most tools make, and one you can keep.

It runs on your laptop, against a database you already have, no production access:

regresql init postgres://localhost/yourdb

From there the SQL passes through the harness the way your code already passes through the type checker. The getting-started guide walks the whole loop, from zero to a plan check in CI, in about ten minutes.

Don't let AI touch your production database

2026-04-06T22:24:00+02:00

Not so long ago, the biggest threat to production databases was the developer who claimed it worked on their machine. If you've attended my sessions, you know this is a topic I'm particularly sensitive to.

These days, AI agents are writing your SQL. The models are getting incredibly good at producing plausible code. It looks right, it feels right, and often it passes a cursory glance. But "plausible" isn't a performance metric, and it doesn't care about your execution plan or locking strategy.

AI-generated SQL is syntactically correct, which is the easy part. The hard part is knowing what a statement does to a running system: which locks it takes, how long it holds them, whether it rewrites the table on disk.

The spectacular failures get the headlines. In July 2025, an AI coding agent wiped a production database during a code freeze -- ran destructive commands, panicked, then lied about what it had done.

But the biggest damage is quieter. It's the migration that passes every test, ships through CI and then locks a production table during peak traffic. The query written with random assumptions. Indexes added based on copy/paste from psql. The cumulative effect that builds over time, when nobody is looking.

How to give AI agent "eyes"

When you're running Claude Code or any other agentic coding tool, the bottleneck isn't the model's intelligence. It's the fidelity of the environment. Standard AI coding involves the agent guessing column names based on your description, or in better cases parsing a schema.sql file or using a local database with seed data.

None of these give the agent the one thing it actually needs: awareness of your production schema. The table sizes, the indexes, the constraints, the statistics that determine whether a query index-scans in 2ms or sequential-scans for 40 seconds.

The obvious fix is to give it a database connection. Let it query pg_catalog, read table sizes, check existing indexes. This is how Anthropic's reference PostgreSQL MCP server worked, and that's the one Datadog Security Labs found a SQL injection in. The vulnerable npm package still gets 21,000 downloads per week. The server wrapped queries in read-only transactions, but accepted semicolons. COMMIT; DROP SCHEMA public CASCADE; works because the COMMIT ends the read-only transaction and everything after runs with full privileges. Read-only transactions are not a security boundary when the protocol accepts semicolons.

The AI doesn't need a connection. It needs the information. Schema metadata is structural data. You can extract it once, save it as a JSON file, commit it to git, and hand that file to every tool in your stack.

MCP is an open standard for connecting AI assistants to external tools and data sources. Instead of relying on training data, the agent calls structured tools exposed by an MCP server and gets deterministic answers back.

What's been missing is a standard way to wire that information into the agent's workflow. Not a one-time dump that sits in a docs folder, but a protocol that lets the agent call tools and get structured answers back. That protocol is MCP (Model Context Protocol).

What are PostgreSQL MCP servers actually for?

This has been bugging me. If you search for "PostgreSQL MCP server" today, you'll find two kinds.

The first is a connection wrapper. You give it a DATABASE_URL, it exposes tools like query and execute. This is psql with extra steps. The Anthropic reference server that got the SQL injection? Connection wrapper. The dozen community forks that replaced it? Connection wrappers with slightly better input validation.

The second is a managed-provider gateway. Cloud database platforms expose their platform APIs through MCP. Create a branch, provision a database, manage users. Useful if you're on that platform, but not database intelligence. Control-plane operations wearing a protocol hat.

Neither kind answers the question an AI agent actually needs answered: "is this SQL going to be a problem?"

A connection wrapper can run EXPLAIN, if you trust it with credentials. But who tells the agent that SET NOT NULL on a 4-million-row table takes an ACCESS EXCLUSIVE lock? Who tells it that its query defeats partition pruning? That's not a connection problem. It's a knowledge problem.

dryrun

I didn't set out to build an AI tool. My primary project is RegreSQL, a statistics-based SQL regression testing framework. RegreSQL needs production statistics to produce realistic query plans in test environments, and dryrun's schema capture was originally just the plumbing to get those statistics into the right place.

But driving adoption of a testing tool is hard. The best place to plug in is where the coding actually happens. And these days, that's increasingly AI agents. Instead of fighting it, I turned the plumbing into a proper tool.

dryrun is a PostgreSQL schema intelligence tool. It ships as a CLI and as an MCP server. It doesn't connect to your database at runtime. Instead, it works from a JSON snapshot of your schema, captured once, committed to git, and available to everyone.

Offline first

One person with production access captures the schema:

dryrun dump-schema --source "$PROD_URL" --pretty --name "production" -o schema.json

The JSON contains everything structural: tables, columns, constraints, indexes, and the rest of what you'd find in pg_catalog. It also captures per-table statistics from pg_statistic (histograms, most common values, correlation, n_distinct) and runtime counters from pg_class like reltuples, seq_scan, and idx_scan. One file, typically 200-500KB. Structural metadata, not row data.

If your schema uses COMMENT ON (and it should), dryrun captures those too. A column named status is ambiguous. COMMENT ON COLUMN orders.status IS 'pending | completed | refunded, set by payment processor webhook' tells the agent what the values mean, where they come from, and that it shouldn't invent new ones.

If you don't take my word for it, check what Bruce Momjian has to say about it. MCP clients can only work with schemas, object names, and constraints. Comments are the simplest way to supply the semantic context that makes AI-generated queries meaningful.

Import it and every subsequent command works offline:

dryrun import schema.json

No connection needed, no credentials on developer machines or in CI.

As an MCP server

claude mcp add dryrun -- dryrun mcp-serve

Claude now has 16 tools for exploring your schema, validating queries, checking migrations, and suggesting improvements, all from the offline snapshot.

What the agent gets

dryrun exposes 16 tools through MCP. The four that matter most for the argument in this article:

check_migration is the one I find myself using most often, even outside of AI workflows. It analyzes DDL statements for lock implications, table rewrites, and version-specific behavior, and returns the safe multi-step alternative when one exists. Every migration example in this article was caught by this tool.

suggest_index reads column statistics from the snapshot and recommends indexes informed by your actual data distribution. Not "add an index on this column" but "add a partial index on the 3% of rows that match your query pattern." It also uses correlation data to recommend BRIN indexes where appropriate: 3MB instead of 400MB for a B-tree on a column with near-perfect insertion order.

validate_query parses SQL against the schema snapshot without executing it. It catches references to tables and columns that don't exist, detects anti-patterns, and flags issues before the query goes anywhere near the database.

lint_schema runs 18 convention rules and 14 structural audit checks: naming, types, constraints, timestamps, partition health, duplicate indexes, foreign key columns missing indexes, tables without primary keys. All disableable in dryrun.toml, because not every convention fits every project.

The remaining tools cover schema exploration (list_tables, describe_table, search_schema, find_related), operations (compare_nodes, detect, vacuum_health), execution plans (explain_query), and schema maintenance (refresh_schema, check_drift). The full list is in the documentation.

Try it yourself

Here's a schema and seed data you can run in any local PostgreSQL:

CREATE TABLE customers (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    name text NOT NULL,
    email text,
    created_at timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE orders (
    id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    customer_id bigint NOT NULL,
    status text NOT NULL DEFAULT 'pending',
    total_amount numeric(10,2),
    created_at timestamptz NOT NULL DEFAULT now()
);

Seed it with data that has the distributions that matter, like skewed status values and nullable emails:

-- 100K customers, ~5% with NULL email
INSERT INTO customers (name, email, created_at)
SELECT
    'customer_' || i,
    CASE WHEN random() < 0.05 THEN NULL
         ELSE 'user' || i || '@example.com'
    END,
    '2023-01-01'::timestamptz + (i || ' seconds')::interval
FROM generate_series(1, 100000) AS i;

-- 1M orders, 97% completed / 3% pending
INSERT INTO orders (customer_id, status, total_amount, created_at)
SELECT
    1 + (random() * 99999)::int,
    CASE WHEN random() < 0.03 THEN 'pending'
         ELSE 'completed'
    END,
    (random() * 500)::numeric(10,2),
    '2024-01-01'::timestamptz + (i || ' seconds')::interval
FROM generate_series(1, 1000000) AS i;

ANALYZE;

Now capture the schema for dryrun and install it as an MCP server:

dryrun dump-schema --source "$DATABASE_URL" -o schema.json
dryrun init
dryrun import schema.json
claude mcp add dryrun -- dryrun mcp-serve

Open Claude Code and try these prompts. First without the dryrun MCP server, then with it.

"Add an index to speed up queries that filter orders by status"

Without dryrun, Claude sees a text column and does the obvious thing:

CREATE INDEX idx_orders_status ON orders (status);

A full B-tree index on 1 million rows. It works, but 97% of those rows are 'completed' and nobody queries for completed orders.

With dryrun, Claude calls suggest_index, sees most_common_freqs showing 'completed' at 97% and 'pending' at 3%, and suggests:

CREATE INDEX idx_orders_status_pending ON orders (status)
    WHERE status = 'pending';

The full index maintains 1 million entries. The partial index maintains 30,000. Every INSERT that creates a 'completed' order no longer pays for index maintenance it'll never use.

"Make the email column on customers required"

Without dryrun, Claude writes a single statement and moves on:

ALTER TABLE customers ALTER COLUMN email SET NOT NULL;

This fails immediately because 5% of rows have NULL emails. Claude didn't know the data existed. Even if the column had no NULLs, the statement would still take an ACCESS EXCLUSIVE lock and perform a full table scan to verify.

With dryrun, Claude calls check_migration and describe_table. It sees 100,000 rows, knows the column is nullable, and knows from the statistics that NULLs exist. It generates a three-step migration:

-- Step 1: backfill NULLs
UPDATE customers
    SET email = 'unknown-' || id || '@example.com'
    WHERE email IS NULL;

-- Step 2: add constraint without full table scan
ALTER TABLE customers
    ADD CONSTRAINT customers_email_not_null
    CHECK (email IS NOT NULL) NOT VALID;

-- Step 3: validate in the background with a weaker lock
ALTER TABLE customers
    VALIDATE CONSTRAINT customers_email_not_null;

The first step handles the data. The second adds the constraint without scanning existing rows. The third validates with a weaker lock that allows concurrent reads and writes. Three statements instead of one.

Keeping the snapshot fresh

dryrun is only as good as your last snapshot. If the schema changed yesterday and nobody re-dumped, the agent is working from stale data. The check_drift tool can detect this, but only if someone runs it.

The best approach is to treat schema snapshots like you treat production backups: automate them. If you're already capturing production schema dumps on a schedule, add dryrun to the same pipeline. A nightly dump-schema from the primary gives you current structure and statistics. If you run read replicas with different workloads, add --stats-only dumps from each node for higher-fidelity statistics across the cluster. Commit the resulting schema.json to your repository so every developer and every AI agent starts the day with a current view.

The snapshot also serves as a lightweight audit trail. If someone adds a column through psql without a migration, the next scheduled dump captures it, and the diff in git shows exactly what changed and when.

What dryrun is not

dryrun does not execute arbitrary SQL against your database. It does not manage migrations. It does not provision infrastructure. It is not an ORM, a query builder, or a database admin panel. It's read-only and narrow by design, trustworthy enough that you'd let an AI agent use it unsupervised.

What's next

dryrun ships today at v0.5.0. The next releases are focused on:

Shadow EXPLAIN (v0.6.0): inject production statistics into a local PostgreSQL instance so EXPLAIN produces realistic plans from your laptop, without any production data. Currently explain_query runs against whatever local database you point it at, but if that database has 10 rows in every table, the planner gives you toy plans that tell you nothing about production behavior.
CI/CD Trust Gate: a single command that takes a migration file and runs safety analysis, schema diff, lint, and index suggestions in one pass. Designed to block unsafe migrations before they merge.
Improved migration support: deeper analysis of multi-statement migrations, version-specific advice, and integration with common migration frameworks.

Getting started

dryrun is MIT licensed and available on GitHub. The tutorial covers installation, offline and online workflows, multi-node setups, and MCP integration. It's part of the boringSQL suite.

RegreSQL: Regression Testing for PostgreSQL Queries

2025-11-13T22:47:00+00:00

TL;DR - RegreSQL brings PostgreSQL's regression testing methodology to your application queries, catching both correctness bugs and performance regressions before production.

As puzzling as it might seem, the common problem with production changes is the ever-present "AHA" moment when things start slowing down or crashing straight away. Testing isn't easy as it is, but there's a widespread practice gap when it comes to testing SQL queries. Some might pretend to "fix it" by using ORMs to abstract away the problem. Others treat SQL as "just glue code" that doesn't deserve systematic testing. Most settle for integration tests that verify the application layer works, never actually testing whether their queries will survive the next schema change or index modification.

For PostgreSQL development itself, the project has a robust regression test suite that has been preventing disasters in core development for decades. The database itself knows how to test SQL systematically - we just don't use those same techniques for our own queries. Enter RegreSQL, a tool originally created by Dimitri Fontaine for The Art of PostgreSQL book (which is excellent for understanding and mastering PostgreSQL as a database system), designed to bring the same regression testing framework to our application queries.

I've been trying to use it for some time, but due to missing features and limitations gave up several times. Until now. I decided to fork the project and spend the time needed to take it to the next level.

Introduction

The RegreSQL promise starts with the biggest strength and perceived weakness of SQL queries. They are just strings. And unless you use something like sqlc (for Go), PG'OCaml or Rust's SQLx toolkit giving you compile-time checking, your queries are validated only when they are executed. Which in better case mean either usually slow-ish test suite or integration tests, in worst scenario only when deployed. ORMs are another possibility - completely abstracting away SQL (but more on that later).

But even with compile-time checking, you are only checking for one class of problems: schema mismatches. What about behavior changes after schema migration or performance regressions? What about understanding whether your optimization actually made things faster or just moved the problem elsewhere?

This is where RegreSQL comes in. Rather than trying to turn SQL into something else, RegreSQL embraces "SQL as strings" reality and applies the same testing methodology PostgreSQL itself uses: regression testing. You write (or generate - continue reading) your SQL queries, provide input data, and RegreSQL verifies that future changes don't break those expectations.

The features don't stop there though - it tracks performance baselines, detects common query plan regressions (like sequential scans), and gives you framework for systematic experimentation with the schema changes and query change management.

Basic regression testing

Enough with theory. Let's jump in straight into the action and see what a sample run of RegreSQL looks like

$ regresql text
Connecting to 'postgres://radim:password123@192.168.139.28/cdstore_test'… ✓

Running regression tests...

✓ album-by-artist_list-albums-by-artist.1.json (0.00s)
✓ album-by-artist_list-albums-by-artist.2.json (0.00s)

✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)
✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)

✓ artist_top-artists-by-album.1.json (0.00s)

✓ genre-topn_genre-top-n.top-1.json (0.00s)
✓ genre-topn_genre-top-n.top-3.json (0.00s)

✓ genre-tracks_tracks-by-genre.json (0.00s)

Results: 8 passed, 0 failed, 8 skipped (0.00s)

In this example based on Chinook database (as used originally in The Art of PostgreSQL book), RegreSQL scans the current directory (or one provided by -C /path/to/project) for *.sql files and attempts to run all queries against the configured PostgreSQL connection.

The individual files can contain either single or multiple sql queries. Like following example

-- name: top-artists-by-album
-- Get the list of the N artists with the most albums
SELECT
    artist.name,
    count(*) AS albums
FROM
    artist
    LEFT JOIN album USING (artist_id)
GROUP BY
    artist.name
ORDER BY
    albums DESC
LIMIT :n;

The syntax for the queries supports both positional arguments (like $1 known from libpq library) or (preferred) psql style variable (:varname). The each identified query (not file) is then executed for 0..N times, based on number of predefined plans and verified to the expected results - validating the expected data matches the one returned. The support for SQL files handling is available separately with https://github.com/boringSQL/queries (Go version only for now).

This gives you what original RegreSQL tool has introduced - change your schema, refactor a query, run regresql test and see immediately what broke. The test suite now has ability to catch regressions before they are committed / shipped. The current version built on top of it, giving you better console formatter instead of TAP style output, as well as jUnit, JSON and GitHub actions formatters for better integration into your CI/CD pipelines.

Performance regression testing

Basic regression testing catches correctness issues - wrong results, broken queries, schema mismatches. But there's another class of production issues it misses. Performance regressions. No matter how unbelievable it might sound but queries get deployed without appropriate indexes — which might then be ignored entirely — or they change over time. Simple fix - both for handwritten SQL or ORM code - can switch from milliseconds to seconds. You add index that helps one query, but tanks another. You modify conditionals and accidently force a sequential scan of millions of rows. This is where it hurts.

RegreSQL addresses this by tracking performance baselines alongside correctness. Once baselines are generated

$ regresql baseline
Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓
Creating baselines directory: regresql/baselines
Creating directory 'regresql/baselines'

Creating baselines for queries:

  ./
  Created baseline: album-by-artist_list-albums-by-artist.1.json
  Created baseline: album-by-artist_list-albums-by-artist.2.json
  Created baseline: album-tracks_list-tracks-by-albumid.1.json
  Created baseline: album-tracks_list-tracks-by-albumid.2.json
  Created baseline: artist_top-artists-by-album.1.json
  Created baseline: genre-topn_genre-top-n.top-1.json
  Created baseline: genre-topn_genre-top-n.top-3.json
  Created baseline: genre-tracks_tracks-by-genre.json

Baselines have been created successfully!
Baseline files are stored in: regresql/baselines

the test command not only tests the regressions to the captured times, but also detects the common bad patterns in query execution plans. For now it provides warnings for detection of sequential scans - both on their and/or with nested loops and multiple sort operations. I believe this alone might provide a valuable insights and reduce the mishaps in production. It's also a place where further development of RegreSQL will take place.

To demonstrate this, let's review the test output with the baselines.

Connecting to 'postgres://appuser:password123@192.168.139.28/cdstore_test'… ✓

Running regression tests...

✓ album-by-artist_list-albums-by-artist.1.json (0.00s)
✓ album-by-artist_list-albums-by-artist.2.json (0.00s)
✓ album-by-artist_list-albums-by-artist.1.cost (22.09 <= 22.09 * 110%) (0.00s)
  ⚠️  Sequential scan detected on table 'artist'
    Suggestion: Consider adding an index if this table is large or this query is frequently executed
  ⚠️  Nested loop join with sequential scan detected
    Suggestion: Add index on join column to avoid repeated sequential scans
✓ album-by-artist_list-albums-by-artist.2.cost (22.09 <= 22.09 * 110%) (0.00s)
  ⚠️  Sequential scan detected on table 'artist'
    Suggestion: Consider adding an index if this table is large or this query is frequently executed
  ⚠️  Nested loop join with sequential scan detected
    Suggestion: Add index on join column to avoid repeated sequential scans

✓ album-tracks_list-tracks-by-albumid.1.json (0.00s)
✓ album-tracks_list-tracks-by-albumid.2.json (0.00s)
✓ album-tracks_list-tracks-by-albumid.1.cost (8.23 <= 8.23 * 110%) (0.00s)
✓ album-tracks_list-tracks-by-albumid.2.cost (8.23 <= 8.23 * 110%) (0.00s)

✓ artist_top-artists-by-album.1.json (0.00s)
✓ artist_top-artists-by-album.1.cost (35.70 <= 35.70 * 110%) (0.00s)
  ⚠️  Multiple sequential scans detected on tables: album, artist
    Suggestion: Review query and consider adding indexes on filtered/joined columns

✓ genre-topn_genre-top-n.top-1.json (0.00s)
✓ genre-topn_genre-top-n.top-3.json (0.00s)
✓ genre-topn_genre-top-n.top-1.cost (6610.59 <= 6610.59 * 110%) (0.00s)
  ⚠️  Multiple sequential scans detected on tables: genre, artist
    Suggestion: Review query and consider adding indexes on filtered/joined columns
  ⚠️  Multiple sort operations detected (2 sorts)
    Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting
  ⚠️  Nested loop join with sequential scan detected
    Suggestion: Add index on join column to avoid repeated sequential scans
✓ genre-topn_genre-top-n.top-3.cost (6610.59 <= 6610.59 * 110%) (0.00s)
  ⚠️  Multiple sequential scans detected on tables: artist, genre
    Suggestion: Review query and consider adding indexes on filtered/joined columns
  ⚠️  Multiple sort operations detected (2 sorts)
    Suggestion: Consider composite indexes for ORDER BY clauses to avoid sorting
  ⚠️  Nested loop join with sequential scan detected
    Suggestion: Add index on join column to avoid repeated sequential scans

✓ genre-tracks_tracks-by-genre.json (0.00s)
✓ genre-tracks_tracks-by-genre.cost (37.99 <= 37.99 * 110%) (0.00s)
  ⚠️  Multiple sequential scans detected on tables: genre, track
    Suggestion: Review query and consider adding indexes on filtered/joined columns

Results: 16 passed (0.00s)

As you can see, despite from not having baseline, RegreSQL is able to detect the basic bad patterns that should be addressed before queries can be considered "production ready".

In some cases, having the detection of sequential scans, or just tracking query costs baselines might be considered undesirable, which would lead to false positives. RegreSQL enables this to be addressed by query metadata as demonstrated below.

-- name: query_name
-- metadata: key1=value1, key2=value2
SELECT ...;

At this point RegreSQL recognizes

notest to skip the query testing altogether (not just cost tracking)
nobaseline to skip cost tracking
noseqscanwarn to keep cost tracking but disable sequential scan warnings
and difffloattolerance to cost failure threshold (default 10% at the moment).

-- name: query_name
-- regresql: notest, nobaseline
-- regresql: noseqscanwarn
-- regresql: difffloattolerance:0.25
-- query that can vary in cost by 20% without being considered a failure
SELECT ...;

ORM enters the room

ORMs abstract away SQL, but they still generate it - much like view inlining where the planner rewrites your SQL behind the scenes - and that generated SQL can have performance problems you won't catch until production. Consider this common scenario: you start with a simple SQLAlchemy query that works fine, then months later add eager loading for related data:

orders = (
    session.query(Order)
    .filter(Order.user_id == user_id)
    .options(
        joinedload(Order.user),
        joinedload(Order.shipping_address),
        selectinload(Order.items)  # NEW: Load order items
    )
    .all()
)

That innocent selectinload(Order.items) generates a separate query - and without an index on order_items.order_id, it performs a sequential scan.

RegreSQL can catch this by intercepting ORM-generated SQL using SQLAlchemy's event system:

@event.listens_for(engine, "before_cursor_execute")
def capture_sql(conn, cursor, statement, *args):
    captured_queries.append(statement)

Run your ORM code, capture the SQL, save it as a .sql file, and test it with RegreSQL. The performance baseline testing will flag the missing index before it hits production. This is currently experimental, but ORM integration is a key area for RegreSQL's future development.

Test Data Management

Up until now we have covered how RegreSQL verifies query correctness and tracks performance regressions. But there's a critical prerequisite we've only skimmed through. Every regression test needs consistent, reproducible data. Change the data, change their cardinality, and your expected results become meaningless. Your performance baselines drift. Your tests become flaky.

Traditional approach to create test data might involve

Database dumps become unmanageable - 500MB files you can't review, can't understand, that break with every schema migration, and whose data becomes stale as production evolves. Which version of the dump are your tests even using?
SQL scripts might be better than dumps, but still imperative and hard to maintain. You end up with INSERT statements scattered across multiple files, managing foreign keys manually, and debugging constraint violations.
Factories in application code might work great for integration tests, but we're testing SQL directly. Do you really want to maintain parallel data generation in your application language just for SQL tests?
Shared test database is the synonym for classic "works on my machine" problem. State leaks between tests. Parallel execution becomes impossible. Debugging is a nightmare.

What we need is something that's declarative (what data, not how to insert it), reproducible (similar data every time), composable (build complex scenarios from simple pieces), and scalable (from 10 rows to 100,000).

This is where next improvement in RegreSQL's fixture system comes in. Think of it as infrastructure-as-code for your test data. You describe the data you need in YAML files, and RegreSQL handles the rest - dependencies, cleanup, foreign keys, and even realistic data generation at scale.

RegreSQL's fixture system lets you define test data in YAML files stored in regresql/fixtures/. Here's a simple example

  fixture: basic_users
  description: a handful of test users
  cleanup: rollback

  data:
    - table: users
      rows:
        - id: 1
          email: alice@example.com
          name: Alice Anderson
          created_at: 2024-01-15
        - id: 2
          email: bob@example.com
          name: Bob Builder
          created_at: 2024-02-20

To use this fixture in your tests, reference it in the query's plan file (regresql/plans/get-user.yaml) you can just reference the fixture

  fixtures:
    - basic_users

  "1":
    email: alice@example.com

  "2":
    email: bob@example.com

And when you run regresql test, the fixture is automatically loaded before the query executes, and cleaned up afterward. No manual setup scripts, no state leakage between tests. But it does not stop with static fixtures. When you want to test queries against realistic volumes you can use range of data generators including

sequences, random integer, decimal, string, uuid, email and name generators
date_between for generating random timestamps within a range
foreign key references to be able to reuse data from other table's fixtures
range to select value from predefined sources
Go template support

 fixture: realistic_orders
  generate:
    - table: customers
      count: 1000
      columns:
        id:
          generator: sequence
          start: 1
        email:
          generator: email
          domain: shop.example.com
        name:
          generator: name
          type: full
        created_at:
          generator: date_between
          start: "2023-01-01"
          end: "2024-12-31"

    - table: orders
      count: 5000
      columns:
        id:
          generator: sequence
          start: 1
        customer_id:
          generator: int
          min: 1
          max: 1000
        amount:
          generator: decimal
          min: 10.00
          max: 999.99
          precision: 2
        order_date:
          generator: date_between
          start: "2023-01-01"
          end: "2024-12-31"

This generates 1,000 customers and 5,000 orders with realistic-looking data - names, emails, dates, and amounts that feel production-like.

The fixtures are also stackable and can be build on top of each other. For example if you need to make sure users fixtures are created before orders fixtures, just declare the dependency (the already planned improvement is to include the support automatic foreign-key detection to avoid ID hard-coding). RegreSQL loads fixtures in dependency order and handles cleanup in reverse.

  fixture: orders_with_shipping
  depends_on:
    - basic_users

  data:
    - table: orders
      rows:
        - id: 101
          user_id: 1  # References Alice from basic_users
          total: 99.99
          status: shipped

Should the available options for fixtures (manual data or data generators) not be enough, you always have options to use good old SQL based data generation.

  fixture: mixed_setup
  description: Combine SQL with YAML and generated data
  cleanup: rollback

  # SQL executes first (either as file or inline)
  sql:
    - file: sql/setup_schema.sql
    - inline: "INSERT INTO config (key, value) VALUES ('version', '1.0');"

  # followed YAML data
  data:
    - table: users
      rows:
        - id: 1
          email: admin@example.com

  # and finally generated data
  generate:
    - table: orders
      count: 100
      columns:
        id:
          generator: sequence
          start: 1
        user_id:
          generator: int
          min: 1
          max: 1

RegreSQL provides commands to inspect and validate your fixtures

  # List all available fixtures
  regresql fixtures list

  # Show fixture details and dependencies
  regresql fixtures show realistic_orders

  # Validate fixture definitions
  regresql fixtures validate

  # Show dependency graph
  regresql fixtures deps

  # Apply fixture manually (for debugging)
  regresql fixtures apply basic_users

The fixture system has been design to transforms test data from a maintenance burden into a documented, version-controlled process. Your YAML files become the single source of truth for what data your tests need, making it easy to understand test scenarios and maintain test data as the application evolves.

(EDIT 2025-11-20) The dynamically generated fixtures on each tests will break the correctness testing of RegreSQL. You have to use fixtures according to your use

Fixed generated fixtures re-used between tests
Use RegreSQL to test fixtures before/after migration

I'm working on next release of RegreSQL to address DX of fixtures usage.

RegreSQL future

Introducing a new open source project is an ambitious goal, and RegreSQL is just starting up. Despite the fork being in works for almost 2 years. In coming weeks and months I plan further improvements, as well as better documentation and more tutorials. The project is maintained as part of my boringSQL brand, where it's vital component (together with pgTap) for building SQL Labs which (as I sincerely hope) will provide a foundation for its further development.

At the same time RegreSQL is an attempt to give back to welcoming PostgreSQL community, make developer user experience slightly better if possible and (just maybe) provide one more argument against the case that SQL queries are not testable.

RegreSQL is available at boringsql.com/products/regresql - feel free to open issue, or drop me email about the project at radim@boringsql.com or connect on LinkedIn.

Are SQL & Databases Boring? Absolutely - and That's a Good Thing!

2024-04-21T00:00:00+00:00

Twenty years ago, I would have laughed if you had told me I'd be promoting databases as the technology to turn to. Back then, databases were just a 'dummy' storage for me—a necessary evil, a sentiment shared by many developers. At that point, I felt so strongly about it that I was a trainer for Hibernate, a tool which helped me keep those pesky databases at arm's length.

Fast forward, and here I am, starting boringSQL, a site dedicated to helping demystify the depths of SQL and databases in general. Whether you've just started or want to gain expert knowledge. What changed, you might ask? Nothing actually. Well, a few things did. The frameworks changed. I moved from the Java world to Rails, jumped to Python and then to Go, worked with countless frontend paradigms, and oversaw the implementation of a number of technologies. The same experiments were tried again and again. Only one part of the tech stood the test of time. Yes, it's the database. And out of many, one in particular—PostgreSQL. And while alternatives promised, and still promise, that the grass would be greener—I can honestly say that whatever revolution comes in the technology landscape, databases will hold the same place in the next twenty years.

This is not the first boring site, and rest assured, the concept of celebrating the mundane in technology is far from new. The tech landscape is paved with many "boring" things, from frameworks to methodologies to complete stacks. Businesses depend on these technologies to run their operations day in, day out. And while it might be fun to try new things every month, it's not the headline-driven development that is the way forward.

Hence, if you ask—is SQL boring? Are databases boring? I would also say YES. They are among the finalists of the least popular conversation starters. But it's the quiet power of reliability that has a healing effect in a world where technology changes at the speed of light.