Published
Yesterday, in 1.6.0, we stopped readers from holding up writers. Today we returned the favour.
Galactus DB 1.6.1 stops writers holding up readers. Until now, while a write transaction was open, every reader on that database queued until it finished. With writers busy, readers barely got a turn. In 1.6.1 they don't queue at all: with eight clients running write transactions and eight reading, reads went from 9 a second to 30,635, and the slowest reads from about 10 seconds to under half a millisecond.
That is not a typo, and it is not a new cluster. It is the same database on the same machine, with readers that no longer wait in line.
Yesterday's paper
Think of a newsroom. While today's edition is being typeset, nobody hands you half a page with the ink still wet. But nobody makes you stand at the printer's door until it is finished, either. They hand you yesterday's paper: complete, consistent, and yours straight away.
That is what a snapshot read does. When a writer changes something, Galactus DB keeps the old value for as long as a reader might still need it. A reader that arrives mid-write is shown the last finished version of the graph: it never sees half-done changes, and it never waits. When nobody needs the old copies any more, they are gone.
1.6.0 built this machinery into the engine. 1.6.1 puts it to work in the server, and turns it on by default.
What you get
- Reads that never queue behind writes. An autocommit read sees everything committed so far, and nothing from a transaction still in progress. With group durability it reads the last commit that is already on disk, so it never waits for a sync either.
- Repeatable read transactions. A read transaction (your driver's
execute_read) now reads one snapshot fromBEGINtoCOMMIT. Ask the same question twice and you get the same answer, whatever commits in between. - Fair writers. Writers now take turns in the order they arrived. Before, a writer that had just finished could jump straight back in ahead of the others, and one unlucky writer could wait seconds for its turn.
- A memory cap. Old values are kept only while someone needs them, and
never more than 64 MiB per database by default. A read transaction that
stays open through a flood of writes gets a polite, retryable
SnapshotTooOldand runs again; the official drivers retry it for you in managed transactions.
It works in every durability mode, and the slowest reads improved most where the wait was longest:
| Durability | Reads/s, 1.6.0 | Reads/s, 1.6.1 | Slowest 1% of reads, 1.6.0 | 1.6.1 |
|---|---|---|---|---|
| Buffered | 9 | 30,635 | 10.0 s | 0.4 ms |
| Group | 26 | 29,115 | 5.0 s | 0.5 ms |
| Sync | 1 | 18,888 | 10.0 s | 2.2 ms |
One footnote
Readers who no longer queue still need a turn at the graph. Writers now hand
them one between statements, so while readers are busy, writers go a little
slower: a third fewer write transactions in buffered and group, a fifth
fewer in sync. With nobody reading, writers go exactly as fast as before.
We think that is the right trade. A third off the writers' pace in exchange for thousands of times more reads, and reads that answer in a fraction of a millisecond instead of seconds. If your workload says otherwise, it is one setting to turn off (see below).
Settings
Snapshot reads are on by default. Two settings, both new:
GDB_SNAPSHOT_READS(defaulttrue): reads and read transactions read the last committed state instead of waiting for writers.falsebrings back 1.6.0's behaviour.GDB_SNAPSHOT_MAX_BYTES(default67108864, 64 MiB): memory old values may use per database for open read transactions.0removes the cap.
The transactions guide has the details.
How we measured
A release build of the server ran natively on an Intel Core i9-13900K with
64 GB of RAM and a Samsung 990 PRO NVMe drive, on Windows 11. Clients used the
official Neo4j Python driver 5.28, one process per client, against a graph of
20,000 people with an index on name and 40,000 KNOWS relationships.
Writers ran execute_write transactions of ten property updates and two new
relationships, each statement answered before the next was sent. Readers
alternated an index lookup with a two-hop count. Each run lasted ten seconds;
figures are medians of three rounds that alternated between snapshot reads
off and on. "1.6.0" in the charts is 1.6.1 with GDB_SNAPSHOT_READS=false,
which reads exactly as 1.6.0 did.
With snapshot reads off, workloads that never take a snapshot are level with 1.6.0, within 2% in alternating runs.
Upgrading
Pull and restart:
docker pull ianknowles/galactus-db:1.6.1
The data, commit log and snapshot formats are unchanged, so 1.6.0 and 1.6.1 open each other's databases. Nothing to configure: snapshot reads are on as soon as it starts.
What's next
Readers no longer wait for writers, and writers no longer wait for readers. Next, writers that work side by side rather than taking turns.
Two releases in two days. Nobody waits for anybody, including you.