Published
They say lightning never strikes the same place twice. This week we took a third off memory, then did it again. We have stopped listening to what they say.
The first two strikes, 1.5.0 and 1.5.5, were about luggage. This one is about traffic.
Galactus DB 1.6.0 lets readers and writers share the road. With group durability, every acknowledged write is on disk before the client hears "done", and until now a busy reader could bring writers almost to a halt. In 1.6.0 they don't: with 32 writers and 32 readers on the same database, writes go from 600 a second to 8,598, the same pace the writers manage with nobody reading at all.
The traffic jam
Group durability is the polite way to be durable. Instead of every commit booking its own trip to disk, commits that arrive together share one: the first writer to need a sync runs it, and everyone queued behind it rides along. Under load, one sync can carry a dozen or more commits.
Readers have a promise to keep too. A reader must never show a client something that is not yet on disk, because if the power went out a moment later, that client would have seen a write that no longer exists. So when a reader found a commit still waiting for its sync, it waited for the sync.
The trouble was where it waited. It held the database's read lock while it stood there, like someone holding a door open while they wait for their taxi. Writers need that door for a few microseconds each, and they queued behind every reader for a whole disk sync. Add enough readers and the queue never cleared: 32 readers cut 32 writers to 600 commits a second.
The fix: let go of the door, then wait
In 1.6.0 a reader runs its query, notes the last commit it could see, lets go of its locks, and only then waits for that commit to reach disk before it replies. The promise is unchanged: no client ever receives data that isn't durable. But the door is free, so writers walk straight through.
The numbers in the chart came from the same machine and data on both versions:
| Readers | 1.5.5 writes/s | 1.6.0 writes/s |
|---|---|---|
| 0 | 8,900 | 8,900 |
| 4 | 2,235 | 8,848 |
| 16 | 838 | 8,620 |
| 32 | 600 | 8,598 |
There is one footnote. In 1.5.5, with 32 readers, reads flew along at 52,000 a second, but only because they were starving the writers. In 1.6.0 each read that overlaps a write waits for one disk sync, about 4 ms here, so 32 readers manage about 7,800 reads a second while the writers keep their full pace. Readers that don't overlap a write don't wait at all. Neither side starves any more, and the next part of this release story takes the readers' wait away too.
Group is now the mode to reach for
If your server has several clients and an acknowledged write must survive a power cut, use group durability:
GDB_DURABILITY=group
It gives the same guarantee as sync, it shares each disk sync between
everyone committing at once, and from 1.6.0 it no longer falls over when
readers join in. The durability guide now
says so.
Two cases still point elsewhere. A single client writing one commit after
another can't share a sync with anyone, so it is limited to a few hundred
commits a second on fast NVMe: batch its writes with UNWIND or larger
transactions, or choose buffered. And buffered, which stays the default,
is still the quickest way to bulk load or to hold data you can rebuild, as
long as you accept that a power cut can lose the writes acknowledged since
its last background sync (every 50 ms by default).
Under the bonnet: the first half of MVCC
The traffic fix ships in the server today. Underneath it, 1.6.0 also brings the first half of multi-version concurrency control into the engine: the machinery that lets a reader see a consistent snapshot of the graph while writers carry on changing it.
We tried snapshots once before, and it went badly. That version copied the whole graph at the start of every transaction: on a large graph, a second of work and twice the memory, every time. This one copies nothing.
- A snapshot is just a number: the commit it reads as of.
- Old versions come from the undo log, which every change already writes so that a transaction can roll back. Until now a commit threw those notes away. Now, if a reader still needs them, the commit keeps them, stamped with its number, and drops them as soon as nobody does.
- A reader takes the live answer and corrects it for the few nodes and relationships that changed since its snapshot. Everything else reads exactly as before.
With nothing to correct, the cost is one check. With a writer busy, a snapshot reader sees past another session's open transaction, and with group durability it can read the last commit that is already on disk instead of waiting for the next one. In our engine benchmarks, with 32 writers and 8 readers under group durability, that took writes from about 1,500 a second to 8,000 and reads from about 2,600 a second to 220,000.
The server doesn't use snapshot reads yet. Switching it over, so that readers stop waiting for open transactions and for disk syncs altogether, is what we are working on next.
Nothing else got slower
Splitting reads from writes touched every place the query engine looks at the graph, which is exactly the kind of change that quietly costs a few per cent everywhere. We measured until it didn't:
Reads from eight clients at once come out 1.8% quicker and commits 0.8%; everything else is level. Not a hare, this one. A tortoise that kept up.
How we measured
Both versions were built with the same release settings and run on an Intel Core i9-13900K with 64 GB of RAM and a Samsung 990 PRO NVMe drive, on Windows 11. The traffic figures use a 100,000-node graph with group durability, 32 writers each committing one Cypher statement at a time, and readers alternating an index seek with a two-hop count, driven through the same locks the server uses. They are medians of four five-second rounds that alternated between versions. The per-operation figures are medians of ten interleaved rounds on the same data.
Upgrading
Pull and restart:
docker pull ianknowles/galactus-db:1.6.0
The data, commit log and snapshot formats are unchanged, so 1.5.5 and 1.6.0
open each other's databases. If you run several clients and want every
acknowledged write on disk, add GDB_DURABILITY=group while you are there.
What's next
Snapshot reads in the server: readers that never wait for a writer, a disk sync or an open transaction. After that, writers that work side by side rather than taking turns.
Three strikes. We are told that's a rule in some sport or other. We are not sure which, and we are not stopping.