# Lightning strikes thrice: Galactus DB 1.6.0

They say lightning never strikes the same place twice. This week we took a
third off memory, then did it again. We have stopped listening to what they
say.

The first two strikes, [1.5.0](/blog/galactus-db-1-5-less-memory) and
[1.5.5](/blog/galactus-db-1-5-5-we-did-it-again), were about luggage. This one
is about traffic.

Galactus DB 1.6.0 lets readers and writers share the road. With group
durability, every acknowledged write is on disk before the client hears
"done", and until now a busy reader could bring writers almost to a halt.
In 1.6.0 they don't: with 32 writers and 32 readers on the same database,
writes go from **600 a second to 8,598**, the same pace the writers manage
with nobody reading at all.

![Writes per second as readers join, 1.5.5 against 1.6.0](/blog/charts/release-160-mixed.svg)

## The traffic jam

Group durability is the polite way to be durable. Instead of every commit
booking its own trip to disk, commits that arrive together share one: the
first writer to need a sync runs it, and everyone queued behind it rides
along. Under load, one sync can carry a dozen or more commits.

Readers have a promise to keep too. A reader must never show a client
something that is not yet on disk, because if the power went out a moment
later, that client would have seen a write that no longer exists. So when a
reader found a commit still waiting for its sync, it waited for the sync.

The trouble was *where* it waited. It held the database's read lock while
it stood there, like someone holding a door open while they wait for their
taxi. Writers need that door for a few microseconds each, and they queued
behind every reader for a whole disk sync. Add enough readers and the queue
never cleared: 32 readers cut 32 writers to 600 commits a second.

## The fix: let go of the door, then wait

In 1.6.0 a reader runs its query, notes the last commit it could see,
**lets go of its locks**, and only then waits for that commit to reach disk
before it replies. The promise is unchanged: no client ever receives data
that isn't durable. But the door is free, so writers walk straight through.

The numbers in the chart came from the same machine and data on both
versions:

| Readers | 1.5.5 writes/s | 1.6.0 writes/s |
|---:|---:|---:|
| 0 | 8,900 | 8,900 |
| 4 | 2,235 | **8,848** |
| 16 | 838 | **8,620** |
| 32 | 600 | **8,598** |

There is one footnote. In 1.5.5, with 32 readers, reads flew along at
52,000 a second, but only because they were starving the writers. In 1.6.0
each read that overlaps a write waits for one disk sync, about 4 ms here, so
32 readers manage about 7,800 reads a second while the writers keep their
full pace. Readers that don't overlap a write don't wait at all. Neither
side starves any more, and the next part of this release story takes the
readers' wait away too.

## Group is now the mode to reach for

If your server has several clients and an acknowledged write must survive a
power cut, **use group durability**:

```env
GDB_DURABILITY=group
```

It gives the same guarantee as `sync`, it shares each disk sync between
everyone committing at once, and from 1.6.0 it no longer falls over when
readers join in. The [durability guide](/docs/operations/durability) now
says so.

Two cases still point elsewhere. A single client writing one commit after
another can't share a sync with anyone, so it is limited to a few hundred
commits a second on fast NVMe: batch its writes with `UNWIND` or larger
transactions, or choose `buffered`. And `buffered`, which stays the default,
is still the quickest way to bulk load or to hold data you can rebuild, as
long as you accept that a power cut can lose the writes acknowledged since
its last background sync (every 50 ms by default).

## Under the bonnet: the first half of MVCC

The traffic fix ships in the server today. Underneath it, 1.6.0 also brings
the first half of multi-version concurrency control into the engine: the
machinery that lets a reader see a consistent snapshot of the graph while
writers carry on changing it.

We tried snapshots once before, and it went badly. That version copied the
whole graph at the start of every transaction: on a large graph, a second of
work and twice the memory, every time. This one copies nothing.

- **A snapshot is just a number**: the commit it reads as of.
- **Old versions come from the undo log**, which every change already writes
  so that a transaction can roll back. Until now a commit threw those notes
  away. Now, if a reader still needs them, the commit keeps them, stamped
  with its number, and drops them as soon as nobody does.
- **A reader takes the live answer and corrects it** for the few nodes and
  relationships that changed since its snapshot. Everything else reads
  exactly as before.

With nothing to correct, the cost is one check. With a writer busy, a
snapshot reader sees past another session's open transaction, and with
group durability it can read the last commit that is already on disk
instead of waiting for the next one. In our engine benchmarks, with 32
writers and 8 readers under group durability, that took writes from about
1,500 a second to 8,000 and reads from about 2,600 a second to 220,000.

The server doesn't use snapshot reads yet. Switching it over, so that
readers stop waiting for open transactions and for disk syncs altogether,
is what we are working on next.

## Nothing else got slower

Splitting reads from writes touched every place the query engine looks at
the graph, which is exactly the kind of change that quietly costs a few per
cent everywhere. We measured until it didn't:

![Time saved per operation, 1.6.0 against 1.5.5](/blog/charts/release-160-speed.svg)

Reads from eight clients at once come out 1.8% quicker and commits 0.8%;
everything else is level. Not a hare, this one. A tortoise that kept up.

### How we measured

Both versions were built with the same release settings and run on an Intel
Core i9-13900K with 64 GB of RAM and a Samsung 990 PRO NVMe drive, on
Windows 11. The traffic figures use a 100,000-node graph with group
durability, 32 writers each committing one Cypher statement at a time, and
readers alternating an index seek with a two-hop count, driven through the
same locks the server uses. They are medians of four five-second rounds
that alternated between versions. The per-operation figures are medians of
ten interleaved rounds on the same data.

## Upgrading

Pull and restart:

```text
docker pull ianknowles/galactus-db:1.6.0
```

The data, commit log and snapshot formats are unchanged, so 1.5.5 and 1.6.0
open each other's databases. If you run several clients and want every
acknowledged write on disk, add `GDB_DURABILITY=group` while you are there.

## What's next

Snapshot reads in the server: readers that never wait for a writer, a disk
sync or an open transaction. After that, writers that work side by side
rather than taking turns.

Three strikes. We are told that's a rule in some sport or other. We are not
sure which, and we are not stopping.