What you are looking at
Three servers hold one database, orders, between them. Each holds a full
copy. One of them leads at any moment, and has the lime frame. Your
application, on the left, is connected to one server over plain Bolt; click a
server to connect it to another.
The row of squares under each server is the end of its log. An entry is amber when it has been written and green once most of the servers have it on disk, which is the moment the write is acknowledged.
- A write goes to whichever server your application is connected to. If that server does not lead, it carries the write to the one that does, on the same connection. The leader writes it, sends it to the others, and answers once two of the three hold it.
- A read runs on the server it was sent to. That server first asks how far the cluster has committed and waits until it has caught up, so the read sees every write that was acknowledged before it.
The change of leader plays at the speed we measured. Everything else is slowed down so you can follow it.
Things to try
- Reads and writes: connect to a server that does not lead and send a write. It reaches the leader anyway. It needs no routing driver and no proxy.
- Lose a server: stop the leader. The other two choose a new one between them, and the next write is acknowledged. Start the old leader again and it is sent what it missed.
- Lose a server: stop a follower. Two of three are still enough, so nothing changes for your application. Stop a second server and the cluster says it cannot acknowledge a write; it never reports a success it cannot keep.
- Lose a server: cut the leader off mid-write. Your application is told the outcome is unknown. When that server returns, it drops the write no other server ever had and takes what the others committed.
- One address: send your application through the router, a small program of ours in front of the cluster. Each new connection goes to a server that is ready, the one with the fewest connections. Stop the server your application is on: its driver connects again to the same address and is given another.
- Add a server: start a fourth server and enable it with one statement. Drain an old server with one more: the cluster gives the new server a copy, then a vote, and takes the old server out by itself, with writes going on throughout. Then drop the server that holds nothing.
- Repairs itself: lose a server for good. The cluster marks it down, waits ten minutes in case it returns (played in seconds), then makes its copy again on the spare server. Start the lost server again and it lets go of the copy it held.
The numbers
With three server processes and the default timings, a write sent through another server was accepted 2.6 seconds after we killed the leader's process: between 2,562 and 2,624 ms in six runs out of six. The three ran on one machine, so that figure has no network delay in it.
The code that decides who leads and what is committed is our own, written on the standard library alone. Before any server ran it, a simulator ran it one million times for 10,000 steps each, delaying, reordering, duplicating and dropping messages, cutting links and crashing servers. Those runs committed 367 million entries, elected 7.5 million leaders and checked 154.6 million reads against what they should have seen, with no rule broken.
Measured on Nebula Cluster 3.0.0-alpha.1, 19 November 2026, on an Intel Core i9-13900K with 64 GB of RAM.
The ten minutes a server may be gone before its copies are made again is the default, and a setting.
Learn more
- Nebula Cluster
- Enterprise
- Replica databases, for a read-only copy you promote yourself
