Published
The last time we wrote, in 1.6.1, readers stopped waiting for writers, and we signed off with "nobody waits for anybody, including you". We meant it. Four days and four releases later, here's everything that happened while nobody was waiting.
The short version: a faster way across a map, an Explorer you can put on the internet, backups that no longer stop the database, and replicas that live on another server. Also, we now host it for you, if you'd rather we did.
1.6.2: the scenic route, fixed
Sat-navs sometimes take the scenic route. Ours did too, without telling anyone.
gdb.astar added straight-line distance in metres to relationship weights in
whatever units you had used. Weight your roads in kilometres, and it believed
every destination was a thousand times further away than it was. It still
found a path, just not always the shortest: Exeter to Newcastle came back as
634 km instead of 624. Ten kilometres is a nice walk, but it isn't what
"shortest path" promises. A* now scales its guesses to your units, so it's
exact again.
While we were in there, A* learned to read Geography and Geometry points
as well as point(). Before, it quietly ignored them and became a plain
Dijkstra search, which is correct but rather misses the point of A*.
Then the headline act: Hourglass, a new exact shortest-path algorithm,
now built in as gdb.shortestPath.hourglass. It searches from both ends at
once, keeps the two searches in balance, and sorts the frontier into buckets
instead of a heap, so the two sides meet in the middle. It works on directed or
undirected graphs, with or without locations, as long as the weights are
whole numbers. You can watch it race across a road network in the
Hourglass demo, and the
pathfinding guide has the Cypher.
Explorer also got a haircut: the new Node G mark, the GALACTUS/DB wordmark
and a lime favicon you can find among forty open tabs.
1.6.3: Explorer goes outside
Galactus Explorer used to be a homebody. Point it at any address and it would happily connect, which is lovely on your laptop and much less lovely on a public website, where "any address" includes the machine it runs on.
1.6.3 adds GDB_EXPLORER_ALLOWED_HOSTS. Give it a list such as
*.db.galactusdb.com:7687, and Explorer only connects to those hosts, only
over verified TLS, and only to public internet addresses. No sneaking round
the back to the private network or the cloud's metadata service.
It also learned to open the login form already filled in, from a link:
/login?endpoint=bolt%2Bs://HOST:7687&user=NAME&name=LABEL. The password is
never taken from the URL, because some things should be typed.
1.6.4: backups without the hush
Until 1.6.4, taking a backup was like vacuuming during a dinner party: everyone stopped talking until you'd finished. A dump took the whole database for itself, and every reader and writer queued politely until it was done.
Now a dump reads a snapshot of the database as it was when the dump began, using the same machinery that let readers stop waiting in 1.6.1. It takes the database for one batch of 1,024 nodes or relationships at a time, and writes carry on in between. The file holds exactly the moment it started: nothing half-written, nothing from anyone else's unfinished transaction.
There is one catch, and we'd rather tell you than have you find it. While a
dump runs, every change to data it hasn't reached yet keeps the old value in
memory, so the dump can still copy it. If a busy writer makes that pile grow
past half of GDB_SNAPSHOT_MAX_BYTES, writers start pausing briefly after
each commit, and pause for longer the fuller it gets. Think of it as the
vacuum on its quiet setting. Dumps themselves are slower than before, and the
database is a little slower while one runs, but nobody has to stop talking.
1.6.5: a replica with its own postcode
Since 1.4.0, Galactus DB has kept replicas: read-only copies that follow a database a few milliseconds behind. Until now they had to live on the same server, like a backup singer who refused to leave the stage.
In 1.6.5 a replica can follow a database on another Galactus DB server, over Bolt:
CREATE DATABASE orders_ro AS REPLICA OF 'bolt+s://primary.example.com:7687/orders';
It starts from the non-blocking snapshot that 1.6.4 introduced, so the source
carries on serving while the copy is taken. After that it applies every
transaction, schema changes included, under the same transaction numbers as
the source. PAUSE, RESUME, RESYNC, PROMOTE, delayed replicas and the
rest all work as before.
A few thoughtful touches:
- The source keeps your place. If a replica pauses or loses its connection, the source holds its change stream from wherever the replica got to, so it catches up without starting again. It lets go when the replica is promoted or dropped, or after 24 hours of silence. That's enough time for a long weekend, though not for a fortnight in Portugal.
- Idle replicas sit quietly. A replica with nothing to do asks for news and waits a couple of seconds for an answer, rather than asking "anything yet?" over and over like a child in the back of the car.
- Credentials stay out of the database. The replica reads its user and
password from the environment each time it connects, and never stores them.
bolt+s://checks the source's certificate, and you can trust your own certificate authority withGDB_REPLICA_SOURCE_CA_FILE. CALL gdb.replication.followers()on the source shows who's following and how far behind they are.
Building this turned up a bug in RESYNC that dated back to 1.4.0. A replica
that was copied again kept its old transaction numbering, and could skip the
first transaction that followed. That's fixed for every replica, same-server
or not. If it can't line up exactly with the source, a copy now fails loudly
instead of skipping quietly.
The replicas guide
has the setup. The server being copied needs an Enterprise licence; the
replicas themselves are happy on Commercial. Both servers need
GDB_REPLICATION=on. The server is about 0.3 MB bigger for it:
the Bolt client a replica uses, and the list of certificate authorities it
trusts. We've decided
we can live with that.
Elsewhere in the galaxy
- Galactus Cloud is in beta. If you'd rather not run a server at all, we'll run one for you: a dedicated machine with its own TLS hostname, daily backups you can download, an optional IP allowlist, and six regions at the same price. Ordering takes a few minutes, and you connect with any Bolt driver or with Explorer, with the login already filled in (that's where 1.6.3 earned its keep).
- There's an Enterprise licence. It's everything in Commercial, plus replicas on other servers and support from the people who wrote it. See pricing, or get in touch.
Upgrading
Pull and restart:
docker pull ianknowles/galactus-db:1.6.5
The data, commit log, snapshot and dump formats haven't changed since 1.6.1,
so any 1.6 release opens the others' databases, and there's nothing to
migrate. Remote replicas are off until you set GDB_REPLICATION=on.
What's next
Replicas on another server are the first step towards Nebula: a cluster of Galactus DB servers that keeps running when one of them doesn't. One server fails, another takes over, and nobody has to get up at 3am.
Four releases in four days. We're going to have a cup of tea now.