Replica databases

Wiki / Operations

On this page

A replica is a read-only copy of another database: on the same instance from version 1.4.0, and on another instance from version 1.6.5. It is kept up to date from the source's change stream a few milliseconds after each commit. Use it to serve reads and reports, as a standby, or, with applyDelay, as a delayed copy that protects against an accidental mass delete.

Creating a replica

The source must have its change stream on, which needs GDB_CHANGE_STREAMS=on:

ALTER DATABASE orders SET CHANGE STREAM ON;
CREATE DATABASE orders_ro AS REPLICA OF DATABASE orders;
CREATE DATABASE orders_standby AS REPLICA OF DATABASE orders
  OPTIONS {mode: 'backup', applyDelay: '1h'};
OptionDefaultMeaning
mode'readonly''readonly' or 'backup'. Both serve reads under normal permissions; backup marks a standby
applyDelaynoneApply each source transaction only once it is this old

Creating a replica takes a consistent copy of the source at a known transaction, made durable at the source first, then follows from exactly that transaction. It needs CREATE DATABASE and ALTER DATABASE on the source, and it counts towards the edition's database limit.

Replicas on another instance

From version 1.6.5 a replica can follow a database on another Galactus DB server over Bolt. Turn replication on for both instances, and give the replica's instance a user on the source to connect as:

# source
GDB_CHANGE_STREAMS: "on"
GDB_REPLICATION: "on"

# replica
GDB_REPLICATION: "on"
GDB_REPLICA_SOURCE_USER: replicator
GDB_REPLICA_SOURCE_PASSWORD: ${REPLICATOR_PASSWORD}

Then turn on the source database's change stream and create the replica on the other instance, naming the source by its address and database:

// on the source
ALTER DATABASE orders SET CHANGE STREAM ON;

// on the replica's instance
CREATE DATABASE orders_ro AS REPLICA OF 'bolt+s://primary.example.com:7687/orders';
CREATE DATABASE orders_standby AS REPLICA OF 'bolt+s://primary.example.com:7687/orders'
  OPTIONS {mode: 'backup', applyDelay: '1h'};
  • Addresses: bolt:// is unencrypted, bolt+s:// checks the source's certificate against the public certificate authorities, and bolt+ssc:// accepts a self-signed certificate. For a certificate from your own authority, set GDB_REPLICA_SOURCE_CA_FILE to its PEM file and use bolt+s://.
  • Credentials: read from the environment each time the replica connects and never stored with the database. The source user needs the same privileges as a backup (gdb.dump and gdb.dumpChunk).
  • Licence: the source instance needs an Enterprise licence. The replica's instance needs only a Commercial licence, and the replica counts towards its database limit.
  • No pause at the source: the first copy uses the same snapshot as backups, so the source's readers and writers carry on while it is taken.
  • Catching up: the source keeps its change stream from each replica's last applied transaction, so a paused or disconnected replica catches up without a RESYNC. It lets go when the replica is promoted or dropped, or when the replica has not read for GDB_REPLICATION_HOLD_MS (24 hours by default). A replica that falls further behind stops with an error asking for a RESYNC.
  • On the source, CALL gdb.replication.followers() lists the replicas following the current database: their id, address, applied transaction and how long ago they last read.

Everything else, from SHOW REPLICAS to PROMOTE, works the same as for a replica on the same instance.

How it follows

  • Same transaction numbers: each source transaction is written to the replica's own commit log under the same transaction number and applied the way crash recovery applies it. The replica's position is its last transaction number, so no transaction is ever applied twice.
  • Identical copy: the replica is the same graph, with the same ids, indexes and constraints. A test checks that the replica's dump is byte-identical to the source's.
  • How quickly: a replica only applies transactions that are durable at the source, so it can never be ahead of what the source has safely on disk. With group or sync durability it follows within a few milliseconds (measured p50 4 ms in a Linux container, including one Bolt round trip). With buffered durability, the Docker image's default, it follows the periodic sync, every GDB_SYNC_INTERVAL_MS (50 ms by default), so about 60 ms.
  • Read-only: every write to a replica fails with Galactus.ClientError.Database.ReadOnlyReplica, including schema changes, imports and gdb.restore.
  • Retention: while a replica follows, the source's stream keeps everything the replica has not yet applied.

Managing replicas

SHOW REPLICAS;
ALTER DATABASE orders_ro PAUSE REPLICATION;
ALTER DATABASE orders_ro RESUME REPLICATION;
ALTER DATABASE orders_ro RESYNC;     // re-copy from the source as it is now
ALTER DATABASE orders_ro PROMOTE;    // stop following, become writable
DROP DATABASE orders_ro;

SHOW REPLICAS reports each replica's state (following, paused, diverged, failed or stopped), its applied and the source's latest transaction numbers, the lag in transactions, the apply delay, the commit time of the last applied transaction, and any error.

Restarts, restores and failures

  • Restarts: after a restart a replica is read-only immediately and resumes from its last applied transaction.
  • Source restores: if the source starts a new stream history (a restore, or its stream turned off and on), the replica stops as diverged and keeps serving its last consistent state. RESYNC copies the source again.
  • Errors: an error while following is shown in SHOW REPLICAS and retried. It never affects the source.
  • Promotion: PROMOTE turns a replica into an ordinary database. It stops following and accepts writes. From version 2.5.1, new nodes and relationships take the space of what the source deleted while the replica followed. For a database promoted on 2.5.0 or earlier, run CALL gdb.compact() once to reuse that space.

Limits in this release

  • No change stream of its own: a replica cannot yet have one, so cascading replicas, and subscriptions on a replica, are not supported.
  • Driver routing: replicas are not advertised to drivers' routing. Address a replica by its database name.

Change streams · Subscriptions · Backup, restore and transfer

Planning a deployment? Review compatibility and licence setup for your instance.