Start a cluster

Wiki / Nebula Cluster

On this page

A cluster is formed from servers that are given the same description of it. One of them is named as where the cluster starts, and its users and data become the cluster's.

What each server needs

  • An odd number of servers. Three survive the loss of one, five the loss of two.
  • A disk of its own for its data directory (GDB_DATA). A commit waits for more than half the servers to write it to disk, so servers that share a disk slow each other down and can be lost together.
  • A certificate. Run a certificate authority for the cluster and sign one certificate and key for each server. Each certificate carries the DNS name node-<the node id as 16 hexadecimal digits>.nebula.internal, for example node-0000000000000002.nebula.internal for node 2. A server accepts another only when its certificate is from that authority and names the node it claims to be.
  • A port for the other servers, 7688 by default. Applications never use it.

Keys and certificates are read from files at start. Mount them at run time and keep them out of images.

Settings

Set these in every server's environment. All but the node id and the certificate files are the same on every server.

VariableDefaultMeaning
GDB_CLUSTERfalsetrue turns clustering on
GDB_CLUSTER_IDThe cluster's id, up to 32 hexadecimal digits. A server refuses messages from any other cluster
GDB_NODE_IDThis server's id: a number from 1, different on every server
GDB_CLUSTER_BIND0.0.0.0:7688Where this server listens for the others
GDB_CLUSTER_PEERSEvery server and where it listens, this one included: 1=a.example:7688,2=b.example:7688,3=c.example:7688
GDB_CLUSTER_DATABASESnoneThe databases the cluster begins with, by name (galactus,orders). One that does not exist is created. system, which holds users and roles, is always replicated and is not listed
GDB_CLUSTER_INITfalsetrue on one server: the one the cluster starts from
GDB_CLUSTER_TLS_CERTThis server's certificate, PEM
GDB_CLUSTER_TLS_KEYIts private key, PEM
GDB_CLUSTER_TLS_CAThe cluster authority's certificate, PEM
GDB_ADVERTISED_ADDRESSthe Bolt bind addressWhere applications reach this server. Set it to a name they can resolve

A server's environment looks like this:

GDB_CLUSTER=true
GDB_CLUSTER_ID=5157
GDB_NODE_ID=1
GDB_CLUSTER_PEERS=1=a.example:7688,2=b.example:7688,3=c.example:7688
GDB_CLUSTER_DATABASES=orders
GDB_CLUSTER_INIT=true
GDB_CLUSTER_TLS_CERT=/run/secrets/node.crt
GDB_CLUSTER_TLS_KEY=/run/secrets/node.key
GDB_CLUSTER_TLS_CA=/run/secrets/ca.crt
GDB_ADVERTISED_ADDRESS=a.example:7687

The first start

Choose the server whose users and data the cluster should begin with and set GDB_CLUSTER_INIT=true on it. Start all the servers.

  • That server's users and roles become the cluster's, on every server.
  • A database named in GDB_CLUSTER_DATABASES that already holds data is kept on that server and copied to the others. On every other server it must be empty: a server that holds different data for it stops at start and says so, so that two servers holding different data are never taken for copies of each other.
  • Once the cluster has formed, the setting does nothing more. It can stay or be removed.
  • A server without the setting never offers its own users to a cluster. One that lost its disk comes back with nothing, is given the cluster's users by another server, and brings no default credentials with it.

Check the cluster from any server:

SHOW SERVERS;
SHOW PLACEMENT;
CALL gdb.cluster.status();

Timing

The defaults suit servers in one region. Change them on every server alike.

VariableDefaultMeaning
GDB_CLUSTER_HEARTBEAT_MS200How often a leader tells the others it is there
GDB_CLUSTER_ELECTION_MS1000Silence from the leader before another server may stand. The actual limit is random between this and twice this, and at least three times the heartbeat
GDB_CLUSTER_COMMIT_TIMEOUT_MS10000How long a commit waits for more than half the servers, and how long a read waits to be made current
GDB_CLUSTER_REPORT_TIMEOUT_MS10000A server not heard from for this long is marked down
GDB_CLUSTER_DOWN_GRACE_MS600000How long a server may be down before its copies are made again on other servers. At most an hour. Read by the server the cluster starts from
GDB_CLUSTER_AUTO_ENABLEfalsetrue puts a server to use as soon as it joins. Read by the server the cluster starts from
GDB_CLUSTER_STALE_MS30000A server out of touch with the cluster for this long takes no new logins and answers /ready with 503

Connect applications · Servers · Server configuration

Planning a deployment? Review compatibility and licence setup for your instance.