Start a cluster
On this page
A cluster is formed from servers that are given the same description of it. One of them is named as where the cluster starts, and its users and data become the cluster's.
What each server needs
- An odd number of servers. Three survive the loss of one, five the loss of two.
- A disk of its own for its data directory (
GDB_DATA). A commit waits for more than half the servers to write it to disk, so servers that share a disk slow each other down and can be lost together. - A certificate. Run a certificate authority for the cluster and sign one certificate
and key for each server. Each certificate carries the DNS name
node-<the node id as 16 hexadecimal digits>.nebula.internal, for examplenode-0000000000000002.nebula.internalfor node 2. A server accepts another only when its certificate is from that authority and names the node it claims to be. - A port for the other servers, 7688 by default. Applications never use it.
Keys and certificates are read from files at start. Mount them at run time and keep them out of images.
Settings
Set these in every server's environment. All but the node id and the certificate files are the same on every server.
| Variable | Default | Meaning |
|---|---|---|
GDB_CLUSTER | false | true turns clustering on |
GDB_CLUSTER_ID | The cluster's id, up to 32 hexadecimal digits. A server refuses messages from any other cluster | |
GDB_NODE_ID | This server's id: a number from 1, different on every server | |
GDB_CLUSTER_BIND | 0.0.0.0:7688 | Where this server listens for the others |
GDB_CLUSTER_PEERS | Every server and where it listens, this one included: 1=a.example:7688,2=b.example:7688,3=c.example:7688 | |
GDB_CLUSTER_DATABASES | none | The databases the cluster begins with, by name (galactus,orders). One that does not exist is created. system, which holds users and roles, is always replicated and is not listed |
GDB_CLUSTER_INIT | false | true on one server: the one the cluster starts from |
GDB_CLUSTER_TLS_CERT | This server's certificate, PEM | |
GDB_CLUSTER_TLS_KEY | Its private key, PEM | |
GDB_CLUSTER_TLS_CA | The cluster authority's certificate, PEM | |
GDB_ADVERTISED_ADDRESS | the Bolt bind address | Where applications reach this server. Set it to a name they can resolve |
A server's environment looks like this:
GDB_CLUSTER=true
GDB_CLUSTER_ID=5157
GDB_NODE_ID=1
GDB_CLUSTER_PEERS=1=a.example:7688,2=b.example:7688,3=c.example:7688
GDB_CLUSTER_DATABASES=orders
GDB_CLUSTER_INIT=true
GDB_CLUSTER_TLS_CERT=/run/secrets/node.crt
GDB_CLUSTER_TLS_KEY=/run/secrets/node.key
GDB_CLUSTER_TLS_CA=/run/secrets/ca.crt
GDB_ADVERTISED_ADDRESS=a.example:7687
The first start
Choose the server whose users and data the cluster should begin with and set
GDB_CLUSTER_INIT=true on it. Start all the servers.
- That server's users and roles become the cluster's, on every server.
- A database named in
GDB_CLUSTER_DATABASESthat already holds data is kept on that server and copied to the others. On every other server it must be empty: a server that holds different data for it stops at start and says so, so that two servers holding different data are never taken for copies of each other. - Once the cluster has formed, the setting does nothing more. It can stay or be removed.
- A server without the setting never offers its own users to a cluster. One that lost its disk comes back with nothing, is given the cluster's users by another server, and brings no default credentials with it.
Check the cluster from any server:
SHOW SERVERS;
SHOW PLACEMENT;
CALL gdb.cluster.status();
Timing
The defaults suit servers in one region. Change them on every server alike.
| Variable | Default | Meaning |
|---|---|---|
GDB_CLUSTER_HEARTBEAT_MS | 200 | How often a leader tells the others it is there |
GDB_CLUSTER_ELECTION_MS | 1000 | Silence from the leader before another server may stand. The actual limit is random between this and twice this, and at least three times the heartbeat |
GDB_CLUSTER_COMMIT_TIMEOUT_MS | 10000 | How long a commit waits for more than half the servers, and how long a read waits to be made current |
GDB_CLUSTER_REPORT_TIMEOUT_MS | 10000 | A server not heard from for this long is marked down |
GDB_CLUSTER_DOWN_GRACE_MS | 600000 | How long a server may be down before its copies are made again on other servers. At most an hour. Read by the server the cluster starts from |
GDB_CLUSTER_AUTO_ENABLE | false | true puts a server to use as soon as it joins. Read by the server the cluster starts from |
GDB_CLUSTER_STALE_MS | 30000 | A server out of touch with the cluster for this long takes no new logins and answers /ready with 503 |