# Start a cluster

[Wiki](../README.md) / [Nebula Cluster](README.md)

**On this page**

- [What each server needs](#what-each-server-needs)
- [Settings](#settings)
- [The first start](#the-first-start)
- [Timing](#timing)

A cluster is formed from servers that are given the same description of it. One of them is
named as where the cluster starts, and its users and data become the cluster's.

## What each server needs

- **An odd number of servers.** Three survive the loss of one, five the loss of two.
- **A disk of its own** for its data directory (`GDB_DATA`). A commit waits for more than
  half the servers to write it to disk, so servers that share a disk slow each other down
  and can be lost together.
- **A certificate.** Run a certificate authority for the cluster and sign one certificate
  and key for each server. Each certificate carries the DNS name
  `node-<the node id as 16 hexadecimal digits>.nebula.internal`, for example
  `node-0000000000000002.nebula.internal` for node 2. A server accepts another only when
  its certificate is from that authority and names the node it claims to be.
- **A port for the other servers**, 7688 by default. Applications never use it.

Keys and certificates are read from files at start. Mount them at run time and keep them
out of images.

## Settings

Set these in every server's environment. All but the node id and the certificate files are
the same on every server.

| Variable | Default | Meaning |
|---|---|---|
| `GDB_CLUSTER` | `false` | `true` turns clustering on |
| `GDB_CLUSTER_ID` | | The cluster's id, up to 32 hexadecimal digits. A server refuses messages from any other cluster |
| `GDB_NODE_ID` | | This server's id: a number from 1, different on every server |
| `GDB_CLUSTER_BIND` | `0.0.0.0:7688` | Where this server listens for the others |
| `GDB_CLUSTER_PEERS` | | Every server and where it listens, this one included: `1=a.example:7688,2=b.example:7688,3=c.example:7688` |
| `GDB_CLUSTER_DATABASES` | none | The databases the cluster begins with, by name (`galactus,orders`). One that does not exist is created. `system`, which holds users and roles, is always replicated and is not listed |
| `GDB_CLUSTER_INIT` | `false` | `true` on **one** server: the one the cluster starts from |
| `GDB_CLUSTER_TLS_CERT` | | This server's certificate, PEM |
| `GDB_CLUSTER_TLS_KEY` | | Its private key, PEM |
| `GDB_CLUSTER_TLS_CA` | | The cluster authority's certificate, PEM |
| `GDB_ADVERTISED_ADDRESS` | the Bolt bind address | Where applications reach this server. Set it to a name they can resolve |

A server's environment looks like this:

```text
GDB_CLUSTER=true
GDB_CLUSTER_ID=5157
GDB_NODE_ID=1
GDB_CLUSTER_PEERS=1=a.example:7688,2=b.example:7688,3=c.example:7688
GDB_CLUSTER_DATABASES=orders
GDB_CLUSTER_INIT=true
GDB_CLUSTER_TLS_CERT=/run/secrets/node.crt
GDB_CLUSTER_TLS_KEY=/run/secrets/node.key
GDB_CLUSTER_TLS_CA=/run/secrets/ca.crt
GDB_ADVERTISED_ADDRESS=a.example:7687
```

## The first start

Choose the server whose users and data the cluster should begin with and set
`GDB_CLUSTER_INIT=true` on it. Start all the servers.

- That server's users and roles become the cluster's, on every server.
- A database named in `GDB_CLUSTER_DATABASES` that already holds data is kept on that
  server and copied to the others. On every other server it must be empty: a server that
  holds different data for it stops at start and says so, so that two servers holding
  different data are never taken for copies of each other.
- Once the cluster has formed, the setting does nothing more. It can stay or be removed.
- A server without the setting never offers its own users to a cluster. One that lost its
  disk comes back with nothing, is given the cluster's users by another server, and brings
  no default credentials with it.

Check the cluster from any server:

```cypher
SHOW SERVERS;
SHOW PLACEMENT;
CALL gdb.cluster.status();
```

## Timing

The defaults suit servers in one region. Change them on every server alike.

| Variable | Default | Meaning |
|---|---|---|
| `GDB_CLUSTER_HEARTBEAT_MS` | `200` | How often a leader tells the others it is there |
| `GDB_CLUSTER_ELECTION_MS` | `1000` | Silence from the leader before another server may stand. The actual limit is random between this and twice this, and at least three times the heartbeat |
| `GDB_CLUSTER_COMMIT_TIMEOUT_MS` | `10000` | How long a commit waits for more than half the servers, and how long a read waits to be made current |
| `GDB_CLUSTER_REPORT_TIMEOUT_MS` | `10000` | A server not heard from for this long is marked down |
| `GDB_CLUSTER_DOWN_GRACE_MS` | `600000` | How long a server may be down before its copies are made again on other servers. At most an hour. Read by the server the cluster starts from |
| `GDB_CLUSTER_AUTO_ENABLE` | `false` | `true` puts a server to use as soon as it joins. Read by the server the cluster starts from |
| `GDB_CLUSTER_STALE_MS` | `30000` | A server out of touch with the cluster for this long takes no new logins and answers `/ready` with 503 |

## Related articles

[Connect applications](connect.md) · [Servers](servers.md) · [Server configuration](../operations/configuration.md)
