# Servers

[Wiki](../README.md) / [Nebula Cluster](README.md)

**On this page**

- [Adding a server](#adding-a-server)
- [Taking a server out](#taking-a-server-out)
- [When a server is lost](#when-a-server-is-lost)
- [How the cluster keeps itself in order](#how-the-cluster-keeps-itself-in-order)
- [Watching and steering it](#watching-and-steering-it)
- [When a database has lost most of its copies](#when-a-database-has-lost-most-of-its-copies)

Servers are added and taken out while the cluster serves. Each statement below is sent to
any server.

## Adding a server

1. Start the new server with the cluster's settings: its own `GDB_NODE_ID`, and
   `GDB_CLUSTER_PEERS` naming every server, itself included. Leave `GDB_CLUSTER_INIT` off.
2. Tell the cluster where it listens:

   ```cypher
   CALL gdb.cluster.addNode(4, 'd.example:7688');
   ```

   The server is now `joined`. It is given the cluster's users and its record of
   databases, and serves every database by carrying requests to the servers that hold
   them.
3. Put it to use:

   ```cypher
   ENABLE SERVER 4;
   ```

   Copies are moved onto it until the servers are even, and new databases may be placed
   on it. Every database goes on serving while its copy moves.

## Taking a server out

1. `DRAIN SERVER 2`. Each copy it holds is made on another server first, and only then
   removed from it. `SHOW SERVERS` shows how many it still holds. It goes on serving
   meanwhile.
2. `DROP SERVER 2`, once it holds none.
3. Stop it.

## When a server is lost

| When | What happens |
|---|---|
| At once | Each database it led is led by another copy. Applications that were connected to it connect again, to another server or to the router |
| After 10 seconds without a word | It is shown as `down` and applications are no longer sent to it. A database with three copies goes on serving with two |
| If it returns within ten minutes | It catches up, and nothing else happens |
| After ten minutes | Each copy it held is made again on another server in use that does not already hold that database, and the lost copy is taken out |
| If it returns after that | It holds nothing and is given copies like a new server |

Ten minutes is the grace period, `GDB_CLUSTER_DOWN_GRACE_MS`. A cluster of exactly three
servers keeps each database on all three, so a lost server's copies wait for it to return
or for a fourth server to be put to use.

## How the cluster keeps itself in order

The cluster's record of its servers and databases is itself a replicated database, and
whichever server leads it looks after the rest:

- **It places** each new database on the servers that hold the fewest.
- **It repairs.** A database with fewer copies than it asks for is given another.
- **It evens out.** When one server holds at least two more copies than another, a copy is
  moved.
- **It drains** a server that is being taken out.

It works with care for what is running:

- One change is made to a database at a time, and a server is sent one copy at a time.
- A copy is always added before one is removed.
- A leader hands leadership on before its copy goes.
- A copy is never removed if that would leave the database without a majority of copies
  that are up.

If the server doing this is lost, the one that takes over the record carries on from where
things stood. There is no separate service to install or keep alive, and a database fails
over without it.

## Watching and steering it

| Statement or procedure | Does |
|---|---|
| `SHOW SERVERS` | Each server: `node`, `state` (`joined`, `enabled`, `draining`, `down` or `removed`), `address`, `clientAddress`, the `copies` it holds, `self` |
| `CALL gdb.cluster.status()` | One row: servers, servers down, databases, databases with fewer copies than they ask for, changes under way, whether the cluster is paused |
| `CALL gdb.cluster.operations()` | Each change to placement under way: the database, what is being done, from and to which server, why, and what it is waiting for |
| `CALL gdb.cluster.rebalance({dryRun: true})` | What the cluster would begin now, without beginning it |
| `CALL gdb.cluster.rebalance()` | Begins one round of changes now, paused or not |
| `CALL gdb.cluster.pause()` / `resume()` | Stops and starts the cluster beginning changes to placement. Changes under way finish |

## When a database has lost most of its copies

A database with three copies serves with two. With one left it takes no writes, because
one copy is not a majority.

- If the lost servers will return, wait. The database resumes as soon as two of its three
  copies are up, with nothing lost.
- If they will not, begin the database again from a copy that is left:

  ```cypher
  CALL gdb.cluster.recoverGroup('orders', [1], 'orders');
  ```

  The list names the servers that still hold a copy; the one whose copy holds the most is
  used. The database takes writes again at once, and the cluster makes it up to the number
  of copies it asks for.

This is the one operation that can drop writes that were acknowledged: a write is
acknowledged once two of three copies hold it, and those two may be the ones that were
lost. So it asks for the database's name a second time, says how far the best copy goes
when the name is left out, and is refused while the database still has a leader. The
cluster never does it by itself. Keep a majority of the servers that hold `system`.

## Related articles

[Databases and copies](databases.md) · [Monitoring](monitoring.md) · [Back up and restore](backups.md)
