Servers

Wiki / Nebula Cluster

On this page

Servers are added and taken out while the cluster serves. Each statement below is sent to any server.

Adding a server

  1. Start the new server with the cluster's settings: its own GDB_NODE_ID, and GDB_CLUSTER_PEERS naming every server, itself included. Leave GDB_CLUSTER_INIT off.

  2. Tell the cluster where it listens:

    CALL gdb.cluster.addNode(4, 'd.example:7688');
    

    The server is now joined. It is given the cluster's users and its record of databases, and serves every database by carrying requests to the servers that hold them.

  3. Put it to use:

    ENABLE SERVER 4;
    

    Copies are moved onto it until the servers are even, and new databases may be placed on it. Every database goes on serving while its copy moves.

Taking a server out

  1. DRAIN SERVER 2. Each copy it holds is made on another server first, and only then removed from it. SHOW SERVERS shows how many it still holds. It goes on serving meanwhile.
  2. DROP SERVER 2, once it holds none.
  3. Stop it.

When a server is lost

WhenWhat happens
At onceEach database it led is led by another copy. Applications that were connected to it connect again, to another server or to the router
After 10 seconds without a wordIt is shown as down and applications are no longer sent to it. A database with three copies goes on serving with two
If it returns within ten minutesIt catches up, and nothing else happens
After ten minutesEach copy it held is made again on another server in use that does not already hold that database, and the lost copy is taken out
If it returns after thatIt holds nothing and is given copies like a new server

Ten minutes is the grace period, GDB_CLUSTER_DOWN_GRACE_MS. A cluster of exactly three servers keeps each database on all three, so a lost server's copies wait for it to return or for a fourth server to be put to use.

How the cluster keeps itself in order

The cluster's record of its servers and databases is itself a replicated database, and whichever server leads it looks after the rest:

  • It places each new database on the servers that hold the fewest.
  • It repairs. A database with fewer copies than it asks for is given another.
  • It evens out. When one server holds at least two more copies than another, a copy is moved.
  • It drains a server that is being taken out.

It works with care for what is running:

  • One change is made to a database at a time, and a server is sent one copy at a time.
  • A copy is always added before one is removed.
  • A leader hands leadership on before its copy goes.
  • A copy is never removed if that would leave the database without a majority of copies that are up.

If the server doing this is lost, the one that takes over the record carries on from where things stood. There is no separate service to install or keep alive, and a database fails over without it.

Watching and steering it

Statement or procedureDoes
SHOW SERVERSEach server: node, state (joined, enabled, draining, down or removed), address, clientAddress, the copies it holds, self
CALL gdb.cluster.status()One row: servers, servers down, databases, databases with fewer copies than they ask for, changes under way, whether the cluster is paused
CALL gdb.cluster.operations()Each change to placement under way: the database, what is being done, from and to which server, why, and what it is waiting for
CALL gdb.cluster.rebalance({dryRun: true})What the cluster would begin now, without beginning it
CALL gdb.cluster.rebalance()Begins one round of changes now, paused or not
CALL gdb.cluster.pause() / resume()Stops and starts the cluster beginning changes to placement. Changes under way finish

When a database has lost most of its copies

A database with three copies serves with two. With one left it takes no writes, because one copy is not a majority.

  • If the lost servers will return, wait. The database resumes as soon as two of its three copies are up, with nothing lost.

  • If they will not, begin the database again from a copy that is left:

    CALL gdb.cluster.recoverGroup('orders', [1], 'orders');
    

    The list names the servers that still hold a copy; the one whose copy holds the most is used. The database takes writes again at once, and the cluster makes it up to the number of copies it asks for.

This is the one operation that can drop writes that were acknowledged: a write is acknowledged once two of three copies hold it, and those two may be the ones that were lost. So it asks for the database's name a second time, says how far the best copy goes when the name is left out, and is refused while the database still has a leader. The cluster never does it by itself. Keep a majority of the servers that hold system.

Databases and copies · Monitoring · Back up and restore

Planning a deployment? Review compatibility and licence setup for your instance.