Servers
On this page
- Adding a server
- Taking a server out
- When a server is lost
- How the cluster keeps itself in order
- Watching and steering it
- When a database has lost most of its copies
Servers are added and taken out while the cluster serves. Each statement below is sent to any server.
Adding a server
-
Start the new server with the cluster's settings: its own
GDB_NODE_ID, andGDB_CLUSTER_PEERSnaming every server, itself included. LeaveGDB_CLUSTER_INIToff. -
Tell the cluster where it listens:
CALL gdb.cluster.addNode(4, 'd.example:7688');The server is now
joined. It is given the cluster's users and its record of databases, and serves every database by carrying requests to the servers that hold them. -
Put it to use:
ENABLE SERVER 4;Copies are moved onto it until the servers are even, and new databases may be placed on it. Every database goes on serving while its copy moves.
Taking a server out
DRAIN SERVER 2. Each copy it holds is made on another server first, and only then removed from it.SHOW SERVERSshows how many it still holds. It goes on serving meanwhile.DROP SERVER 2, once it holds none.- Stop it.
When a server is lost
| When | What happens |
|---|---|
| At once | Each database it led is led by another copy. Applications that were connected to it connect again, to another server or to the router |
| After 10 seconds without a word | It is shown as down and applications are no longer sent to it. A database with three copies goes on serving with two |
| If it returns within ten minutes | It catches up, and nothing else happens |
| After ten minutes | Each copy it held is made again on another server in use that does not already hold that database, and the lost copy is taken out |
| If it returns after that | It holds nothing and is given copies like a new server |
Ten minutes is the grace period, GDB_CLUSTER_DOWN_GRACE_MS. A cluster of exactly three
servers keeps each database on all three, so a lost server's copies wait for it to return
or for a fourth server to be put to use.
How the cluster keeps itself in order
The cluster's record of its servers and databases is itself a replicated database, and whichever server leads it looks after the rest:
- It places each new database on the servers that hold the fewest.
- It repairs. A database with fewer copies than it asks for is given another.
- It evens out. When one server holds at least two more copies than another, a copy is moved.
- It drains a server that is being taken out.
It works with care for what is running:
- One change is made to a database at a time, and a server is sent one copy at a time.
- A copy is always added before one is removed.
- A leader hands leadership on before its copy goes.
- A copy is never removed if that would leave the database without a majority of copies that are up.
If the server doing this is lost, the one that takes over the record carries on from where things stood. There is no separate service to install or keep alive, and a database fails over without it.
Watching and steering it
| Statement or procedure | Does |
|---|---|
SHOW SERVERS | Each server: node, state (joined, enabled, draining, down or removed), address, clientAddress, the copies it holds, self |
CALL gdb.cluster.status() | One row: servers, servers down, databases, databases with fewer copies than they ask for, changes under way, whether the cluster is paused |
CALL gdb.cluster.operations() | Each change to placement under way: the database, what is being done, from and to which server, why, and what it is waiting for |
CALL gdb.cluster.rebalance({dryRun: true}) | What the cluster would begin now, without beginning it |
CALL gdb.cluster.rebalance() | Begins one round of changes now, paused or not |
CALL gdb.cluster.pause() / resume() | Stops and starts the cluster beginning changes to placement. Changes under way finish |
When a database has lost most of its copies
A database with three copies serves with two. With one left it takes no writes, because one copy is not a majority.
-
If the lost servers will return, wait. The database resumes as soon as two of its three copies are up, with nothing lost.
-
If they will not, begin the database again from a copy that is left:
CALL gdb.cluster.recoverGroup('orders', [1], 'orders');The list names the servers that still hold a copy; the one whose copy holds the most is used. The database takes writes again at once, and the cluster makes it up to the number of copies it asks for.
This is the one operation that can drop writes that were acknowledged: a write is
acknowledged once two of three copies hold it, and those two may be the ones that were
lost. So it asks for the database's name a second time, says how far the best copy goes
when the name is left out, and is refused while the database still has a leader. The
cluster never does it by itself. Keep a majority of the servers that hold system.