Monitoring

Wiki / Nebula Cluster

On this page

Each server reports on its own copies through the endpoint it already has. Set GDB_METRICS_BIND on every server, as for a single server.

Metrics

/metrics adds these for each replicated database:

MetricMeaning
gdb_cluster_nodeThis server's node id
gdb_cluster_leader{database}Node id of the leader as this server knows it; 0 when it knows none
gdb_cluster_is_leader{database}1 on the server that leads
gdb_cluster_term{database}Goes up by at least one each time leadership changes
gdb_cluster_commit_lsn{database}How far the cluster has committed, as this server knows
gdb_cluster_applied_lsn{database}How far this server's copy has applied. Behind the commit position means it is catching up
gdb_cluster_member_failed{database}1 when this server's copy has stopped
gdb_cluster_log_syncs{database}Disk syncs of this server's log since start. Under load one sync carries several commits

Three things are worth an alert:

  • gdb_cluster_leader at 0 for more than a few seconds: a database has no leader.
  • gdb_cluster_term rising steadily: leadership keeps changing, which points at the network or a disk.
  • gdb_cluster_commit_lsn pulling away from gdb_cluster_applied_lsn on one server: its copy is falling behind.

Readiness

/ready answers 503 while a replicated database on this server has no known leader or its copy has stopped, and while the server has been out of touch with the cluster for GDB_CLUSTER_STALE_MS. A load balancer in front of the servers then sends applications only to those that can serve them. The router uses the same check.

The list of servers

/cluster/servers lists the cluster's servers, one to a line: node id, state, where applications connect and where the server answers /ready, separated by tabs, with - for an address a server has not given.

From a session

SHOW SERVERS;
CALL gdb.cluster.status();
CALL gdb.cluster.operations();
CALL gdb.cluster.members('orders');

gdb.cluster.status() is the one row to watch: servers, servers down, databases, databases with fewer copies than they ask for, changes under way, and whether the cluster is paused.

Monitoring a server · Servers · Connect applications

Planning a deployment? Review compatibility and licence setup for your instance.