Monitoring
On this page
Each server reports on its own copies through the endpoint it already has. Set
GDB_METRICS_BIND on every server, as for a single server.
Metrics
/metrics adds these for each replicated database:
| Metric | Meaning |
|---|---|
gdb_cluster_node | This server's node id |
gdb_cluster_leader{database} | Node id of the leader as this server knows it; 0 when it knows none |
gdb_cluster_is_leader{database} | 1 on the server that leads |
gdb_cluster_term{database} | Goes up by at least one each time leadership changes |
gdb_cluster_commit_lsn{database} | How far the cluster has committed, as this server knows |
gdb_cluster_applied_lsn{database} | How far this server's copy has applied. Behind the commit position means it is catching up |
gdb_cluster_member_failed{database} | 1 when this server's copy has stopped |
gdb_cluster_log_syncs{database} | Disk syncs of this server's log since start. Under load one sync carries several commits |
Three things are worth an alert:
gdb_cluster_leaderat 0 for more than a few seconds: a database has no leader.gdb_cluster_termrising steadily: leadership keeps changing, which points at the network or a disk.gdb_cluster_commit_lsnpulling away fromgdb_cluster_applied_lsnon one server: its copy is falling behind.
Readiness
/ready answers 503 while a replicated database on this server has no known leader or its
copy has stopped, and while the server has been out of touch with the cluster for
GDB_CLUSTER_STALE_MS. A load balancer in front of the servers then sends applications
only to those that can serve them. The router uses the same check.
The list of servers
/cluster/servers lists the cluster's servers, one to a line: node id, state, where
applications connect and where the server answers /ready, separated by tabs, with - for
an address a server has not given.
From a session
SHOW SERVERS;
CALL gdb.cluster.status();
CALL gdb.cluster.operations();
CALL gdb.cluster.members('orders');
gdb.cluster.status() is the one row to watch: servers, servers down, databases, databases
with fewer copies than they ask for, changes under way, and whether the cluster is paused.