# Monitoring

[Wiki](../README.md) / [Nebula Cluster](README.md)

**On this page**

- [Metrics](#metrics)
- [Readiness](#readiness)
- [The list of servers](#the-list-of-servers)
- [From a session](#from-a-session)

Each server reports on its own copies through the endpoint it already has. Set
`GDB_METRICS_BIND` on every server, as for [a single server](../operations/monitoring.md).

## Metrics

`/metrics` adds these for each replicated database:

| Metric | Meaning |
|---|---|
| `gdb_cluster_node` | This server's node id |
| `gdb_cluster_leader{database}` | Node id of the leader as this server knows it; 0 when it knows none |
| `gdb_cluster_is_leader{database}` | 1 on the server that leads |
| `gdb_cluster_term{database}` | Goes up by at least one each time leadership changes |
| `gdb_cluster_commit_lsn{database}` | How far the cluster has committed, as this server knows |
| `gdb_cluster_applied_lsn{database}` | How far this server's copy has applied. Behind the commit position means it is catching up |
| `gdb_cluster_member_failed{database}` | 1 when this server's copy has stopped |
| `gdb_cluster_log_syncs{database}` | Disk syncs of this server's log since start. Under load one sync carries several commits |

Three things are worth an alert:

- `gdb_cluster_leader` at 0 for more than a few seconds: a database has no leader.
- `gdb_cluster_term` rising steadily: leadership keeps changing, which points at the
  network or a disk.
- `gdb_cluster_commit_lsn` pulling away from `gdb_cluster_applied_lsn` on one server: its
  copy is falling behind.

## Readiness

`/ready` answers 503 while a replicated database on this server has no known leader or its
copy has stopped, and while the server has been out of touch with the cluster for
`GDB_CLUSTER_STALE_MS`. A load balancer in front of the servers then sends applications
only to those that can serve them. The [router](connect.md#the-router) uses the same check.

## The list of servers

`/cluster/servers` lists the cluster's servers, one to a line: node id, state, where
applications connect and where the server answers `/ready`, separated by tabs, with `-` for
an address a server has not given.

## From a session

```cypher
SHOW SERVERS;
CALL gdb.cluster.status();
CALL gdb.cluster.operations();
CALL gdb.cluster.members('orders');
```

`gdb.cluster.status()` is the one row to watch: servers, servers down, databases, databases
with fewer copies than they ask for, changes under way, and whether the cluster is paused.

## Related articles

[Monitoring a server](../operations/monitoring.md) · [Servers](servers.md) · [Connect applications](connect.md)
