Skip to main content

Monitoring and observability

Each node serves Prometheus metrics and a health endpoint on --metrics-port (default 9090). On by default.

  • GET /metrics: Prometheus text format.
  • GET /health: always 200 with {"status":"ok","node_id":"...","shards":N}. It says the process is up and the HTTP server answers. It does not say the node is leader, caught up, or accepting writes. Use it for liveness, not readiness.
Metrics port is open

The metrics server is plaintext HTTP with no auth, and it binds 0.0.0.0 whatever --listen-address says. Firewall it or keep it on a private network.

Many metrics carry a shard_id label. A metric shows up in /metrics only after its first update, so a counter that has never fired is absent, not zero.

What to watch​

The server registers 182 metric descriptions. These are the ones worth a dashboard.

Who is leader

  • celeriant_node_role: 1 on the leader (or a standalone node), 0 otherwise. Reported from shard 0. Summed across a two-node cluster it should be exactly 1.
  • celeriant_node_status_effective_code: per-shard state with the lease clock applied. 0 BootCatchup, 1 Follower, 2 FollowerCatchingUp, 3 Promoting, 4 Leader, 5 Fenced, 6 Standalone. Catches a leader whose lease has lapsed while its raw status still says Leader.
  • celeriant_node_role_transitions_total (label to): actual leader/follower flips. Should be flat.
  • celeriant_leader_elections_total: counts every S3 lease CAS round, renewals included, not only handovers. Flat while heartbeats succeed, because steady state never touches S3. Movement means the nodes lost contact.

Is the peer link healthy

  • celeriant_heartbeat_failures_total, celeriant_heartbeat_kernel_blocked_total: failed heartbeats and heartbeats cut by the hard timeout.
  • celeriant_heartbeat_duration_seconds: a late ack shows up here before it becomes a failure.
  • celeriant_follower_auto_fence_total: the follower fenced itself.
  • celeriant_clock_drift_ms: observed drift between the nodes. Compare it with --max-clock-drift-ms.

Is replication keeping up

  • celeriant_replication_s3_fallbacks_total: each increment is a batch that went to S3 instead of the follower. Sustained growth means the follower is effectively gone.
  • celeriant_replication_queue_bytes against celeriant_replication_queue_high_water_bytes: replication backlog and the threshold where it spills to S3.
  • celeriant_writes_rejected_backpressure_total (label cause): writes refused with 2012. cause is inflight_pressure, rollback_cooldown, or follower_heartbeat_starved.
  • celeriant_wal_seq and celeriant_read_wal_seq: durable tip and committed read cursor. Compare celeriant_wal_seq per shard across the two nodes for lag.

Latency and throughput

  • celeriant_write_duration_seconds: shard-internal write latency. Excludes the wire read, response write, and cross-shard redirect.
  • celeriant_read_duration_seconds: end-to-end read latency.
  • celeriant_fsync_duration_seconds, celeriant_replication_duration_seconds: where write latency goes.
  • celeriant_writes_total, celeriant_reads_total, celeriant_write_errors_total and celeriant_read_errors_total (label error_code).

Resources and stability

  • celeriant_client_connections_active (label port_type), celeriant_watch_subscribers_active.
  • celeriant_cache_recent_write_hits_total / _misses_total, celeriant_cache_aggregate_snapshot_hits_total / _misses_total, celeriant_cache_log_file_hits_total / _misses_total. A falling hit rate means the working set has outgrown the memory budget.
  • celeriant_rotation_out_of_space_total: a WAL rotation hit ENOSPC. The shard stays up, but writes that need a new segment fail.
  • celeriant_shard_panics_total, celeriant_shard_restarts_total: should be absent or zero.
  • celeriant_tls_handshake_failures_total: failed or timed-out handshakes per port.

A starting alert set​

  • sum(celeriant_node_role) across the cluster is not 1 for longer than --heartbeat-lease-duration-ms plus a scrape interval.
  • increase(celeriant_node_role_transitions_total[10m]) > 0.
  • rate(celeriant_replication_s3_fallbacks_total[5m]) > 0 for more than a few minutes.
  • celeriant_clock_drift_ms approaching --max-clock-drift-ms.
  • Write p99 from celeriant_write_duration_seconds over your SLO.
  • Any increase in celeriant_shard_panics_total or celeriant_rotation_out_of_space_total.

Grafana dashboard​

deploy/local-cluster in the server repo runs two nodes with Prometheus, Loki, Grafana, and MinIO. Its dashboard, grafana/dashboards/celeriant-cluster.json, is a working starting point: node role, elections, clock drift, replication saturation, latency percentiles, cache hit rates, and per-shard drill-down. Import it, then point the instance variable at your own scrape targets.