Skip to main content

Troubleshooting

Symptom, cause, fix. The full error catalogue is in the error codes reference.

The server will not start​

The quoted messages are logged at boot, before the ports open, and the process exits.

Direct I/O (O_DIRECT) is not enforced on this filesystem. The --data-root filesystem silently turns O_DIRECT into buffered I/O, which would void the durability guarantee. Encrypted filesystems are a common cause. Put the data root on ext4 or XFS on a real block device. A variant says a 512-byte aligned write failed: that filesystem needs larger alignment than the engine uses.

Immutable configuration in server_meta.toml does not match current settings. A setting fixed at first boot changed. The message names each one, saved against configured. Put the original value back. The usual culprit is an unset --num-shards on a host with a different core count. See configuration.

compression.dictionary_sha256 mismatch. dictionary.zstd_dict in the data root is not the file the WAL was written with. Restore the original. If the file is missing and the dictionary is the built-in one, the server rewrites it itself.

compaction_temp_dir ... is on a different filesystem than data_root. Compaction swaps segments with an atomic rename(2), which cannot cross devices. Put --compaction-temp-dir on the same filesystem, or leave it unset.

Port ... is already in use. Another server, often a previous instance, holds the client or replication port.

S3 storage unavailable at startup. Wrong endpoint, bad credentials, or a missing bucket. Credentials come only from --s3-access-key-id / --s3-secret-access-key or the EC2 instance profile; AWS_* env vars are ignored.

Client authentication is configured but TLS is not enabled. api_keys.toml exists in the data root or --require-client-identity is set, and TLS is off. Enable TLS. --insecure-allow-plaintext-auth bypasses the check for development.

Kernel kTLS support check failed. TLS mode needs kernel TLS. Load the tls module or use a kernel built with CONFIG_TLS.

The container exits at once. The storage engine uses io_uring, which the default Docker seccomp profile blocks. Run with --security-opt seccomp=unconfined.

A boot warning that the clock is not NTP-synchronized does not stop the server. Fix it anyway; the cluster depends on synced clocks.

Writes are rejected​

NotLeader (2011). The write reached the follower, or a node still catching up. The error carries the leader's --advertised-client-address. The pooled clients follow it; if they then fail to connect, that address is wrong for your network. Set it explicitly.

WriteReplicationBackpressure (2012). The leader is shedding writes to protect the replication path. celeriant_writes_rejected_backpressure_total says why, in its cause label:

  • inflight_pressure: un-replicated bytes reached --internode-max-request-size. The follower's disk or network is slower than the write rate, or it is mid catch-up after a restart. Clears when it catches up. Chronic means the follower is undersized.
  • rollback_cooldown: replication to both the follower and S3 failed, the leader rolled back, and it refuses writes for --replication-rollback-cooldown-us. Look at S3 and the follower, not the client.
  • follower_heartbeat_starved: a heartbeat has been in flight longer than --heartbeat-starve-threshold-ms, usually a saturated NIC. The leader holds writes so the ack can land.

Back off and retry in all three cases.

OptimisticConcurrencyViolation (2003). Not an operational fault. Another writer moved the aggregate past your expected version. Re-read and retry. See optimistic concurrency.

WatchTooManySubscribers (8005). The watch's shard is at --max-watch-subscribers. Close idle watches or raise the cap.

Writes fail after the disk fills. celeriant_rotation_out_of_space_total rises. The shard stays up and reads keep working, but any write that needs a new WAL segment fails until space is freed.

The cluster is unhealthy​

No leader (celeriant_node_role sums to 0). Nobody holds the lease. Check S3 reachability and credentials from both nodes, and that the store supports conditional writes. By default S3 lease and election calls retry an unreachable S3 forever, so a long outage stalls election without an error. --s3-retry-max-duration-secs caps the retries; a failed attempt then shows in the logs (S3 retry budget exhausted) and in celeriant_election_challenge_failed_total, and the node retries on its next cycle.

Flapping leadership. celeriant_node_role_transitions_total climbing. The usual cause is clock skew. On every heartbeat the follower compares the leader's timestamp with its own clock; past --max-clock-drift-ms it fences all of its shards at once. celeriant_clock_drift_ms shows the gap it measured. Fix NTP on both hosts. Next suspects: heartbeat failures from a saturated or lossy link (celeriant_heartbeat_failures_total), and advertised addresses the other node cannot reach.

Two clusters in one bucket. Without distinct --s3-subfolder values they share one lease object and one two-slot membership record. Give each cluster its own subfolder.

Sustained S3 fallbacks. celeriant_replication_s3_fallbacks_total rising steadily means the leader cannot reach the follower over TCP. Treat it as a down follower: check the replication port, TLS between the nodes (the intracluster CA, if you split trust), and the follower's own logs.

A replacement node never receives data. It started with an empty data root, so it generated a new private_key and a new node ID. The membership record in S3 holds two nodes and nothing removes the old one, so the leader keeps replicating toward the dead node. Start the replacement with the old node's private_key. See backup and recovery.

Clients cannot connect or authenticate​

Identity handshake errors (10001 to 10004). IdentifyInvalidNonce (10001): an expired or malformed nonce, usually client clock skew. IdentifyInvalidSignature (10002): the signature did not verify against the public key. IdentifyMismatch (10003): the clientId in a write does not match the identified client. IdentifyRequired (10004): the server runs with --require-client-identity and the client sent no identity. See identity.

TLS handshake failures. celeriant_tls_handshake_failures_total rises on the client port. Under --tls-mode strict the client must speak TLS, and under --tls-client-auth require it must present a certificate signed by --tls-ca-cert. With --tls-intracluster-ca-cert set, client certificates must come from the client CA, not the intracluster one.

When in doubt​

Raise --log-level to debug (or set RUST_LOG), watch the metrics, and reproduce on the deploy/local-cluster stack, where killing nodes is safe.