Running a two-node cluster
A leader and a follower, coordinated through S3. The leader takes writes and replicates each batch to the follower over the replication port; the follower fsyncs before it acks. No third node, no quorum: the S3 lease decides who may write. Two is a hard ceiling. The membership object has two slots, and a third node cannot join.
Ports
| Port | Flag | Default | Carries |
|---|---|---|---|
| Client | --client-port | 10000 | Client connections |
| Replication | --replication-port | 10001 | Leader-to-follower batches and heartbeats |
| Metrics | --metrics-port | 9090 | Prometheus /metrics and /health. See Monitoring. |
Required configuration
Leave out --standalone and turn S3 on. --s3-enabled refuses to parse without --s3-region and --s3-bucket.
celeriant \
--data-root /var/lib/celeriant \
--num-shards 8 \
--s3-enabled \
--s3-region us-east-1 \
--s3-bucket celeriant-prod \
--advertised-client-address 10.0.1.11:10000 \
--advertised-replication-address 10.0.1.11:10001
Run the same on the second node with its own addresses. Replication and S3 fallback both run shard by shard, and nothing cross-checks the two nodes, so give them the same --num-shards and --routing-rule.
There is no node-id flag and no seed list. Each node derives its id from the private_key it generates in --data-root on first boot, registers itself in cluster/membership.json, and finds its peer there.
The advertised addresses are what the node publishes to that membership object, and the client address is what the leader hands back in a NotLeader redirect. They default to {listen_address}:{port}, which is 0.0.0.0:10000 unless you change --listen-address. The peer cannot dial 0.0.0.0, so set both. The project's own deployments use IPs: the Pi cluster notes that its hostnames resolve to 127.0.1.1 locally.
A node started without --standalone and without --s3-enabled has nowhere to register. It never gets past boot and retries membership registration every 2 s.
S3 settings
| Flag | Default | Notes |
|---|---|---|
--s3-access-key-id / --s3-secret-access-key | none | Static credentials. Leave them out on EC2 and the client uses the instance profile, as the EC2 reference stack does. |
--s3-subfolder | none | Single-level prefix. Required when two clusters share a bucket, or they fight over the same cluster/lease.json. |
--s3-endpoint-override | none | For MinIO and other S3-compatible stores. |
--s3-allow-http | false | Plain HTTP to the endpoint. The Pi cluster uses it against MinIO on its LAN. |
--s3-skip-signature | false | Unsigned requests. The local Docker stack uses it with a public MinIO bucket. Never in production. |
--s3-retry-max-duration-secs | unset | Unset retries S3 forever with exponential backoff. Set it to fail fast instead of blocking through a long outage. |
At boot, the server lists the bucket prefix and exits if S3 is unreachable or the bucket does not exist.
A local cluster to try
deploy/local-cluster in the source tree is a Docker Compose stack: two nodes built from the working tree, MinIO standing in for S3, Prometheus, Loki and Grafana.
cd deploy/local-cluster
docker compose up -d --build
Node 1's client port is on localhost:10000, node 2's on localhost:10002. Grafana is on localhost:3001 with a cluster dashboard. Stop the leader's container and watch celeriant_node_role flip. failover-test.sh in the same directory runs a benchmark while it kills a node.
When a node goes down
The follower. The leader keeps taking writes and uploads each batch to S3 in place of the follower. That adds an S3 round-trip plus the --s3-replication-delay-us batching window (500 ms default) to write latency. The follower catches up from S3 when it returns.
The leader. Heartbeats stop. The follower's heartbeat lease runs out after --heartbeat-lease-duration-ms (1500 ms), and it challenges the S3 lease. If that lease has already expired, which it has in a healthy cluster, the CAS wins and the follower promotes. If the lease was renewed in the last --s3-lease-duration-ms (30 s), the follower waits for it to expire. Writes pause for the handoff.
The leader election page has the exact timing. Upgrading has the rolling-restart procedure.