Skip to main content

Leader election and S3 leases

Why run Raft for exactly two nodes when an object store is already in the stack for fallback durability? Celeriant elects its leader with S3 conditional writes on one object. No third node, no log matching, no quorum to misconfigure. The price is a dependency on synchronised clocks and on S3 during partitions.

The lease object​

The lease is cluster/lease.json in the bucket (under --s3-subfolder if set). It holds the leader's node id, a lease_epoch, and an expiry. An election reads it and acts:

  • No lease. Create-only PUT with this node as leader, epoch 1. If the peer created it first, become follower.
  • Valid lease held by the peer. Become follower. No write.
  • Expired lease, or one this node already holds. Compare-and-swap on the object's ETag: renew your own at the same epoch, or take the peer's at epoch + 1. If the CAS loses, re-read and follow.

The conditional PUT is the whole safety argument, so the client pins S3's ETag-match mode explicitly rather than trusting a library default. lease_epoch only rises on a handoff and is stamped on every metablock a leader writes, so a superseded leader cannot produce current-epoch data.

Two leases​

  • Heartbeat lease. The leader sends a heartbeat on shard 0 every --heartbeat-interval-ms (500). Each one extends the follower's lease to the leader's timestamp plus --heartbeat-lease-duration-ms (1500). Each ack extends the leader's own lease the same way. This is the live authority.
  • S3 lease. --s3-lease-duration-ms (30000) from the last CAS. The durable record, and the arbiter when the nodes cannot see each other.

In a healthy cluster the leader never touches S3. Not for renewal, not for data. The S3 lease written at election sits there and expires. The leader only CASes S3 when it has to prove authority without the follower:

  1. The heartbeat that first fails after the follower was reachable triggers an immediate renewal, before other shards start uploading fallback batches.
  2. While the follower stays unreachable, it renews again once less than half the S3 lease remains.
  3. An S3 fallback upload is an acknowledgement, so it requires a CAS-confirmed lease less than --s3-lease-duration-ms old. Past half that age the shard asks for a renewal. Past the full age it refuses the upload and waits.

Failover timing​

The follower challenges when its own lease expires. That lease is the later of two things: the last heartbeat's timestamp plus 1500 ms, and the S3 lease expiry it saw at its last election.

  • Warm case. The S3 lease expired long ago. The follower challenges 1500 ms after the last heartbeat, wins the CAS, and promotes. Promotion commits the replicated tail, catches up from S3 and uploads the unconfirmed range before writes open, so the write pause is the heartbeat lease plus that work.
  • Cold case. The S3 lease is still live because it was written recently: a fresh election, or a renewal after a follower blip. The follower sees a valid lease held by the peer and waits for it to expire. Up to --s3-lease-duration-ms.

Lowering --heartbeat-lease-duration-ms detects failure sooner and tolerates less jitter on the replication link. Keep it well below --s3-lease-duration-ms.

A restarted node that finds the peer's expired lease in S3 does not challenge straight away. It waits up to min(--heartbeat-lease-duration-ms, 5000) for a heartbeat. The peer's lease is expired because a healthy leader never renews it, not because the peer is dead.

Fencing and split-brain​

The leader fences --max-clock-drift-ms (500) before its lease expires. The follower challenges at full expiry. With default timings the old leader stops writing 1000 ms after its last successful heartbeat ack; the follower cannot challenge until 1500 ms after it. That gap is the guarantee, and it only holds while the clocks agree. A follower that sees a heartbeat timestamp more than --max-clock-drift-ms from its own clock fences all its shards and rejects the heartbeat. Run NTP; the design doc recommends chrony, which keeps two nodes within about 1 ms.

Behind the fence sit four more checks. Every write checks the node's effective status before it enters the pipeline. The follower rejects replication from a lower epoch with StaleLease, and the leader that receives it fences itself. The leader re-checks its status after replication succeeds and before it acknowledges. And catch-up never truncates a node's own acknowledged history.

The chaos suite tests the zombie case directly: sigstop_leader freezes the leader past its lease, lets the follower promote, resumes the old leader, and checks it demotes without a second writer. See Correctness testing.

When S3 is down​

  • S3 only. Heartbeats keep the leader's lease alive, so it keeps serving and replicating over TCP.
  • S3 and the follower. The leader cannot renew anywhere. It fences when its lease runs out, and writes stop.
  • A long S3 outage stalls elections, because the lease lives in S3. Acknowledged data is already on disk and is not at risk. S3 calls retry with backoff forever unless --s3-retry-max-duration-secs caps them.

What S3 needs​

  • Conditional writes (If-None-Match and If-Match). Non-negotiable. AWS S3 has them. Test any S3-compatible store before trusting it.
  • Permissions to get, put, delete and list objects under the cluster prefix. Catch-up lists and deletes fallback batches. The EC2 reference stack grants read-write on the bucket.
  • One bucket per cluster, or a distinct --s3-subfolder per cluster in a shared bucket.

S3 is coordination and replication fallback here, not a backup. See Backup and recovery.