Durability and safety

What Celeriant guarantees and the mechanism behind each claim. Each one is something you can test.

An ack means two durable copies

The leader acknowledges a write only after two things succeed:

1. Its own fdatasync of the batch.
2. Replication: the follower has applied the batch and run its own fdatasync, or, if the follower is unreachable, the leader has uploaded the batch to S3.

If both replication paths fail, the leader rolls back the unreplicated writes and the client gets an error, not an ack. Deletes and trims follow the same contract.

Readers never see a write early. On the leader a batch stays invisible until replication completes; on the follower it waits for the leader to confirm. In a cluster, no read returns a write that has not reached both durable copies.

The degraded path puts an S3 upload on every ack, so write latency rises until the follower rejoins. See Two-node cluster.

Direct I/O, verified at boot

WAL files are written with O_DIRECT, bypassing the page cache. Buffered I/O can report a successful fsync and still lose the data when a later writeback fails. Direct I/O removes the page cache from that path.

Some filesystems accept O_DIRECT and quietly buffer anyway. So on startup the server writes 41 unaligned bytes to a probe file in the data directory. Real Direct I/O rejects that with EINVAL, then a 512-byte aligned write must succeed. Any other result and the process exits.

Each segment file is preallocated to --shard-log-preallocate-bytes (1 GiB by default) when it is created, so appends land in space the filesystem already reserved.

fsync is amortised, not skipped

Every acknowledged write is covered by an fdatasync. What gets shared is the call. When the server is idle, a lone write syncs immediately. Under load, the first waiting writer holds the sync open for --fsync-delay-us (default 4000 µs) so others can join, and one fdatasync covers the whole group. That trades a little latency for throughput; the ack still waits for the disk. See Performance.

Memory is bounded

The server does not hold the log in memory. Per-shard caches (recent writes, aggregate snapshots, idempotency state, compiled schemas) split a fixed budget: --memory-consumption-percent (default 80) of physical RAM or the cgroup limit, whichever is lower, or an explicit --memory-budget-bytes. An evicted entry is rebuilt from disk on the next miss. Eviction costs a scan, never correctness.

Thread-per-core

Each shard runs single-threaded on its own pinned executor. No shared mutable state on a shard's write path, so data races and lock-ordering deadlocks have nowhere to live there. Cross-shard coordination goes through explicit messages.

Failover and its S3 dependency

Leadership is a lease arbitrated by an S3 conditional write. While heartbeats between the nodes succeed, the leader does not touch S3 at all. S3 matters when leadership has to move.

• Leader dies, S3 healthy. The follower waits out its heartbeat lease (--heartbeat-lease-duration-ms, default 1500), wins the S3 CAS and runs S3 catch-up before accepting writes. Writes pause for that window. Reads keep serving. --s3-lease-duration-ms (default 30000) matters only when no live heartbeat backs the lease, such as a cold boot.
• S3 unreachable, both nodes healthy. Nothing changes; replication runs over TCP. The exposure is losing a node before S3 returns. A dead leader cannot be replaced, and a dead follower leaves no second copy for the ack. Writes stop until S3 is back. Reads keep serving, possibly stale.

Clock skew is the third edge. --max-clock-drift-ms (default 500) is the margin: the follower rejects replication batches stamped further than that from its own clock, and the leader fences itself that long before its lease expires. Skew past the margin turns into rejected replication and lease churn. Run NTP.

The chaos harness drives each of these faults under load. Details in Leader election on S3.

Tamper evidence and encryption

The WAL is a BLAKE3 hash chain per shard; see The audit chain. The server does not encrypt data at rest. Payloads can be encrypted client-side and carry an iv; see Encryption.

Outside the envelope

Losing both copies at once: both nodes' disks, or the leader's disk and S3 while the follower is down. Anything upstream of the write: a bug that appends the wrong event is durably wrong. For recovery beyond the cluster, see Backup and recovery.

Pre-1.0

The server is at 0.2.0. Cluster and failover internals can still change before 1.0.