Architecture at a glance

The whole shape on one page. Each piece has its own concept or operations page; this is the map.

Thread-per-core

Celeriant is a Rust server built on Glommio: one single-threaded executor per shard, pinned to a CPU, running io_uring. --num-shards defaults to the CPU count. A shard owns its aggregates, its WAL and its caches outright, so there is no shared mutable state and no locking on the write path. Shards talk to each other through bounded channels. S3 traffic, which needs an HTTP client, runs on a separate Tokio runtime and never touches the hot path.

Every shard listens on the client port. A connection that lands on the wrong shard is handed over to the owning shard's executor, and the client never sees it.

Routing

An aggregate is org_id / aggregate_type_id / aggregate_id. It maps to a shard by routing_id % num_shards, where routing_id is one of the three ids, chosen by --routing-rule (default aggregate_id). With --reserve-coordinator-shard, shard 0 keeps only coordination work and data routes to shards 1 to n-1. The shard count and routing rule are recorded at first start and a restart with different values is refused.

Plain modulus on an id you control is deliberate. Two aggregates can be written atomically only if they land on the same shard, so you co-locate them by choosing ids, or by routing on org_id or aggregate_type_id. A write that spans shards is rejected with 9001 (ShardRoutingMultipleShards). See Consistency boundaries.

The write path

Four phases, and the client's ack waits for all of them:

1. Validate. Optimistic concurrency against the expected version, then the idempotency check on client id and sequence, then schema validation.
2. Append to the shard's in-memory pending state.
3. Fsync. Concurrent writes are coalesced for up to --fsync-delay-us (default 4000) and written with Direct I/O, then one fdatasync.
4. Replicate. Coalesced again, for up to --replication-delay-us (default 17000), and sent to the follower, which writes and fdatasyncs the batch before it answers.

Only after the follower's ack does the leader make the write visible to readers and watchers and answer the client. On the follower, a replicated batch stays durable but invisible until the leader confirms it committed. See Durability and safety.

Direct I/O is deliberate: it bypasses the page cache, so an fsync failure surfaces instead of being lost in writeback.

The cluster

Two nodes, one leader and one follower. No Raft, no ZooKeeper, no third node. Leadership is a lease in an S3 object, taken by a conditional PUT on its etag, with an epoch that only rises. In steady state S3 is never touched: the leader heartbeats the follower every 500 ms, and each ack extends both leases. The leader fences itself 500 ms (--max-clock-drift-ms) before its lease runs out; the follower challenges only at full expiry, so the old leader has stopped writing before a new one can win.

When the follower is unreachable, the leader replicates to S3 instead, so an acknowledged write still lives on two storage systems. The follower catches up from S3 when it returns. See Leader election and S3 leases.

--standalone runs one node with no replication and no S3. See Single node.

Storage and memory

Each shard writes a rotating WAL of preallocated segment files (1 GiB by default). Every segment carries a bloom filter of the aggregates written to it. Hot data sits in per-shard caches sized from --memory-consumption-percent (default 80% of available memory). A cold aggregate is found by scanning the log backwards and skipping every segment whose bloom filter rules it out. No per-aggregate index lives in memory, so memory tracks the working set, not the number of aggregates. The price is a slower first read of a cold aggregate.

Compaction rewrites segments to drop events you trimmed and aggregates you deleted.

Reading

Reads are per aggregate, ordered, and filtered by version range and event type (Reads and ordering). There is no query language; you project the log into a read store and query that (Building a read model). A watch tells you which aggregates changed, so a projection can follow the tail.

Crate map

Crate	Owns
celeriant	the server binary, flags, startup
celeriant_runtimes	executors, connection handling, routing, the shard loop
celeriant_shard	per-shard write pipeline, fsync, replication, S3 catchup, compaction
celeriant_wal, celeriant_rotating_log, celeriant_disk	WAL format, segment rotation, Direct I/O
celeriant_memcache	per-shard caches and pending state
celeriant_distributed	node states and the S3 lease
celeriant_sidecar	the Tokio runtime for S3
celeriant_watch	watch subscriptions
celeriant_ktls	kernel TLS offload
celeriant_client_tokio, celeriant_client_wire	the Rust client

Where to go next

• The model: Event sourcing and CQRS.
• The guarantees: Durability and safety.
• Running it: Deployment overview.
• Using it: the Quickstart.