Skip to main content

Deployment overview

Celeriant runs in one of two shapes. They have different durability stories, so pick deliberately.

Standalone​

One process, --standalone, no replication, no lease. A write is fdatasync'd to local disk before it is acknowledged. Right for development and for workloads that accept one machine's durability.

What you give up: failover and a second copy. Lose that disk, lose the data. See Running a single node.

Two-node cluster​

A leader and a follower, coordinated through a lease object in S3. No Raft, no ZooKeeper, no third node. The leader fsyncs each batch and replicates it to the follower, which fsyncs before it acks. If the follower is unreachable the leader uploads the batch to S3 instead. Either way, no write is acknowledged until it sits on two storage systems.

This is the production shape. It survives losing either node; writes pause while the follower takes over. See Running a two-node cluster and Leader election and S3 leases.

What both need​

  • Linux with io_uring. The storage engine runs on Glommio. The project README asks for kernel 5.11 or later.
  • A filesystem that honours O_DIRECT. XFS or ext4. Not tmpfs, not overlayfs. The server checks this at boot and exits if Direct I/O is silently ignored.
  • In a container: --security-opt seccomp=unconfined and an unlimited memlock ulimit, both for io_uring.

What only the cluster needs​

  • An S3 bucket with conditional writes. The lease is a create-only PUT plus an ETag compare-and-swap. AWS S3 has both. For an S3-compatible store, check before you trust it. See Leader election and S3 leases.
  • NTP on both hosts. Lease timing compares wall clocks. A follower that sees more than --max-clock-drift-ms (500 ms) of skew on a heartbeat fences itself. The boot check only warns when the kernel clock is unsynchronised; it does not refuse. Treat NTP as mandatory anyway. The project's own Pi cluster runs chrony rather than systemd-timesyncd, because timesyncd let the clocks drift past the threshold under load.

Sizing​

  • Shards default to the CPU count, one core per shard. The count is written to server_meta.toml on first boot and cannot change afterwards, so pin --num-shards explicitly in production. A default that tracks the CPU count breaks the day you move the data directory to a bigger machine.
  • Memory defaults to 80% of detected memory (--memory-consumption-percent), where "detected" is the smaller of physical RAM and the cgroup limit. The caches are bounded, so memory does not grow with the number of aggregates.
  • Disk: the design assumes fast local disk. The EC2 reference stack defaults to instance-store NVMe and the Pi cluster runs on NVMe drives.

See Configuration for every flag.