Skip to main content

Backup and recovery

Celeriant has no backup tooling. No snapshot command, no export, no restore. The server and celeriant_cli do not have one. What you get is two replicas and a set of files you can copy yourself.

Replication protects against losing a node. It does not protect against a bad delete, a bad migration, or losing both disks. That part is yours.

S3 is not a backup​

In a cluster, S3 holds the leader lease, the two-slot membership record, and fallback batches the leader uploads while the follower is unreachable. None of that is a copy of your data. Fallback batches only cover the windows when the follower was unreachable, so they cannot rebuild a node on their own.

Fallback batches are not cleaned up. The catchup code has a delete path, but nothing calls it, so every batch uploaded during a follower outage stays in the bucket. Budget for the storage. If you prune them, do it only while both nodes are healthy and replicating over TCP, never while a follower is catching up.

What lives in --data-root​

  • shard_<n>/: the WAL segments and their .summary sidecars, one directory per shard. This is the data.
  • server_meta.toml: the immutable settings recorded on first boot.
  • dictionary.zstd_dict: the WAL compression dictionary. Its SHA-256 is pinned in server_meta.toml; a mismatched file stops the node.
  • private_key: the node identity. The node ID is derived from it at every start.
  • api_keys.toml: API key hashes, if you use them.

What you can do​

  1. Volume snapshots of --data-root. Stop the node first. Stopping the follower costs nothing but write latency: the leader keeps serving and replicates through S3 until the follower returns. A snapshot of a node under live writes has not been tested as a restore source.
  2. Logical export through the client. Page through list-aggregates and read each aggregate. For changes after that, a watch tells you which aggregates moved; read them again. A watch stream can end with an error when a notification may have been missed, for example across a leader change, so reconcile against a fresh listing after any error.

Pick 1 for fast whole-node recovery. Add 2 if you need a copy that outlives the storage format, or one you can load somewhere else.

Restoring​

There is no documented or tested restore procedure. Known constraints:

  • Identity travels with the directory. A copy of private_key is a copy of the node. Restore a snapshot only onto the node it came from. Two nodes started from one snapshot share one node ID.
  • Membership has two slots and nothing removes an entry. A replacement node with a fresh private_key has a new ID and finds both slots taken. Keep each node's private_key so a rebuilt node comes back as itself.
  • Immutable settings must match. Start the restored node with the flags recorded in server_meta.toml, or it refuses to boot.

Test the restore before you need it.

Recovery the cluster does for you​

  • Follower restarts. It boots, reads the lease, applies what it missed from the S3 fallback batches and from the leader over TCP, and rejoins as follower.
  • Leader dies. The follower's heartbeat lease expires after --heartbeat-lease-duration-ms (1500 ms by default) and it takes the lease through a conditional write in S3. See leader election.
  • Disk corruption. Every on-disk block carries a CRC32C, checked on read. A corrupt block is an error, not bad data served to a client.