Skip to main content

Correctness testing

"No split-brain, no lost writes" costs nothing to claim. The evidence is a harness that attacks exactly those guarantees and fails to break them. Celeriant has three layers of it, plus unit and contract tests inside the crates. This page says what each one does and where the gaps are.

Chaos harness: celeriant_chaos​

The heaviest layer. It drives an already-deployed two-node cluster plus S3-compatible storage (MinIO on the Raspberry Pi 5 rig, the default target; an EC2 deploy is also supported). Same server binary you would ship, real TCP, real disks. Every action goes over SSH through the deploy directory's Makefile.

Each scenario tears down, brings the cluster up, waits for a stable leader, then runs a bench while the fault lands. The bench defaults to 4000 concurrent client tasks. Correctness is judged on a saturated cluster, mid-fault, with writes in flight.

Faults​

  • SIGKILL: crash with no cleanup and no lease release.
  • Graceful stop: systemctl stop, to check the planned path separately from the crash path.
  • SIGSTOP / SIGCONT: freeze the leader past its lease, let the follower take over, then thaw the old leader and make it discover it was deposed.
  • Partitions: firewall DROP rules on a host and port. One direction or both, replication port and S3 port separately.
  • Clock skew: the host clock moved with NTP disabled. One node, or seeded random skew on both.
  • Disk full: the data volume filled to a small reserve with fallocate.
  • S3 outage: MinIO stopped and restarted, short and long.

Scenarios​

--full runs 28 named scenarios. The sharp ones:

  • leader_sigkill, leader_graceful_stop: the follower must promote and serve writes.
  • leader_restart_loop: three kill/restart cycles; leadership must actually change hands, not bounce back to the same node.
  • partition_asymmetric: one-way packet loss, the classic split-brain maker.
  • bridge: replication cut both ways while both nodes keep S3. The leader holds on S3 renewal; the follower's challenges must lose the CAS.
  • single_node_isolation: the leader loses its peer and S3. It must fence itself inside the lease, and the survivor promotes with no window where both ack.
  • partition_then_kill_minio: follower killed and MinIO stopped together. The leader has no durable second copy, so it must fence and stop acknowledging.
  • sigstop_leader: the zombie-leader test above.
  • idempotency_audit_*, duplicate_replay: idempotent writes through outages and blackouts, then an audit that every acknowledged client_seq landed exactly once. duplicate_replay resubmits every acked write and expects 2002 back.
  • cas_storm, cas_storm_partition: many writers at one expected_version. Exactly one wins; losers get a definite OCC conflict, not a timeout.
  • nemesis_composition: seeded concurrent partition-flap and clock-jitter loops over one bench window.
  • schema_under_partition: a schema registered during a partition must reach the follower and be enforced after it promotes.

A few more run only with --scenario: a load sweep, a heavy cardinality fill, a cold-connect herd, and two defect reproductions (write_outage_selfheal, promotion_failure_survival) that the harness source labels as expected to fail until the server fix lands.

Checks​

Metric checks come from a 2 Hz Prometheus scrape of both nodes. Each scenario declares the disruption it expects; anything past that fails the run even when no data was lost.

  • ExactlyOneLeader: counts scrape ticks where the two nodes' roles do not sum to one leader. Baseline allows zero. Failover scenarios allow a bounded number, since a failover has a leaderless gap.
  • FailoverWithinBudget: longest leaderless run, measured to within one scrape interval.
  • NoTruncateDroppedSelfAcked: server counter for a rollback that dropped a write this node had acknowledged. Must stay zero.
  • NoSameEpochDivergence: server counter for two S3 fallback batches with the same lease epoch and different content. Must stay zero.
  • EventualConvergence, WalSeqAdvanced, FinalLeaderWroteDuringBench, DistinctLeaderHosts, BenchThroughputFloor, BenchErrorsBounded: the cluster made progress, the failover actually happened, throughput and errors stayed in bounds.
  • Bounds on elections, S3 fallbacks, heartbeat failures, shard panics, node restarts and role flips.

Metrics can miss a fork both nodes agree about, so the heavier scenarios stack oracles on the durable state:

  • History checkers over the per-operation client log: idempotency, OCC, WAL monotonicity, final-read parity from both nodes. The idempotency check fails closed if any history record was dropped.
  • Tip-fork check: after the cluster stops, SSH to both nodes, read each shard's WAL header with celeriant-wal-inspect, compare write sequence and tip hash. Same sequence with different hashes is a silent fork that a sequence comparison would pass. When one node lags, the leader's entry just past the lag point must chain to the lagging node's tip.
  • Epoch oracle: lease epochs never decrease on a node and agree across nodes.
  • Disk truth: every aggregate the read-back audit flags as missing a sequence is rechecked by running celeriant-wal-inspect on both nodes' raw WAL files. The audit does not get to trust the read path it is auditing.
  • Journal and resource checks: panics, aborts and error storms in journalctl; fd count and RSS against a pre-bench snapshot.

Soak​

--soak <seconds> repeats the scenario set until the deadline, each iteration in its own report directory. It stops at the first failing iteration, or records it and keeps going with --soak-continue-on-failure. --full refuses to start on a dirty working tree, so every report maps to a commit.

Integration tests: celeriant_integration_tests​

213 named tests in one binary. Each spawns its own server processes on localhost with temporary data directories; tests that need S3 start a MinIO container. Categories cover replication, election, fencing, compaction, schema, security and durability. Two kinds stand out:

  • Crash durability: SIGKILL a server under concurrent idempotent load after the log has rotated, restart on the same data, and check that every acknowledged (client, seq) is present exactly once with byte-exact content.
  • Metamorphic parity: run the same workload two ways (standalone vs cluster, leader vs follower, before vs after failover or divergence recovery) and require identical reads.

Fuzzing: fuzz/​

AFL++ campaigns over the decode paths that hostile or corrupted bytes reach first: wire headers, bincode payloads, datablocks, metablock accessors, bloom filters, S3 fallback batches, and CRC-valid headers and metablocks carrying hostile fields. Eight targets. The goal is fail-closed: bad bytes must error, never panic or read out of bounds.

Crashes found so far live in fuzz/known_crashes/ as reproducers. verify_known_crashes.sh asserts they still crash, so they are tracked as open bugs, not fixed ones.

What this is not​

  • Not deterministic simulation. No single-threaded, seed-replayable simulator of the whole system. --seed replays the chaos fault schedule and jitter, not the execution, so a failure reproduces by re-running, not bit-for-bit.
  • Not independently audited. No third-party Jepsen engagement. The author's harness checks the author's code.
  • One topology. Two data nodes plus S3. No sweep across kernels, hardware generations or larger clusters.
  • Metric sampling has limits. A 2 Hz scrape cannot prove there was never an instant with two leaders. The durable-state oracles are what would catch the damage.

Treat the fault list and checks above as the actual state of correctness testing, and weigh the gaps as real.