Skip to main content

Verifying the audit chain

A hash chain only proves something when someone recomputes it. Celeriant recomputes it in one place, replication, and ships no verifier. This page says what the BLAKE3 chain covers, what checks it today, and what an audit has to do by hand.

What the chain covers​

The chain runs per shard, over WAL entries (metablocks), not per event. Each entry is one event batch, delete, trim, or schema registration. Its hash is BLAKE3(previous_hash || metablock_bytes), skipping the CRC and two node-local offsets, so leader and follower compute the same value from different disk layouts. The first entry chains from 32 zero bytes.

Payloads are covered unevenly:

  • Small batches (compressed body of 718 bytes or less in the default build) are stored inside the metablock, so their bytes are in the hash.
  • Larger batches live in a separate datablock. The metablock holds only its CRC32C, so that is all the chain covers. CRC32C catches corruption. It does not stop a deliberate edit, because a replacement with the same checksum is easy to construct.

Treat the chain as tamper-evidence for metadata and small payloads, and as corruption detection for large ones.

What checks it automatically​

Replication, and only replication:

  • Divergence on failover. When leadership moves or a node rejoins, the first incoming batch's predecessor hash must match the local tip. On a mismatch the follower rejects the batch, the leader counts it in celeriant_replication_tip_hash_mismatch_kick_total and kicks the follower into S3 catch-up, which walks back to the common ancestor and truncates the divergent entries.
  • Batch continuity. Before the leader uploads a replication batch to S3, its entries are checked as an unbroken chain; a break is counted in celeriant_replication_intra_batch_chain_break_total. A node catching up from S3 checks the batch's first link against its own tip.

Nothing else. A booting node loads its persisted tip and trusts it. There is no client API that returns a hash, and no command that recomputes one.

Compaction breaks the on-disk chain​

Compaction (every --compaction-check-interval-secs, default 7200) rewrites sealed segments without the entries of deleted and trimmed aggregates. It keeps the segment's original tip hash in the header. The surviving entries no longer chain to it. That is by design: the segment was fully replicated and verified before compaction ran.

For an audit this means a recompute over a compacted segment fails whether or not anyone tampered with it. Audit copies have to be taken before compaction touches a segment.

Inspecting by hand​

The server repo has one tool: celeriant-wal-inspect. It is not in the container image; build it with cargo build --release -p celeriant_wal_inspect. It reads one segment file (<data-root>/shard_<n>/log_<id>.wal) and prints; it does not recompute anything.

# front and rear headers, including write_tip_hash and read_tip_hash
celeriant-wal-inspect /data/shard_3/log_1.wal header

# first and last wal_seq in the file
celeriant-wal-inspect /data/shard_3/log_1.wal bounds

# every metablock in a range, with its previous_tip_hash
celeriant-wal-inspect /data/shard_3/log_1.wal range 133935 133942

Each range entry prints like this:

wal_seq = <n> | lease = <epoch> | offset = <file offset> | server_ts = <ts> | node = <node id, hex>
  previous_tip_hash             = <64 hex chars>
  uncompressed_size             = <bytes>
  compressed_size               = <bytes>
  datablock_position            = <offset>
  kind                          = <EventBatchMetadata | SoftDelete | SoftTrim | SchemaRegistration>(...)

The useful comparison is across nodes. previous_tip_hash at wal_seq N+1 is the chain value through N. Print the same range on both nodes of a cluster: equal hashes mean identical history up to that point, and the first difference is where they forked.

An audit​

  1. Copy the segment files before compaction reaches them, or keep an independent record of tip hashes: the write_tip_hash from header, written somewhere an attacker on the cluster cannot also rewrite.
  2. Recompute the chain over the copy with your own script, using the rule above: BLAKE3 over the previous hash and the metablock bytes minus the CRC and the two node-local fields. The metablock layout is in celeriant_wal/src/metablocks/metablock.rs, the hash in compute_entry_hash (celeriant_shard/src/shard_wal_sync.rs).
  3. Compare against the reference: the other node's copy, an earlier backup, or the recorded head. A match proves the covered bytes are unchanged; the first mismatch localises the edit.

The chain shows that history changed, not who changed it. For integrity a third party can check, anchor the head hash outside Celeriant on a schedule.