Retention and deletion

The log is append-only, but it does not have to grow forever. Two operations remove data: trim drops the head of a stream, delete drops the whole stream. Both are logical first and physical later, and the gap between the two is what matters for erasure.

Trim

TrimStart takes an aggregate and KeepFromAggregateVersion. Batches below that version become unreadable. A read that starts below the new floor fails with error 1000, which reports the minimum available version.

• Trimming to or below the current floor is a no-op success.
• Past the aggregate's latest version is 3004 (TrimIndexOutOfRange). The newest batch always survives.
• A missing aggregate is 3000.
• No ExpectedVersion guard. Trim is a leader write, fsynced and replicated before the ack.

Delete

Delete takes a map of aggregate keys, each with its own options:

• ExpectedVersion: optional OCC guard. A mismatch is 4002.
• AllowRecreate: whether the key can be written again. If false, any later write gets 2006 (AggregateRecreateNotAllowed), permanently.
• AllowSequenceContinuation: on recreate, whether versions and event sequence numbers pick up after the deleted incarnation's last ones, or restart at version 1. Continue if downstream consumers key anything on version; they never see a number reused.

A delete ack means the tombstone is durable on both copies, same contract as a write.

Logical now, physical later

Trim and delete append a tombstone. Reads stop returning the data at once; the bytes stay on disk until compaction rewrites the segment. Compaction runs per shard, on each node, on a timer:

Flag	Default	Effect
--compaction-check-interval-secs	7200	How often each shard looks for work
--compaction-min-reclaimable-ratio	0.20	Skip a segment unless this share of its bytes is dead
--compaction-temp-dir	{shard_dir}/.compaction_tmp/	Scratch space; must be on the data filesystem

Each pass compacts at most one segment per shard: the oldest sealed, fully replicated segment over the ratio. It drops deleted batches, trimmed batches and pre-recreate batches. Tombstones and schema registrations stay.

The active segment is never compacted. Segments are preallocated at --shard-log-preallocate-bytes (1 GiB by default), so on a quiet shard deleted bytes can sit in the active segment until enough new writes fill it. No command forces compaction.

Erasure

Model for it up front. Keep a subject's PII in its own aggregate (a user-profile stream) and reference it by id from business events. Erasure is one Delete, and the order history stays useful. Scatter PII through business events and there is nothing to redact short of deleting the streams you wanted to keep.

Then accept that a delete is not a hard-deadline erasure on its own:

1. Delete and wait for compaction. Timing depends on segment fill and the interval above. Backups and downstream consumers still hold copies, and so does S3: fallback batches uploaded while the follower was unreachable are never deleted by the server (the delete path exists but nothing calls it). See Backup and recovery.
2. Crypto-shred. Encrypt the subject's payloads under a per-subject key and destroy the key. Unreadable everywhere at once, backups included. Metadata such as aggregate ids and timestamps stays in the clear.

For a deadline, use option 2, and delete as well so the ciphertext eventually goes too.

The audit-chain cost

Deletion does not edit retained events. Compaction does rewrite segments, though, and a compacted segment no longer recomputes against its stored chain tip. Erasability and end-to-end chain verification over the same segment do not coexist.