Skip to main content

Idempotent retries

A call can time out after the server committed but before you got the response. Retry naively and you write twice. Celeriant makes retries safe with a per-writer sequence number: the server refuses a sequence it has already committed.

We call it idempotent retries, not "exactly-once". Exactly-once takes two parties: the server refuses to apply a sequence twice, and your client retries with the same sequence until it gets an answer. The server can only promise its half. It cannot stop a client changing its id, minting a fresh sequence per attempt, or giving up halfway.

How it works​

Every event carries a client_seq you assign, and every write carries a client_id. For each (aggregate, client_id) pair the server keeps one number: the highest client_seq it has recorded. With enforceClientIdempotency: true, a write whose lowest client_seq is at or below that number is rejected.

await pool.WriteAsync(
    key,
    events: [new AggregateEvent { ClientSeq = 7, EventId = requestId, /* ... */ }],
    clientId: writerId,
    enforceClientIdempotency: true);

Things that follow from that design:

  • It is a high-water mark, not a set. The server cannot tell "already written" from "never written but below the mark". Issue sequences in increasing order per aggregate and never reuse one for different data. The .NET AggregateEventExtensions.Create helper defaults clientSeq to 1; pass a real value.
  • It is scoped per aggregate and per client id. Two writers each counting from 1 do not collide, because their client_ids differ. Change the client_id and the history starts over.
  • The mark moves on every write, enforced or not. An unenforced write with client_seq = 500 raises the mark to 500, and a later enforced write at 8 is rejected.
  • The history lives in the log. The mark is rebuilt from the WAL when it is not cached, so it survives restarts. There is no dedup table to lose.

Each writer needs a client id that stays stable across restarts. Every write takes it explicitly; the client libraries never invent one. Generate a fresh GUID at startup and the dedup history no longer applies: a retried write lands twice, and the server cannot tell, because to it they came from two clients.

The sequence has to survive restarts too. A service can re-derive it from the stream, since its own past events are replayable. An offline client persists it next to the queued event. Both patterns are in the guide.

Two rejections, not one​

A duplicate sequence comes back as one of two errors, depending on where the earlier write is:

  • InflightDuplicateWrite (2013): the write holding that sequence is queued, or fsynced but not yet replicated. It is not durable yet and can still roll back. Not success. Hold the sequence, back off, retry. It resolves to a clean write or a 2002.
  • ClientIdempotencyViolation (2002): the write holding that sequence is committed. In .NET, IdempotencyViolationException.

What a 2002 proves​

Less than it looks. A 2002 proves that a committed batch on this aggregate, from this client id, recorded a sequence at or above your lowest one. It does not prove that batch holds your event.

If your writer owns its client id alone and never reuses a sequence, the only way to hit 2002 is your own earlier attempt, so it means "already landed". Share a client id between concurrent writers (every request handler in a service, say) and a 2002 can be about a sibling's event. Two requests pick the same sequence; one loses the version check and gets a 2003; that 2003 is lost to a timeout; the loser retries its held sequence and gets a 2002 about the winner's event. Report success and the loser's write never happened.

So with a shared client id, verify a 2002 before trusting it. Put your own id on the event (EventId), point-read the contested sequence, and compare. Yours: the earlier attempt landed. Someone else's: yours never did, so re-derive and write again. Nothing there: refuse to guess and fail retryably. The idempotency guide has the read. The other option is one in-flight write per (aggregate, client_id) at a time.

Combined with optimistic concurrency​

The server checks existence first, then the version guard, then idempotency. So resending a guarded write that already landed, with the same expectedVersion, gets a conflict (2003), never 2002: your own write moved the version.

The retry recipe follows:

  • Timeout: the write may or may not have landed. Read to catch up, keep the same client_seq and payload, send with the fresh expectedVersion. Landed before: the guard passes and you get 2002 (2013 while that attempt is still in flight), which you then verify. Never landed: the write lands now.
  • Conflict (2003): your request was rejected. Catch up, re-decide on the new state, take a fresh client_seq.
  • 2013: hold the sequence, back off, retry.
  • 2002: verify ownership as above.

One catch in the conflict arm, on the published .NET client (0.8.0). Its pool resends a write that timed out or lost its connection to the next known node inside the same WriteAsync call, and a follower redirects it back to the leader. On a multi-node pool that resend can hit the version your first copy already moved, and the 2003 you see hides a write that landed. So catch up before re-deriving, and if your EventId is already in the stream, you are done. The Rust pool returns RequestTimeout or ConnectionLostAfterSend to you instead of resending, and .NET 0.9.0 does the same with RequestOutcomeUnknownException.

See the idempotency guide for the full loop, including HTTP request idempotency on top.