Implementing idempotent writes
A timeout on a write tells you nothing. The event may be in the log or it may not, and a naive retry doubles it. This guide makes a write safe to retry. See Idempotent retries for the concept.
Two kinds of writer need very different amounts of machinery. Most of this guide covers the harder one: a service (a BFF) doing conditional writes. It follows the reference account service almost line for line: Celeriant.Reference/ in the .NET client repo, an HTTP API over Celeriant with a read projection. The same service is ported to Rust as celeriant_reference in the server repo.
The reference ships two interchangeable projection backends, both safe to run as a fleet of replicas behind a load balancer: AccountService.cs (Postgres read model) and AccountServiceMem.cs (in memory, each replica folds the stream itself). The write loop is identical in both. Where the dedup bookkeeping differs, this guide shows both.
The easier kind of writer is the offline client, covered at the end. If that is what you are building, most of this machinery falls away.
Two keys, two owners
There are two idempotency layers, and they are easy to mix up:
ClientSeqis Celeriant's key. The server keeps the highest sequence seen per(aggregate, clientId)and rejects any write whose lowest sequence is at or below it. That stops a retried write from appending twice. It answers one question: did this sequence already land?EventIdis your key. An opaque id you stamp on the event, derived from the request (an HTTPIdempotency-Key, an upstream message id). Celeriant stores it and hands it back on read; it never looks inside it. You use it to recognise your own requests, so a retry gets its original response back instead of just "yes, that landed".
Celeriant's layer protects the log. Your layer protects the caller. You need both, because 2002 (IdempotencyViolationException) carries no result, and your caller asked for one.
The two prerequisites
- A stable client id per writer. Every write takes it explicitly; the client never invents one. Keep it stable across restarts, like durable service config (see identity).
- A monotonic
ClientSeqper event, withenforceClientIdempotency: true. A multi-event write is judged by its lowest sequence, so every event in a write must be new.
Derive the sequence from your own events
ClientSeq must survive restarts, so do not mint it from a process-local counter. The reference derives it from the stream: the projection stores the last sequence next to its cursor, and catch-up advances both while replaying new batches.
foreach (var batch in batches)
{
newBatchIndex = batch.AggregateVersion;
var trackClientSeq = batch.ClientId == Constants.ServiceClientId; // only your own sequence space counts
foreach (var evt in batch.Events)
{
if (trackClientSeq && evt.ClientSeq > maxClientSeq)
maxClientSeq = evt.ClientSeq;
newBalance = AccountEvents.ReplayEvent(newBalance, evt);
}
}
The filter is the part people miss. Every writer numbers its own events, so another writer's sequences mean nothing in your space. Track the max over your batches only, then write with maxClientSeq + 1.
The write loop
The deposit operation from the reference, whole, because every arm matters:
public async Task<WriteResult> DepositAsync(Guid accountId, int amountCents, Guid eventId, CancellationToken ct)
{
// catch-up returns fresh state AND answers "did this request already land?"
var (projection, hit) = await CatchUpAsync(accountId, eventId: eventId, ct: ct);
if (hit is not null)
return hit;
var clientSeq = projection.MaxClientSeq + 1;
var reDerive = false;
for (var attempt = 1; attempt <= MaxRetries; attempt++)
{
if (attempt > 1)
{
await Verify.Backoff(attempt, ct);
(projection, hit) = await CatchUpAsync(accountId, eventId: eventId, ct: ct);
if (hit is not null)
return hit; // a prior attempt landed; original response rebuilt
if (reDerive) { clientSeq = projection.MaxClientSeq + 1; reDerive = false; }
}
if (amountCents <= 0)
throw new ValidationException("Amount must be positive.");
var newBalance = projection.BalanceCents + amountCents; // re-decide on fresh state, every attempt
var evt = AggregateEventExtensions.Create(1L, new Deposited(amountCents), Serializer,
clientSeq: clientSeq, // Celeriant's idempotency key
eventId: eventId); // YOUR idempotency key, carried on the event
try
{
await pool.WriteAsync(Constants.AccountKey(accountId), [evt],
clientId: Constants.ServiceClientId,
allowCreate: true,
expectedVersion: projection.LastBatchIndex, // fresh from THIS attempt's catch-up
enforceClientIdempotency: true,
ct: ct);
var newBatchIndex = projection.LastBatchIndex + 1;
// index entry no later than the cursor bump; see Reconstructing the response
await RecordWriteAsync(accountId, eventId, newBalance, newBatchIndex,
projection.LastBatchIndex, clientSeq, ct);
return new WriteResult(newBalance, newBatchIndex);
}
catch (WriteOccException)
{
reDerive = true; // 2003: the world changed. New decision, new sequence
}
catch (Exception e) when (e is CeleriantTimeoutException or RequestOutcomeUnknownException)
{
// ambiguous: hold clientSeq; catch-up at the top refreshes expectedVersion
}
catch (InflightDuplicateWriteException)
{
// 2013: a prior attempt is accepted but not yet durable. Success now could
// be a false ack if it rolls back. Hold clientSeq, back off, retry.
}
catch (IdempotencyViolationException)
{
// 2002: someone landed this sequence. A timed-out prior attempt of OURS,
// or a sibling request that raced us to the same number. The stream knows which.
switch (await Verify.WhoOwnsSeqAsync(pool, accountId, clientSeq, eventId, ct))
{
case SeqOwnership.Ours:
{
var (p, h) = await CatchUpAsync(accountId, eventId: eventId, ct: ct);
return h ?? new WriteResult(p.BalanceCents, p.LastBatchIndex);
}
case SeqOwnership.Sibling:
reDerive = true;
continue;
default: // never guess
throw new OccExhaustedException("state unverifiable; retry the request");
}
}
}
throw new OccExhaustedException("did not complete after retries; retry the request");
}
OccExhaustedException, ValidationException and WriteResult belong to the reference, not the client library; the reference maps OccExhaustedException to HTTP 409. Every other exception here is from Celeriant.Client.Errors.
RequestOutcomeUnknownException lands in .NET client 0.9.0. From 0.9.0 it is how a fully sent write with no answer reaches you (response timeout, reset, EOF), and the pool never re-sends it. 0.9.0 also adds PoolUnavailableException: this process's own pool refused before dialling anything, so nothing was sent and a plain retry is safe. An exhausted leader walk throws ConnectionFailedException with the last node's error as InnerException; nothing reached a leader.
On 0.8.0, the published version, drop RequestOutcomeUnknownException from the filter: a response timeout surfaces as CeleriantTimeoutException, and a connection lost after sending as ConnectionFailedException, after which the pool may already have re-sent the write to another node. The reference itself still catches CeleriantTimeoutException alone.
Walking the arms:
- Success: record the result in the dedup index, then bump the projection cursor (guarded, so it never goes backwards). Order matters: the cursor bump stops catch-up replaying this event for anyone, so the index must already hold the answer when it lands. Reconstructing the response shows what
RecordWriteAsyncis in each shape. - Conflict (2003): the world changed and your write was not applied. A new decision: catch up, re-check the business rules against fresh state, take a fresh
ClientSeq. See handling conflicts. - Ambiguous (timeout, outcome unknown): your write may or may not have landed. Hold the
ClientSeqand go again. The catch-up at the top of the next attempt refreshesexpectedVersion; the next section explains why that matters. - Inflight duplicate (2013): a prior attempt is accepted but not yet durable: still queued before fsync, or fsynced but past the read cursor, meaning not yet replicated. Hold the
ClientSeq, back off, retry. It resolves into a 2002 or a clean write. - Idempotency violation (2002): this sequence already landed, but with concurrent requests sharing one client id, possibly not by you. Read the contested sequence back. Your
EventIdon it: the prior attempt landed, success. A sibling's: your event never landed, re-derive and go around.
Why the ambiguous arm works
The server checks the version guard before idempotency. Follow what that does to a timed-out write that actually landed: the version has moved, so resending the identical request returns a conflict (2003), never a 2002. Obey the conflict rule, re-derive a fresh sequence, and you deposit twice. That is the exact double-write all of this exists to prevent.
The loop avoids it because every retry catches up first. The expectedVersion is fresh; the ClientSeq is held. Now the server gives a straight answer. If the prior attempt landed, the guard passes and idempotency returns 2002; once the ownership check confirms the event is yours, you are done. If it never landed, both checks pass and the write lands now.
Conflict means re-derive. Ambiguous means hold the sequence and refresh the version.
Why re-deriving on conflict is safe
A 2003 means the server rejected the write. Nothing was appended; your sequence was never used. Taking a fresh one cannot duplicate anything. And you have to take one: if another request on the same client id used your sequence first, the held one only ever bounces.
The risky case is an ambiguous failure followed by a conflict. That 2003 may be your own landed write moving the version. Re-derive blindly and you write it again under a new sequence. The loop guards against this with its ordering: every retry catches up and checks the dedup index before it re-derives, and that catch-up is the read-back. A landed attempt is found by its EventId and returned. If it is still in flight and not yet readable, the catch-up misses it, but the re-derived write carries the stale expectedVersion and bounces with another 2003; it cannot land until the catch-up can see, and therefore return, the original. The same ordering covers a 0.8.0 pool that re-sent your write to a new leader behind your back: you see only the 2003, and the catch-up still finds the landed event. Keep that order if you restructure the loop.
Who owns the sequence
The client id is shared by every concurrent request in the service, so two requests can pick the same sequence. Usually the version guard sorts it out: the loser gets a 2003. The exception is a loser whose 2003 was lost to a timeout. It retries the held sequence, which its sibling has meanwhile used, and gets a 2002 about someone else's event. Take that at face value and you report success for a write that never happened.
So a 2002 is never taken at face value. There is no bookkeeping to maintain for this: the stream knows who owns the sequence, and a 2002 is an error path, so the read is paid only when something already went wrong. The reference's Verify.cs asks with a point read:
public static async Task<SeqOwnership> WhoOwnsSeqAsync(
ICeleriantPool pool, Guid accountId, long clientSeq, Guid eventId, CancellationToken ct = default)
{
ReadResponse resp;
try
{
resp = await pool.ReadAsync(new ReadRequest
{
AggregateKey = Constants.AccountKey(accountId),
Filters = ReadFilters.From(1) with
{
MinClientSeq = clientSeq,
MaxClientSeq = clientSeq,
IncludeClientId = Constants.ServiceClientId,
},
}, ct);
}
catch (AggregateNotFoundException)
{
return SeqOwnership.Unwritten; // a lagging replica hides events; it never misattributes them
}
var evt = resp.EventBatches.SelectMany(b => b.Events)
.FirstOrDefault(e => e.ClientSeq == clientSeq);
if (evt is null) return SeqOwnership.Unwritten;
return evt.EventId == eventId ? SeqOwnership.Ours : SeqOwnership.Sibling;
}
The sequence and client filters match on batch metadata, so the server skips every other batch without reading its events. Scope the read to your client id. Without it, another writer's event at the same number can come back first, read as a sibling's, and send your already-landed write around the loop under a fresh sequence.
Unwritten is not a verdict. It means no event holding the sequence is visible to this read, never that the sequence is free: a lagging replica, a soft-deleted aggregate, or a trimmed range all produce it, and the server still rejects the sequence as consumed. Refuse to guess and return a retryable error. A false failure costs a retry; a false success loses the write.
One gap in the reference: on a trimmed aggregate, a read from version 1 fails with BatchIndexUnavailableException (1000) instead of returning what is left. Catch it and re-read once from its MinimumAvailableVersion. Neither client library ships this check; it lives in the reference only.
Every event needs an EventId for this to work, so the reference mints one per request when the caller sends no Idempotency-Key. And because the answer comes from the stream, it is correct on any replica. No per-instance state to go cold.
Reconstructing the response
2002 says "already landed" and nothing else. Your caller asked for the new balance. That is the EventId's other job: keep a small index keyed (eventId, aggregateId) holding the response each write produced, for a recent window. The reference uses 90 seconds (Verify.DedupWindow).
One rule makes the index safe across a fleet: it lives wherever your projection cursor lives, and it moves with the cursor. Catch-up replays only events newer than the cursor. Once the cursor passes an event, replay never sees it again, so whoever advances the cursor must index the event in the same motion, or nobody ever will.
With an in-memory projection, each replica folds the stream itself, and the fold maintains the index in the same pass that applies each event (AccountServiceMem.cs):
// inside the fold, under the same lock as the cursor. A batch's AGE is measured
// in server time (batch vs tip of this read), so clock skew cannot misjudge it;
// only the REMAINING lifetime runs on the local monotonic clock.
var age = tipTs - batch.ServerTimestamp;
// ... per event:
acc.BalanceCents = AccountEvents.ReplayEvent(acc.BalanceCents, evt);
if (age < window && evt.EventId is { } eid)
acc.Recent[eid] = new RecentWrite
{
BalanceCents = acc.BalanceCents,
BatchIndex = batch.AggregateVersion,
ExpiresAtMs = now + (long)(window - age).TotalMilliseconds,
};
The write path inserts its own entry with the full window. The writer's cursor bump means this replica never re-folds its own event, so this entry is the only record it will ever have. Derive its lifetime from the fold's tip instead and an idle account's entry is born mostly spent; a retry a minute later double-writes inside the stated window.
A retry landing on any replica is caught: that replica either already folded the original event (index hit), or folds it during the retry's own catch-up. Replicas share nothing but the stream.
With a Postgres projection, the cursor is shared, so the index is a table beside it, written in the same statement as the cursor bump (AccountService.cs; this is RecordWriteAsync from the loop):
WITH proj AS (
UPDATE account_balances
SET balance_cents = @balance, last_batch_index = @batchIndex,
last_client_event_index = @clientSeq, updated_at = now()
WHERE account_id = @id AND last_batch_index = @expectedBatchIndex
)
INSERT INTO request_responses (event_id, aggregate_id, balance_cents, batch_index, expires_at)
VALUES (@eid, @id, @balance, @batchIndex, now() + @windowMs * interval '1 millisecond')
ON CONFLICT (event_id, aggregate_id) DO UPDATE
SET balance_cents = EXCLUDED.balance_cents,
batch_index = EXCLUDED.batch_index,
expires_at = GREATEST(request_responses.expires_at, EXCLUDED.expires_at);
One statement, so no replica can observe the bump without the row. Catch-up's replay persists its window the same way, in one statement with its own cursor upsert. The lookup costs no extra round trip: it rides the query that already reads the projection row, as a LEFT JOIN on request_responses. If this statement fails, the reference logs and moves on: the event is in Celeriant, neither the row nor the bump was applied, and the next catch-up replays it.
The half-shared configuration is the broken one: a shared cursor (Postgres) with a per-replica index (memory). Replica A writes the event, indexes it locally, bumps the shared cursor. The retry lands on replica B, whose catch-up starts past the event and replays nothing. B's index is cold, so B derives a fresh ClientSeq and deposits again. No error fires anywhere; a re-derived sequence never collides. Share both or share neither.
This index does not prevent double-writes. The server's (clientId, ClientSeq) check does that. The index only restores the lost response.
It also stores no request fingerprint, so it cannot tell two operations apart. Reuse an Idempotency-Key for a different operation inside the window and you get the first operation's response instead of performing the second. A key names one user intent; that is the caller's side of the contract. An API that cannot trust its callers should store a hash of the request with the key and reject reuse with a different payload.
The 90-second window
The index only holds entries younger than the window. That is where the request-level guarantee ends, so know exactly where.
A retry of the same key arriving after the window finds nothing. The entry expired, and replay cannot restore it: the projection already folded that event. The handler derives a fresh ClientSeq and writes a second event. Celeriant's side held; it really is a new sequence. Your request-level promise is what expired.
Choose the window deliberately:
- Size it to the retry source. 90 seconds covers transport-level retries: your gateway, your HTTP client's backoff. It does not cover a user resubmitting tomorrow.
- Widen it if you must honour late retries. In the Postgres shape the index is already a durable table; pushing
expires_atout costs table size. - Scan on miss as a last resort. The
EventIdis on the event, so a read of the aggregate's recent history finds it, at the cost of that read per miss. Bound the scan withMinServerTimestamp, anchored on the newest server timestamp your fold has seen minus the window. Never the local clock, for the same skew reason as above.
Whatever you pick, make the window a stated property of your API rather than a surprise.
Transfers: two aggregates, one request
A transfer writes TransferredOut to one account and TransferredIn to another in one write request. The server validates every aggregate in the request (version guard first, then idempotency) before appending anything, so the write is all-or-nothing. You never get one leg.
Each leg carries its own per-aggregate ClientSeq. Both events carry the same request EventId.
All-or-nothing is what makes reconstruction work. An index hit on either (eventId, account) pair proves the whole transfer landed, so on a partial hit (one entry expired) return success and rebuild the missing leg from current state. Do not fall through to the write: it would mint fresh sequences and land a second transfer with no error to catch it.
With no hit on either leg, write. A 2002 there is settled by the point read with the same logic: your EventId on either leg's sequence proves the transfer landed, a sibling owning a leg proves it did not. Unwritten on one leg just means the violation was on the other.
Scaling out: many replicas, one client id
Everything above already works for a horizontally scaled service: a k8s Deployment behind a load balancer, autoscaled. Two decisions have to be right, and the reference makes both.
The client id names the service, not the replica. One config-driven id shared by every replica (or one service keypair mounted as a Secret, if identity is enforced). Not one per pod. Never one per request.
- Per request is the expensive trap. The first idempotent write for a
(aggregate, clientId)pair the server has not cached makes it establish where that client's sequence stands: a per-aggregate bloom filter answers "never wrote here" cheaply when one is built, and a backward scan of the aggregate's history runs when it is not. A fresh id per request puts every write on that path and churns the per-client cache for everyone else. TheEventIdfield exists so you never need this. - Per pod buys almost nothing. OCC serialises concurrent writers regardless of whose sequence space they use, and the 2002 check works either way. What it costs: the same first-touch lookup per pod per aggregate, identity churn as the autoscaler cycles pods, and a lost backstop. Two replicas processing the same retried request concurrently collide on the sequence, and the server rejects one. That collision only happens when they share the id.
The dedup index obeys the colocation rule. Reconstructing the response is the whole fleet-safety story. In memory: every replica folds the stream, so every replica can answer every retry. Postgres: cursor and index share a statement and move together. The one configuration that double-writes is the half-shared one.
Nothing else coordinates. Replicas do not know about each other. The stream (plus, in the Postgres shape, the store they already share) is the only common ground, and the error paths resolve against the stream by point read.
Consistent-hash routing by aggregate id is worth adding as an optimisation: each aggregate folds on one replica, memory is not duplicated, index hits stay local. It is not load-bearing. Reshuffles, failovers, and retries landing on the "wrong" replica are absorbed by replay. The reference_account_service integration test in the server repo drives replicas of the Rust port through cross-replica retries, concurrent duplicates, and sibling races. The in-memory variant always runs; the Postgres variant runs only when POSTGRES_URL is set.
The offline client
Everything above is the BFF shape. Its hard parts come from two things: concurrent requests sharing one client id, and the version guard sitting in front of idempotency. An offline-first client (mobile app, browser) has neither.
- The local queue is the outbox. Assign
ClientSeqfrom a local monotonic counter and persist the event with its sequence in one transaction to durable local storage (SQLite on device, IndexedDB in the browser) before any network attempt. A crash re-reads the queue with its numbers intact; nothing is renumbered. This is the replay-trap fix below, built in from the start. - One client id, one thread. The client id is the device's identity, and a single sync loop drains the queue in order. With no sibling requests on the id, a 2002 can only refer to your own event: no
EventIdcheck, no false-success edge. - No version guard on sync. Offline writes are unconditional (no
expectedVersion); conflicts are resolved in the projection instead (see the offline exception). With no OCC check in front, there is no 2003 arm and no re-derive rule. Hold the sequence is the only rule left.
The whole sync loop:
- Take the next unsynced event (or a run of them, in order) from the local queue.
- Write with
enforceClientIdempotency: trueand noexpectedVersion. - Success or 2002: mark the events synced locally and advance. A 2002 here means a previous ack got lost; the events are in the log.
- Ambiguous failure: resend the same events with the same sequences.
- Inflight duplicate (2013): back off, resend the same.
Note what disappeared. No catch-up before each attempt: there is no version to refresh. No response reconstruction: the client's state lives locally, and the ack just means "synced". No window: the local queue holds the dedup state for as long as it takes. Marking events synced need not be atomic with the ack: crash after the ack but before the mark, and the restart resends, gets a 2002, and marks it then.
One rule still applies: a multi-event write is checked by its lowest sequence, so sync in queue order and never skip ahead.
The replay trap
All of the above assumes one logical event maps to one stable ClientSeq. Restarts break that if you regenerate sequences on boot: an outbox worker that crashes mid-batch and renumbers from a reset counter re-issues written events under fresh sequences, and dedup never fires. ClientSeq must be deterministic from durable state, never a counter bumped per attempt. Three patterns work:
- Derive it from the stream, as this guide does: your own max sequence is recoverable by replay, and the projection persists it between catch-ups.
- Derive it from a persistent upstream source: the outbox row's primary key, the upstream message id.
- Persist the next sequence with the data the write is generated from, in the same transaction, so crash recovery rereads it.
The wrong pattern is the default one: a runtime counter that does not survive a crash.