Handling concurrency conflicts

Two writers read version 7, both decide, both write with expectedVersion: 7. One lands and the aggregate moves to 8. The other is rejected with 2003 and nothing of it is appended. The loser has to catch up, decide again on the new state, and retry. See Optimistic concurrency for the concept.

The guard

expectedVersion is compared with the aggregate's current version, the batch count that AggregateEventBatch.AggregateVersion, WriteResponse.MaxAggregateVersion and AggregateDetailsResponse.MaxAggregateVersion all report.

• expectedVersion: n commits only if the aggregate is at exactly n.
• expectedVersion: 0 with allowCreate: true commits only if the aggregate does not exist yet. Two creators race cleanly: one wins, the other gets 2003.
• null (the default) skips the check.

A stale guard throws WriteOccException (2003). It carries ExpectedVersion and CurrentAggregateVersion, the version the server actually holds.

The loop

Keep the folded state and its version between attempts, and catch up from where you stopped instead of replaying from 1.

using Celeriant.Client.Errors;
using Celeriant.Client.Requests;
using Celeriant.Client.Serialization;

const int MaxAttempts = 5;
var state = OrderState.Empty;
long version = 0;

for (int attempt = 1; ; attempt++)
{
    // 1. catch up
    try
    {
        await foreach (var batch in pool.ReadAllAsync(key, ReadFilters.From(version + 1)))
        {
            foreach (var e in batch.Events)
                state = state.Apply(e);
            version = batch.AggregateVersion;
        }
    }
    catch (AggregateNotFoundException)
    {
        // not created yet: state stays empty, version stays 0
    }

    // 2. decide on that state
    if (state.IsShipped)
        return;                                   // already done, possibly by our own earlier attempt
    if (!state.CanShip)
        throw new InvalidOperationException("order cannot ship");

    // 3. write, guarded on the version the decision was made against
    try
    {
        await pool.WriteAsync(key,
            [AggregateEventExtensions.Create(OrderShippedType, new OrderShipped(DateTimeOffset.UtcNow), JsonEventSerializer.Default)],
            clientId: writerId,
            expectedVersion: version);
        return;
    }
    catch (WriteOccException) when (attempt < MaxAttempts)
    {
        // someone moved the aggregate: loop, catch up, re-decide
    }
    catch (RequestOutcomeUnknownException) when (attempt < MaxAttempts)
    {
        // sent, no answer: it may have landed. Catching up shows which.
    }
}

A WriteOccException on the last attempt propagates. That is deliberate.

RequestOutcomeUnknownException needs client 0.9.0

RequestOutcomeUnknownException arrives in the .NET client's 0.9.0 release. It means the request was fully written and no answer came back, and the pool never re-sends it to another node. On 0.8.0 the same situation surfaces as CeleriantTimeoutException or ConnectionFailedException; catch those in its place.

When is the ambiguous case safe to loop on?

Catching up after an unknown outcome works here because the decision is a pure function of state. If the ship event landed, the fold sees IsShipped and stops. If it did not, the version is unchanged and the retry goes out.

It breaks for decisions that do not show up in state as "done". A deposit of 10 is a fine decision at every balance; catching up cannot tell whether your deposit is already in there, and the retry deposits twice. Those writes need a stable ClientSeq with enforceClientIdempotency: true plus an EventId you can look for. Implementing idempotent writes has the full loop.

One trap: the server checks the version guard before the idempotency check. Retry a landed write with the same ClientSeq and the same stale expectedVersion and you get 2003, not 2002. Catch up first, then retry.

Other failures

• PoolUnavailableException (0.9.0): the client's own pool refused before contacting any node. Nothing was sent. Retry.
• InflightDuplicateWriteException (2013, idempotency enforced and the guard passed): a prior attempt with this ClientSeq is fsynced but not yet replicated. Hold the sequence, back off, retry. Not success yet; it can still roll back.
• ConnectionFailedException: in 0.9.0, the request never reached a node, so retry after a backoff; a leader election looks like this. On 0.8.0 it can also follow a send, so treat it as unknown there. In 0.9.0 an exhausted leader walk also throws it, with the last node's error as InnerException, where 0.8.0 rethrew NotLeaderException.

Do not retry forever

A conflict means the world changed. It is not a transient fault to paper over. A loop that keeps losing on one aggregate has found a hot aggregate, which is a modeling problem. Cap the attempts and surface the contention.