Atomic multi-aggregate writes

A transfer debits one account and credits another. Two separate writes leave a window where money has left and not arrived, and a crash in that window needs a saga to clean up. One WriteRequest naming both aggregates commits both or neither, provided they live on the same shard. See Consistency boundaries.

A transfer

using Celeriant.Client.Requests;
using Celeriant.Client.Serialization;

var json = JsonEventSerializer.Default;

await pool.WriteAsync(new WriteRequest
{
    ClientId = writerId,
    Writes = new Dictionary<AggregateKey, SingleAggregateWrite>
    {
        [fromAccount] = new()
        {
            Events = [AggregateEventExtensions.Create(TransferredOutType, new TransferredOut(amount, toId), json)],
            ExpectedVersion = fromVersion,
        },
        [toAccount] = new()
        {
            Events = [AggregateEventExtensions.Create(TransferredInType, new TransferredIn(amount, fromId), json)],
            ExpectedVersion = toVersion,
        },
    },
});

ClientId sits on the request and covers every aggregate in it. Each SingleAggregateWrite carries its own events, ExpectedVersion, AllowCreate and EnforceClientIdempotency.

AllowCreate defaults to false here, unlike the single-aggregate WriteAsync overload where it defaults to true. Set it when the write may create an aggregate.

What atomic means

The server validates every aggregate's guard, existence and idempotency before it enqueues anything. One failure rejects the whole request and no aggregate changes. After validation, each aggregate gets one new batch, all enqueued together and acknowledged together after fsync and replication.

A stale guard on any aggregate throws WriteOccException (2003). Wrap the write in the same read-decide-write loop: catch up every aggregate, re-decide, resubmit. WriteResponse.MaxAggregateVersion is null for a multi-aggregate write; read versions back from the stream or AggregateDetailsAsync.

The shard rule

Every aggregate in one request must route to the same shard. The server checks this when it routes the request, before any shard looks at events or guards, and rejects a split request with 9001 ShardRoutingMultipleShards (ShardRoutingException in .NET, with the cluster's shard count in NumShards).

Placement is plain modulo, no hash:

routing_id = org_id | aggregate_type_id | aggregate_id   (per --routing-rule)
shard      = routing_id % num_shards

The routing id is the Guid read as a 128-bit big-endian integer, which is its canonical hex string read as one number. With --reserve-coordinator-shard, shard 0 carries no data and the formula becomes routing_id % (num_shards - 1) + 1. --routing-rule, --num-shards and --reserve-coordinator-shard are all fixed when the data directory is first initialised; the node refuses to start if they change.

9001 is not a conflict. Retrying the same keys fails the same way every time.

Picking the routing rule

Aggregates you co-commit	Rule	Cost
Any two in the same org (accounts in one tenant)	org_id	That tenant's writes serialise on one shard
Any two of the same aggregate type	aggregate_type_id	That type's writes serialise on one shard
Mostly single-aggregate writes	aggregate_id (default)	Co-committed aggregates need ids chosen to share a shard

Under org_id and aggregate_type_id, co-location is automatic within the org or type and still possible across them, if those ids share a modulus.

Co-locating ids under aggregate_id

Random Guids scatter evenly, which is what you want for single-aggregate writes and exactly what breaks pairs. If two aggregates will ever commit together, mint the second id on the first one's shard:

using System.Globalization;

// Mirrors the server's data_shard_for_routing_id.
static int ShardOf(Guid routingId, int numShards, bool reserveCoordinatorShard = false)
{
    if (numShards <= 1) return 0;
    var id = UInt128.Parse(routingId.ToString("N"), NumberStyles.HexNumber);
    return reserveCoordinatorShard
        ? (int)(id % (UInt128)(numShards - 1)) + 1
        : (int)(id % (UInt128)numShards);
}

static Guid NewIdOnSameShard(Guid partner, int numShards)
{
    int target = ShardOf(partner, numShards);
    while (true)
    {
        var candidate = Guid.NewGuid();
        if (ShardOf(candidate, numShards) == target)
            return candidate;
    }
}

That takes num_shards draws on average. The client has no call that reports the shard count; take it from your deployment config (it cannot change after init) or from ShardRoutingException.NumShards.

This only works for pairs you know about at allocation time. Two accounts created independently and later asked to transfer will usually sit on different shards. If arbitrary pairs must co-commit, aggregate_id is the wrong rule; route by org_id or aggregate_type_id.

Troubleshooting 9001

1. The rule does not match the invariant: a transfer across orgs on an org_id cluster, say. Pick the rule that groups what you co-commit. The rule is fixed at init, so that means a new cluster.
2. aggregate_id routing with ids that were never co-located. Allocate them together as above.