Wire protocol

This is reference material for people writing a client or debugging the wire. If you use the .NET or Rust client, you never touch this.

Transport

Plain TCP on the client port (--client-port, default 10000), with TCP_NODELAY set on every accepted socket. Replication between nodes runs on a separate port (--replication-port, default 10001) with its own message set. Clients never speak it.

A connection carries one request at a time: the server reads a request, writes its response, then reads the next. Want more throughput? Open more connections. A watch request turns the connection into a one-way push stream for the rest of its life.

Any shard can accept a connection. When a request belongs to another shard, the server hands the socket to that shard's core and the conversation continues there.

Frame layout

Every message in both directions is a 17-byte header followed by a body. Header integers are little-endian. No magic number, no checksum.

Offset	Size	Field	Notes
0	4	version	2 (bincode) or 3 (MessagePack). Anything else is rejected.
4	4	message_type	Type id, table below.
8	4	compressed_length	Body length on the wire.
12	4	uncompressed_length	Body length after decompression. Must equal compressed_length when uncompressed.
16	1	compression	0 none, 1 zstd with the cluster dictionary.

The server checks both lengths against its cap (--max-request-size on the client port, 16 MiB default) before it allocates or decompresses anything, so compression buys no headroom. An oversized, malformed or unknown-type frame gets no error response. The server closes the connection.

Body encoding

The version field picks the body encoding:

• V2: bincode, fixed-width integers, little-endian. The Rust client uses it.
• V3: MessagePack as rmp_serde emits it: structs are positional arrays (field order matters, names are not sent), u128 ids are bin8 of 16 big-endian bytes, None is nil. The .NET client uses it.

The server speaks both on the same port, per connection. The first message pins the version; a later frame with a different version drops the connection.

Message types

Id	Request	Response on success	Size class (req / resp)
1	AggregateDetailsRequest	AggregateDetailsResponse (1)	fixed / fixed
2	ReadRequest	ReadResponse (2)	fixed / variable
3	WriteRequest	WriteResponse (3)	variable / fixed
4	TrimStartRequest	TrimStartResponse (4)	fixed / fixed
5	DeleteRequest	DeleteResponse (5)	fixed / fixed
6	WatchRequest	stream of WatchResponse (8)	fixed / variable
7	ListOrgsRequest	ListOrgsResponse (9)	fixed / variable
8	ListAggregateTypesRequest	ListAggregateTypesResponse (10)	fixed / variable
9	ListAggregatesRequest	ListAggregatesResponse (11)	fixed / variable
10	RegisterSchemaRequest	RegisterSchemaResponse (12)	variable / fixed
14	IdentifyRequest	IdentifyResponse (16)	fixed / variable

Request and response type ids are separate spaces; the response id is the number in parentheses. Any request can instead get ErrorResponse (response type 7): correlation_id, a numeric error_code, and an error_message holding a JSON object with the error's context. See error codes. Type 6, ProtocolErrorResponse, is defined but the server does not send it; it closes the connection instead.

Fixed-size bodies must fit in 1007 bytes (a 1024-byte frame minus the header) and are never compressed. That cap bites. A DeleteRequest naming many aggregates, a ReadRequest with a long event-type list, or a WatchRequest with large filter sets will not fit, and the client fails before sending. Variable-size bodies are bounded only by the request or response cap.

Compression

The cluster has one zstd dictionary, the same one the WAL uses (dictionary.zstd_dict in --data-root, level --wal-compression-level, default 3). Clients do not pick an algorithm. The dictionary ships in the Identify handshake:

1. The client sends known_dict_sha256 if it has a dictionary cached.
2. IdentifyResponse always carries compression_dict_sha256, and carries compression_dict_bytes only when the client's sha does not match.
3. From then on both sides compress variable-size bodies of 1024 bytes or more.

No Identify, no dictionary: a connection that skips the handshake gets every response uncompressed.

The identify handshake

IdentifyRequest (type 14) is accepted only as the first message on a connection. It is optional unless the server runs with --require-client-identity. Then any other first message gets IDENTIFY_REQUIRED (10004) and the connection closes.

The request is fixed-size, so the whole body must fit in 1007 bytes.

Field	Content
correlation_id	Optional u128, echoed back.
public_key	Base64 DER SubjectPublicKeyInfo (RSA).
nonce	Current UTC epoch milliseconds as a decimal string.
signature	Base64 RSASSA-PKCS1-v1_5 SHA-256 signature over the nonce's bytes.
api_key	Base64 of exactly 32 bytes.
known_dict_sha256	Hex sha of the client's cached dictionary.

Two independent checks. Both run when both are present:

• RSA identity. The server rejects a nonce more than 2 minutes old or more than 60 s in the future (10001), verifies the signature (10002), then derives the client id from the first 16 bytes of SHA-256(DER public key), read little-endian. Same key, same id, every time. For the rest of the connection a write, trim, delete or register-schema carrying a different client_id gets IDENTIFY_MISMATCH (10003). An RSA-4096 key plus signature overflows the 1007-byte body; use RSA-2048.
• API key. Checked only when api_keys.toml exists in the data root. The server hashes the 32 decoded bytes with SHA-256 and looks the hash up. No key is 10005; bad base64, wrong length or unknown hash is 10006. A read-only key then gets 10007 on any write, trim, delete or register-schema.

IdentifyResponse returns client_id (set only by RSA identity), access_level (ReadWrite or ReadOnly, set only when API keys are configured) and the dictionary fields. See Identity and authentication.

API keys need --require-client-identity

API keys are checked inside the Identify handshake and nowhere else. Without --require-client-identity, a client that never sends Identify is never asked for a key and gets read-write access. Run the two together.

TLS

--tls-mode strict rejects plaintext on both ports; the default disabled is plaintext only. TLS 1.3, no fallback. The server runs the handshake in userspace with rustls, then hands the record layer to the kernel (kTLS), so clients see ordinary TLS. --tls-client-auth is require (mTLS, the default), optional or none. See TLS and mTLS.

The server refuses to start with API keys or --require-client-identity but no TLS, unless you pass --insecure-allow-plaintext-auth. Without TLS the API key crosses the wire in the clear and a captured signed nonce replays for up to 2 minutes.

Pre-1.0

The server is at 0.2.0. The frame carries a protocol version, but message layouts can still change between releases. Build against a client library unless you are prepared to track the wire.