Build on Trellara

Protocol specification

Developer The wire contract, field by field, for anyone writing a consumer that is not ours.

The transaction envelope is the neutrality claim. If it is a private encoding, “destination-neutral” is marketing; if it is a protobuf schema you can generate from, it is a property of the system. This page is the second thing.

Where the authority is

The schema exists twice and both must agree: crates/trellara-protocol/proto/trellara/v1/envelope.proto is the IDL, and the prost attributes on the Rust types are what the code encodes. If this page ever disagrees with either, they win. A conformance suite that would let you prove your consumer correct does not exist yet — it is named in the sidebar rather than quietly implied.

Version

PROTOCOL_VERSION is 1. An envelope whose protocol_version differs is rejected on decode. There is no best-effort interpretation of an unknown version, and a consumer you write should behave the same way.

The envelope

Thirteen fields. The tag numbers are the compatibility surface — treat them as fixed.

Tag	Field	Type	Rule
1	`protocol_version`	uint32	Must equal 1.
2	`source_id`	string	Required, non-blank.
3	`database_id`	string	Required, non-blank.
4	`dataset_id`	string	Required, non-blank.
5	`transaction_id`	string	Required. Every change and DDL event must repeat it.
6	`begin_lsn`	string	Optional. When present, must parse and be `≤ commit_lsn`.
7	`commit_lsn`	string	Required, canonical, non-zero.
8	`commit_timestamp_ms`	int64	Required, greater than zero.
9	`schema_versions`	repeated	Relation schema identities the changes reference.
10	`changes`	repeated	Ordered `ChangeRecord` values.
11	`manifest`	optional	Present only for chunked and partitioned boundaries.
12	`checksum`	uint64	`xxh3_64` of the encoded message with this field zeroed.
13	`ddl_events`	repeated	Ordered DDL events sharing the `total_order` space with `changes`.

One order across DML and DDL

total_order is a single sequence covering both changes and ddl_events. A duplicate total_order between a change and a DDL event in the same transaction is a validation error.

That rule is what stops an event class from disappearing during chunking or reconstruction: you cannot reassemble a transaction, count only its rows, and conclude it is complete. Operator surfaces render the two counts and their sum as dml=N ddl=M source_total=N+M; source_total is a rendering, not a wire field, so do not look for it in the schema.

ChangeRecord

Field	Notes
`transaction_id`	Must match the envelope.
`total_order`, `table_order`, `partition_order`	Three orderings, for the transaction, the relation and the partition lane.
`relation`	Relation identity.
`operation`	`insert`, `update`, `delete`, `truncate`.
`replica_identity`	`default`, `index`, `full`, `nothing`.
`before`, `after`	Optional row images. A column may be marked unchanged-TOAST; see below.
`idempotency_key`	Required, non-blank.

Event identity

The canonical ordered event key is deterministic from four values:

{source_id}:{commit_lsn}:{transaction_id}:{total_order}

TransactionBoundaryKey additionally carries database_id, which participates in its display form but not in the event key. Deduplicate at the transaction level; the record-level key exists so inspection and lake planning are deterministic, not because apply needs it.

Checksum

Compute it by cloning the message, setting checksum to zero, encoding, and hashing the bytes with xxh3_64. The reference implementation validates then verifies before producing bytes, and verifies then validates on the way in — so a consumer never sees a partially-trusted envelope. Do the same.

Manifests and commit markers

A plain strict transaction carries no manifest. Chunked and partitioned boundaries carry both a manifest and a commit marker, and the marker binds the manifest by checksum so the two cannot be mispaired.

`TransactionManifest`	Purpose
`transaction_id`, `source_commit_lsn`, `source_commit_timestamp_ms`	Identity, repeated from the envelope.
`global_event_count`	Total events the complete transaction must contain.
`partitions`	Non-empty. Each entry: `id`, `event_count`, `first_total_order`, `last_total_order`, `checksum`.
`affected_tables`	Per-relation event counts.
`boundary_mode`	`strict_chunked_transaction_order` or `partitioned_scale_mode`.

Manifest validation rejects a zero first_total_order or last_total_order, a reversed range, a zero event_count, an event_count larger than the declared range span, and a last_total_order above global_event_count. Implement all of them: they are what makes a truncated or fabricated manifest detectable.

TransactionCommitMarker repeats the identity, LSN, timestamp and global_event_count, adds participating_partition_count, and carries manifest_checksum.

Rules a correct consumer follows

Reject an unknown protocol version. Do not interpret it.
Verify the checksum before you trust any field.
Never expose a partial transaction as committed state. If a manifest is present, wait for every listed chunk plus the matching commit marker.
Treat an absent non-key column as unchanged when the row image marks it unchanged-TOAST. Writing null there silently destroys data, and it is the single most common way to get this wrong.
Fail closed on a missing key column for an update or delete. There is no best-effort path.
Deduplicate by transaction identity. The stream is at-least-once by design; a duplicate after a relay crash is the chosen failure mode, and your consumer is where it is absorbed.

Reading the local log directly

If you are consuming the brokerless log rather than a broker, the frame format is a four-byte magic, a u32 body length, the encoded body, and a u32 CRC32 of the body. The current magic is TLG2; TLG1 frames remain readable so an existing log survives an upgrade. Any other magic, a body length above the 128 MiB cap, or a checksum mismatch is corruption — not a record to skip. A short read at a frame boundary is a torn tail, and the correct response is to stop at the last complete valid frame.