Concepts
The verification model
Trellara's central claim is that convergence is checkable rather than assumed. A claim like that is only worth anything if its scope is stated precisely, so this page is mostly about the edges.
What verify compares
trellara verify --config trellara.yml trellara verify --config trellara.yml --table public.sales
| Check | What it establishes |
|---|---|
| Row counts | Source and target hold the same number of rows per table. |
| Primary-key checksums | The same keys are present, and their covered content matches — not merely the same count. |
| Source and target watermarks | The target has applied through the source LSN it claims to have applied through. |
| Snapshot handoff state | The run reached stream_handoff_ready at a single consistent LSN before streaming began. |
| Duplicate accounting | Replayed duplicates were skipped by transaction dedup rather than applied twice. |
The result is CONVERGED, or a named divergence with a recovery path. There is deliberately no third answer and no partial credit — “probably fine” is the state this command exists to eliminate.
It is on demand, not continuous: it tells you about the moment you ran it. It compares the tables in the flow's configuration, not the whole database. It cannot detect divergence in columns outside the checksum's coverage on a table without a usable primary key. And it says nothing about how long recovery took — recovery time has never been measured.
Verification as a state transition
Verification is not only a report. It is the only thing that can promote a
snapshot run from stream_handoff_ready to
verified, and it is allowed to move a run from
verified back to
failed_recoverable when new evidence shows a previous
handoff was unsafe. A run whose verification fails stays handoff-ready or
recoverable — never verified. See
snapshot and handoff.
The other half: the failure matrix
verify tells you about your system right now.
The failure matrix tells you about the implementation, on every change,
before it reaches you. They prove different things and neither substitutes for the
other.
trellara chaos run
trellara chaos report --output report.html
63 deterministic scenarios across 28 simulations in five failure-point families. Each scenario names its protected invariant, its proof command and its recovery command — the structure is what makes them usable as a runbook source rather than just a test count.
| Family | Scenarios | Covers |
|---|---|---|
| Base | 8 | Publish/ack loss, crash after publish before source ack, source failover in the ack window, duplicate delivery, target failure before commit, quarantine repair replay, checkpoint failure during apply, stream ack loss after apply. |
| Snapshot | 6 | Source, relay and target crashes during table copy; duplicate copy attempts; DDL during copy; handoff recorded before stream start. |
| Strict chunk | 5 | Relay crash after chunks before manifest and after manifest before source ack; manifest with a missing chunk; manifest arriving before its chunks; target crash after staging before commit. |
| Qualification | 5 | Release qualification labels. |
| Fleet fan-in | 4 | Multi-source completeness and straggler handling. |
Deterministic simulation is a real and unusually strong form of evidence, and it is not the same thing as a soak run against production hardware. There has been no soak run. Recovery is proven correct and has never been timed. The optional native extension has a live CI lane across PostgreSQL 15, 16, 17 and 18 — each job installs it on a real server and runs a crash and acknowledgement suite. The external relay, which is the default path, has no CI Postgres lane at all: its integration tests are environment-gated and only PostgreSQL 16 has been exercised end to end, by hand.
Taking it to a review
The evidence package exists because “we ran the tests” is not forwardable to someone who was not on the call.
trellara pilot-scorecard --config trellara.yml --format text trellara pilot-package --config trellara.yml --output ./pkg
The package collects convergence evidence with checksums, recovery drill results, contract tests with pinned fingerprints, and the failure matrix. The snapshot evidence tables are read directly rather than inferred from logs, so the package reflects durable state rather than whatever the log buffer happened to hold.
The current, honest inventory of what has and has not been established is the evidence table. It is the right place to end an evaluation of this page's claims.