Concepts

Completeness epochs

Developer · Operator Turning “is last night’s data complete?” into a query with an answer.

Every tool in this category reports freshness. Freshness answers “how far behind am I,” which is not the question a revenue job needs answered. The question is: which sources are in this number, and which are not?

Maturity

Lake fan-in is in progress. Planning, DDL, epoch decisions, writer intents and Spark templates ship today. The production catalog writer is deliberately narrow — append-only, Iceberg REST catalog, S3-compatible storage, no native deletes. A continuously running multi-source fleet runtime is the next milestone, not a description of today.

The guarantee being made

For a warehouse epoch, Trellara can say which sources are included, through which source LSN, with which transaction and row counts, with which checksums, and which sources are lagging, missing, quarantined or reseeding.

Note what that is not: it is not a promise that every source was online. It is a promise that the answer is recorded rather than assumed.

Four terms

Term	Meaning
Source	One PostgreSQL origin with a stable `source_id`. In retail usually a store; in SaaS often a tenant, region, shard or customer database.
Dataset	The logical flow identity. Binds selected tables, schema contracts, partition policy, stream topics and lake output names.
Epoch	A named lakehouse visibility boundary. Usually time-shaped — hourly, nightly — but proven by source LSNs and manifest completeness, not by clock time.
Watermark	A source or partition progress boundary: start LSN, end LSN, transaction count, change count, checksum rollup, and optionally a per-partition low watermark.

The transaction boundary rule carries all the way up to this layer: lake fan-in never upgrades partial transaction evidence into a complete epoch. A transaction whose manifest is incomplete does not make its source complete, no matter how close to the boundary it got.

Epoch states

State	Meaning	Default consumption
`open`	Still accepting transaction boundaries.	Do not consume.
`sealing`	Closed to new boundaries; completeness is being computed.	Do not consume.
`complete`	Every required source reached the boundary, every transaction boundary is complete, metadata written.	Safe to consume.
`complete_with_gaps`	Published under policy with explicit missing, lagging or offline sources.	Safe only for consumers that accept gaps.
`quarantined`	Contains conflicting, corrupt, incomplete or schema-unsafe evidence.	Do not consume.
`reseeding`	One or more sources need a snapshot before future epochs can be trusted.	Do not consume for affected sources.
`failed_recoverable`	Writer or catalog failure before confirmed visibility; replay can resume from the durable stream.	Do not consume until replay succeeds.

Source states — complete, lagging, missing, quarantined, reseeding — are described under quarantine and reseed.

Straggler policy

What should happen when a required source has not arrived is a business decision, not a technical default, so it is configured and recorded per epoch.

Policy	Behaviour	Fits
`wait_all_required`	Keep the epoch non-consumable until every required source is complete.	Finance, compliance, end-of-day revenue.
`publish_with_gaps`	Publish with explicit missing or lagging source rows after the grace window.	Operational analytics and ML jobs that tolerate late stores.
`quarantine_on_gap`	Quarantine the epoch when a required source is absent.	Correctness-critical jobs where gaps are unacceptable.

Policy is data, not convention

The policy is written into the epoch metadata. A consumer must read it from the row and must not infer it from the table name or the schedule. Two epochs in the same table can carry different policies if the configuration changed between them.

Querying it

_trellara_epochs holds one row per epoch, alongside companion tables for sources, table rollups, quarantine and verification. The epoch row carries at least:

Column	Purpose
`epoch_id`	Stable epoch identity.
`dataset_id`	Dataset or flow identity.
`state`	Epoch state from the table above.
`policy`	The straggler policy this epoch was published under.
`opened_at` / `sealed_at`	Epoch open and seal timestamps.
`required_source_count`	How many sources were required.
`complete_source_count`	How many reached `complete`.

Which turns the original question into a join rather than a conversation:

SELECT state, policy, required_source_count, complete_source_count
FROM _trellara_epochs
WHERE dataset_id = 'retail_sales' AND epoch_id = '8821';

trellara lake epoch --config fleet.yml --format text
trellara lake fanin verify --config fleet.yml

What is out of scope

Generic single-source Postgres-to-Iceberg positioning. That market is commoditised and Trellara is not aimed at it.
Automatic source DDL propagation to lake targets.
Native Rust current-state Iceberg upserts and deletes.
Merge-on-read and equality-delete authoring, until Iceberg Rust support and partner demand justify it.
Owning a compaction service.
Multi-primary conflict resolution.

Current-state and SCD2 tables are derived from completed epochs by maintained Spark SQL and PySpark templates that you run, using the engine you already operate.