Reference

Configuration

Generated · exhaustive Every key the config parser accepts, extracted from the serde structs. Anything not on this page is rejected at parse time.

The document

A flow config is one YAML document with config_version pinned at 2. Every struct on this page is deny_unknown_fields: a typo in a key is an error, not a setting that is silently ignored.

Key	Type	Requirement
`config_version`	`u16`	required
`environment`	`ConfigurationEnvironment`	required
`source`	`SourceConfig`	required
`dataset`	`DatasetConfig`	required
`stream`	`StreamConfig`	required
`target`	`TargetConfig`	optional

source

Key	Type	Requirement
`id`	`String`	required
`database_url`	`SensitiveString`	required
`database_id`	`String`	optional
`capture`	`SourceCaptureKind`	defaulted
`wal_retention_warn_bytes`	`i64`	optional
`stream_spill_threshold_changes`	`usize`	optional
`stream_spill_dir`	`PathBuf`	optional
`pgoutput`	`PgOutputProtocolConfig`	defaulted
`publication`	`String`	required
`slot`	`String`	required

dataset

Key	Type	Requirement
`id`	`String`	required
`mode`	`DatasetMode`	required
`unknown_table_policy`	`UnknownTablePolicy`	defaulted
`tables`	`TableConfig`	required
`partition`	`PartitionConfig`	optional
`strict_chunking`	`StrictChunkConfig`	optional

dataset.tables[]

Key	Type	Requirement
`schema`	`String`	required
`name`	`String`	required
`verify`	`TableVerifyConfig`	optional
`contract`	`TableContractConfig`	optional

dataset.tables[].verify

Key	Type	Requirement
`primary_key`	`String`	defaulted
`excluded_columns`	`String`	defaulted
`row_filter`	`String`	optional

dataset.partition

Key	Type	Requirement
`partition_count`	`u32`	required
`key_column`	`String`	required
`null_key_policy`	`PartitionNullKeyPolicy`	defaulted
`key_change_policy`	`PartitionKeyChangePolicy`	defaulted

dataset.strict_chunking

Key	Type	Requirement
`max_changes_per_chunk`	`u32`	required

stream

Tagged by kind. Exactly two transports.

kind	Keys
`local`	`path` `consumer_group` `durability`
`kafka`	`bootstrap_servers` `topic` `consumer_group` `profile`

stream.profile (Kafka)

development or production. Under environment: production only production is accepted, and it carries the contract below.

Key	Value in contract v1
`ca_certificate`	secret reference — environment variable or absolute file path
`authentication`	`mutual_tls` or `sasl_tls` with `plain` · `scram_sha256` · `scram_sha512`
`replication_factor`	≥ 3
`min_insync_replicas`	≥ 2, and never above the replication factor
`producer_acks_all`	`true`
`producer_idempotence`	`true`
`consumer_auto_commit`	`false`
`store_offset_after_apply`	`true`
`synchronous_offset_commit`	`true`

target

Optional. Absent for a lake-only flow.

Key	Type	Requirement
`database_url`	`SensitiveString`	required

Every enumerated value

Type	Values	Notes
`ConfigurationEnvironment`	`development` · `production`	default `development`
`DatasetMode`	`strict-transaction-order` · `partitioned-scale-mode`	—
`KafkaProfile`	`development` · `production`	default `development` · tagged by `kind`
`LocalStreamDurability`	`fsync` · `buffered`	default `fsync`
`PartitionKeyChangePolicy`	`quarantine` · `emit-move` · `dual-write-window` · `forbid`	—
`PartitionNullKeyPolicy`	`quarantine` · `route-to-dead-letter-partition` · `route-to-singleton-partition` · `derive-from-primary-key`	—
`SourceCaptureKind`	`pg-output` · `test-decoding`	default `pg-output`
`StreamConfig`	`kafka` · `local`	tagged by `kind`
`UnknownTablePolicy`	`reject` · `allow-compatible`	default `reject`

Defaults and limits

Setting	Default	Bound
`source.pgoutput.protocol_version`	`2`	1 with streaming off, or 2
`source.pgoutput.streaming`	`true`	—
`source.stream_spill_threshold_changes`	`1,024`	max 1,000,000
`stream.durability`	`fsync`	`fsync` or `buffered`
`dataset.unknown_table_policy`	`reject`	`reject` or `allow_compatible`
`dataset.partition.null_key_policy`	`quarantine`	four policies
`dataset.partition.key_change_policy`	`quarantine`	planner fails closed unless `emit_move`

How this page is produced

Generated from the repository at 347a884 by tools/gen_reference.py. If this page and the code disagree, the code is right and this page is a bug.

Configuration

The document

source

dataset

dataset.tables[]

dataset.tables[].verify

dataset.partition

dataset.strict_chunking

stream

stream.profile (Kafka)

target

Every enumerated value

Defaults and limits

On this page