Wire one source to one destination

A job reads from exactly one source and writes to exactly one destination. pipeline.yaml names them.

pipeline.yamlpick the two ends

pipeline:
  from: src_input
  to: dest_output

from to

What the runner sees

Both names must resolve to something declared, and in the right direction.

src_inputsources.yaml

transformationssteps[]

dest_outputdestinations.yaml

from resolves to a declared source

to resolves to a declared destination

direction is source → destination

✓ Manifest resolves

pipeline

Written by hand: this block has no Pydantic model in the engine, it is parsed directly by the runner.

2 keys

key	type	required	default	constraint
from	string	yes	—	an id declared in `sources`
to	string	yes	—	an id declared in `destinations`

A job is atomic. Several sources or several destinations is the role of a workflow, not of pipeline.

from and to hold declared identifiers. The file path lives in extract.table and load.table.