sources.yaml is a catalogue, not a build list

What sources.yaml is

sources.yaml is the file that declares every input the job may read. One root key, sources, holding a mapping: each key is an identifier you invent, each value is a declaration of one system to read from.

It is one of the three required files of a job, next to destinations.yaml and pipeline.yaml. Only transformations.yaml may be omitted.

What it contains

• Identifiers — names local to the job. pipeline.from points to one of them; join.right, merge.right and union.right may point to others.
• Declarations — four keys each: type names the connector, connection says how to reach the system, extract what to read, schema how to type it. Their shape is the subject of the connectors page.

Declared is not opened

This is the idea worth keeping. The file is a catalogue, not a build list. The engine instantiates a connector only when something references its identifier — so a second source sitting in the file untouched is never built, its type never checked, its file never looked for.

What the engine requires

• The file must exist and declare at least one entry. It is one of the three required files of a job, next to destinations.yaml and pipeline.yaml.
• Every referenced identifier must exist. That link is checked from pipeline, which names the two ends of the job.
• Unreferenced entries are not checked at all. Convenient while you draft, worth a second look before you commit.

Close by

• sources.<id>.connection — reaching the system
• pipeline — which source feeds which destination
• version — the first line of the file
• Lesson 1 — the four files of a job
• Reference — every key of every block