The connection block changes shape with the type

What connection is

connection answers one question: how does the engine reach the data? It holds the coordinates and the credentials of a system — a folder, a database server, an HTTP host — and nothing else. It performs no reading, applies no operation, and never contains a row.

It sits inside a source or a destination, beside two other keys. type names the connector that will read it. extract — or load on a destination — says what to read once the connection is established. connection only says where and with what rights.

What it contains

The keys differ from one type to the next, but they always play one of five roles.

• Location — where the system is. base_path for CSV, file for JSON and Parquet, host and port for databases, base_url for a web API.
• Identity — who is asking. user and password for databases, the auth block — type, token — for a web API. Always a reference such as ${SECRET:DB_PASSWORD}, never a literal.
• Scope — which part of the system. database, schema on PostgreSQL, collection on MongoDB.
• Format — how to decode the bytes. delimiter, encoding, quotechar for CSV.
• Transport — how to fetch, in several trips. The pagination block for a web API: strategy, cursor_field, page_size.

Two consequences follow. A connector reads only the keys it knows, so a key meant for another type is ignored rather than rejected — a misspelling has no effect and raises no error. And the block is never correct on its own: it is correct for a type.

Close by

• Lesson 1 — a source inside a job that runs
• pipeline — which source feeds which destination
• Interpolation — how a secret gets in
• Reference — every key of every block
• Guide — connecting to a real database