schema records expectations, not transformations

What sources.<id>.schema is

schema is optional metadata describing the shape a source is expected to expose.

It is not a transformation step. In the current executor the block is parsed and validated, but it does not cast, select, or reject runtime rows; use transformations for those effects.

What it contains

• Declaration mode — auto, manual, infer_strict, or hybrid.
• Field descriptions — Names, types, nested paths, required/default metadata, and array handling.
• Policies — Drift, validation, and sampling settings validated by the source model.

Expected is not transformed

A schema tells readers what the source is meant to look like. Only transformation steps change the rows, so keep operational casts and selections in transformations.yaml.

Run the complete example

Copy this manifest from the verified documentation project, then run it from the repository root.

version: "1.0"
sources:
  src_orders:
    type: csv
    connection:
      base_path: "data/canonical"
    extract:
      table: orders.csv
      batch_size: 10000
    schema:
      mode: manual
      fields:
        - name: order_id
          type: string
          required: true
        - name: amount
          type: float
      drift_policy: warn
      validation_policy: best_effort
      sample_size: 100
  src_orders_extra:
    type: csv
    connection:
      base_path: "data/canonical"
    extract:
      table: orders_extra.csv
      batch_size: 10000

hdrctl run examples/tutorial/dsl-elements

  ✅ Pipeline completed successfully in 2.6s
  Rows read   : 30
  Rows written: 6

The relevant command syntax, copied from hdrctl run --help:

hdrctl run [PATH]
  -s, --sources FILE          explicit path to sources.yaml
  -d, --destinations FILE     explicit path to destinations.yaml
  -t, --transformations FILE  explicit path to transformations.yaml
  -P, --param KEY=VALUE       override a parameter, repeatable

Close by

• type — connector contract
• extract — read request
• Columns — cast and select
• sources.yaml — the full declaration
• Reference — schema fields and policies