extract narrows what the connector reads

What sources.<id>.extract is

extract is the read request handed to the selected source connector.

It does not identify the connector or describe credentials. Those belong to type and connection; extraction begins only after both are resolved.

What it contains

• A target — Use table, query, or collection according to the connector.
• Optional narrowing — filter and limit constrain supported reads.
• Delivery size — batch_size defaults to 10,000 and must be at least 10 in the current parser.

Connection reaches; extract chooses

A connection can reach an entire system. extract makes one concrete request inside it: this file, table, query, or collection, delivered in these batches.

Run the complete example

Copy this manifest from the verified documentation project, then run it from the repository root.

version: "1.0"
sources:
  src_orders:
    type: csv
    connection:
      base_path: "data/canonical"
    extract:
      table: orders.csv
      batch_size: 10000
    schema:
      mode: manual
      fields:
        - name: order_id
          type: string
          required: true
        - name: amount
          type: float
      drift_policy: warn
      validation_policy: best_effort
      sample_size: 100
  src_orders_extra:
    type: csv
    connection:
      base_path: "data/canonical"
    extract:
      table: orders_extra.csv
      batch_size: 10000

hdrctl run examples/tutorial/dsl-elements

  ✅ Pipeline completed successfully in 2.6s
  Rows read   : 30
  Rows written: 6

The relevant command syntax, copied from hdrctl run --help:

hdrctl run [PATH]
  -s, --sources FILE          explicit path to sources.yaml
  -d, --destinations FILE     explicit path to destinations.yaml
  -t, --transformations FILE  explicit path to transformations.yaml
  -P, --param KEY=VALUE       override a parameter, repeatable

Close by

• type — choosing the connector
• connection — reaching the system
• schema — describing expected fields
• pipeline.yaml — selecting this source
• Reference — all extract keys