An ordered list, and the only file you may omit

Four shapes this file takes, all correct. Walk them, then read what each one does.

transformations.yaml4 steps

the file contains

What this example does

The figures below come from running the real engine on this exact manifest.

✓ 30 rows read · 6 written

What `transformations.yaml` is

transformations.yaml holds the ordered list of operations applied between reading and writing. One root key, transformations, containing one key, steps, containing a list.

It is the only file of a job you may omit. Without it the rows travel from the source to the destination unchanged — a copy job, which is a legitimate thing to want.

What it contains

A list, not a set. Position is meaning. Each entry receives what the previous one produced, never the source.
One operation per entry — a mapping with a single key, the operation name, whose value holds its parameters. Eighteen names are accepted; anything else stops the run.
An optional label. A step may carry a name, used in messages. It changes nothing to the result.

Order is the whole idea

The list runs from top to bottom and the engine never reorders it. A CSV yields text in every column, so a step that compares numbers belongs after the step that produces them — which is why the example starts with cast. Note that hdrctl validate checks the shape of the manifest, not the types flowing through it.

Each step sees what the previous one produced, never the source.

Two shapes, one file

The engine accepts two shapes for this file — the same manifest written two ways. Both are complete below: copy either one and it works.

Long form — the transformations key wraps steps. Used throughout this documentation.

transformations:
  steps:
    - cast:
        mapping:
          amount: float
    - filter:
        expr: "amount > 50"

Short form — steps sits at the root. Used by the templates of hdrctl init.

steps:
  - cast:
      mapping:
        amount: float
  - filter:
      expr: "amount > 50"

Run both, and compare

The tutorial project carries the long form. Save the short one as short-form.yaml and point hdrctl run at it with -t: no file is modified.

hdrctl run examples/tutorial/01-first-job
hdrctl run examples/tutorial/01-first-job -t short-form.yaml

Both print the same three lines:

  ✅ Pipeline completed successfully
  Rows read   : 30
  Rows written: 6

The full syntax of the command, with the options that matter here:

hdrctl run <job-directory>
    -t, --transformations FILE   read the steps from this file instead
    --dry-run                    validate the manifest, write nothing
    -v, -vv, -vvv                more detail, up to full logs

Pick a shape per project and hold it. Nothing in the engine will tell you that you drifted, since both are equally valid.