Synchronising a master customer set

Context — a current master must absorb a corrected feed

The task is currently manual and its assumptions are not recorded. Hydra turns it into files that can be reviewed and rerun.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you express this rule with Hydra’s merge operation and prove exactly what it changes?

• Use one minimal input fixture
• Declare one operation per step
• Validate the DSL before running
• Compare the complete before and after states

The solution in one line

One merge step between a deterministic CSV source and a replaceable CSV destination.

Steps

1. capture the before state

1. Use a minimal fixture that exposes the rule.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

master: C-1,Paris · C-2,Lyon
corrections: C-2,Lille · C-3,Nantes

2. declare the fixture

1. Read the known input.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  input:
    type: csv
    connection:
      base_path: "data"
    extract:
      table: input.csv

3. add the merge step

1. Write one operation with explicit parameters.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
steps:
  - merge:
      right: corrections
      key: customer_id
      delete_unmatched: false

4. make the result observable

1. Write a replaceable CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace

---
version: "1.0"
pipeline:
  from: input
  to: result

5. validate the operation

1. Ask Hydra to parse the step.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test merge-workshop

ok  transformations.yaml — 1 step(s) valid
Operations: merge

✅ All tests pass — ready to execute.

6. run and compare after to before

1. Inspect the exact output.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run merge-workshop

✅ Pipeline completed successfully

customer_id,city
C-1,Paris
C-2,Lille
C-3,Nantes

Expected result

• The after state matches the stated rule.
• One operation is counted by `hdrctl test`.
• The fixture can be rerun without accumulating rows.

Reading the results

Compare columns, row count and ordering—not just one visible value. Matching rows update, new keys insert, and unmatched left rows stay unless explicitly deleted.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

Matching rows update, new keys insert, and unmatched left rows stay unless explicitly deleted.

Did we answer the question?

Before and after

• Before — a current master must absorb a corrected feed requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops