Check the manifests before touching data
Validate syntax, Pydantic models, and pipeline references before Hydra opens a source or writes a destination.
1. Objective
Goal
Run hdrctl validate first-job and use its report to prove that every required manifest is present, each DSL model is valid, and both pipeline endpoint IDs resolve.
Keep working from the cli-course folder. The job is named first-job; every command and terminal excerpt in this lesson was run against that same project.
Prerequisites
- Lesson 2 completed and its checklist confirmed.
- The repository clone and hdrctl entry point from lesson 1.
- A terminal opened in the parent folder that contains
first-job.
2. Steps
01Review the validation surface
Objective
Confirm the supported path and strictness options before running the command.
Actions
- Ask the subcommand for help.
- Verify that the job path is positional and that
--strictis optional.
hdrctl validate --help
The help lists the path argument plus strict and explicit-manifest options.
Usage: hdrctl validate [OPTIONS] [PATH] Options: --strict -s, --sources FILE -d, --destinations FILE -p, --pipeline FILE -t, --transformations FILE
02Validate the cumulative job
Objective
Check all generated manifests and their cross-file references in one pass.
Actions
- Stay in the folder that contains
first-job. - Run hdrctl validate against that job directory.
hdrctl validate first-job
Every manifest is present, every Pydantic model is valid, and both endpoint IDs resolve.
🔍 DSL Validation first-job ok pipeline.yaml — present ok sources.yaml — Pydantic valid ok destinations.yaml — Pydantic valid ok transformations.yaml — Pydantic valid ok pipeline.from — resolved: src_csv ok pipeline.to — resolved: dest_csv ✅ DSL valid — no errors detected.
03Trace the resolved IDs
Objective
Connect the validator verdict to the exact declarations it checked.
Actions
- Open
pipeline.yamland notesrc_csvanddest_csv. - Find those IDs under
sourcesanddestinations.
Get-Content first-job\sources.yaml Get-Content first-job\destinations.yaml
The IDs named by the pipeline exist once in their respective manifests.
sources:
src_csv:
type: csv
destinations:
dest_csv:
type: csv
04Record the validation boundary
Objective
Distinguish a DSL verdict from a connector test.
Actions
- Confirm that no output file was created by validation.
- Keep the successful verdict as the baseline for the next lesson.
Get-ChildItem first-job\data
Only the scaffolded input is required at this stage; validation does not write data.
input.csv
Tips and traps
Keep the command beside its result. A verdict alone proves nothing a week later.
The Valid badge means the manifest parses. Only a run proves the job works.
Hydra resolves relative paths from the job folder, not from where you typed.
Trust the exit code, not the wording. A reassuring message can follow a failure.
When a command writes a file, open the file. Output can be stale or partial.
Durations and row counts change between runs. Never quote them as thresholds.
3. Checklist
| Verification | Expected result |
|---|---|
| Final validator line | DSL valid — no errors detected. |
| Source model | sources.yaml — Pydantic valid |
| Destination model | destinations.yaml — Pydantic valid |
| Pipeline references | Both src_csv and dest_csv resolve |
4. Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| A manifest is reported missing | The command targets the wrong folder | Run from the parent folder and pass first-job |
| A pipeline ID does not resolve | The ID differs across files | Make pipeline.from and pipeline.to match the manifest keys exactly |
| YAML parsing fails | Indentation or quoting changed the structure | Restore spaces and compare with the scaffolded shape |
5. Next lesson
The declarations are internally consistent, but that does not yet prove that the local CSV paths can be used. The next lesson runs hdrctl test to check sources, destinations, transformations, and environment resolution without producing rows.