Scaffold your first job
Create the cumulative CSV project from Hydra's built-in template, inspect every generated manifest, and make its numeric filter executable.
1. Objective
Goal
Use hdrctl init with the official csv template to create first-job. Then read the generated manifests and add the one explicit type conversion required by the sample data before later lessons validate and run it.
Keep working from the cli-course folder. The job is named first-job; every command and terminal excerpt in this lesson was run against that same project.
Prerequisites
- Lesson 1 completed and its checklist confirmed.
- The repository clone and hdrctl entry point from lesson 1.
- A terminal opened in the parent folder that contains
first-job.
2. Steps
01Create the CSV job
Objective
Generate the complete file set from the built-in CSV template.
Actions
- From cli-course, run hdrctl init with the job name
first-job. - Select the documented
csvtemplate; do not create the files by hand.
hdrctl init first-job --template csv
Hydra reports the six files it created and gives the next commands.
Initializing: first-job [template: csv] ok sources.yaml ok destinations.yaml ok pipeline.yaml ok transformations.yaml ok data/input.csv ok README.md ✅ Job 'first-job' created successfully.
02Read the pipeline endpoints
Objective
Confirm which source and destination IDs the generated pipeline connects.
Actions
- Open
first-job/pipeline.yaml. - Match
fromtosrc_csvandtotodest_csv.
Get-Content first-job\pipeline.yaml
The manifest contains only the generated source-to-destination flow.
pipeline: from: src_csv to: dest_csv
03Inspect the sample rows
Objective
See the exact values the later run will read and filter.
Actions
- Open
first-job/data/input.csv. - Notice that the third row has a negative value and should be removed.
Get-Content first-job\data\input.csv
The sample contains three rows, including one negative value.
id,name,value 1,Alice,100 2,Bob,200 3,Charlie,-5
04Make the numeric type explicit
Objective
Cast value before the generated filter compares it with zero.
Actions
- Open
first-job/transformations.yaml. - Insert a
caststep afterselectand beforefilter. - Save the file with the mapping shown below.
steps:
- select:
columns: [id, name, value]
- cast:
mapping:
value: float
- filter:
expr: "value > 0"
The ordered transformation list now contains select, cast, and filter.
steps:
- select:
columns: [id, name, value]
- cast:
mapping:
value: float
- filter:
expr: "value > 0"
Tips and traps
Keep the command beside its result. A verdict alone proves nothing a week later.
The Valid badge means the manifest parses. Only a run proves the job works.
Hydra resolves relative paths from the job folder, not from where you typed.
Trust the exit code, not the wording. A reassuring message can follow a failure.
When a command writes a file, open the file. Output can be stale or partial.
Durations and row counts change between runs. Never quote them as thresholds.
3. Checklist
| Verification | Expected result |
|---|---|
first-job folder | Contains four YAML manifests, README, and data/input.csv |
pipeline.yaml | References src_csv and dest_csv |
transformations.yaml | Casts value to float before filtering |
4. Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| The folder already exists | A previous scaffold is present | Choose an empty course folder; use --force only when overwriting is intentional |
| A generated file is missing | The template name or working directory was wrong | Run hdrctl init --help, then scaffold again with --template csv |
| The cast follows the filter | Steps execute in order | Move cast immediately before filter |
5. Next lesson
The project now has real input data, resolved endpoint IDs, and a numeric transformation chain. The next lesson asks the strict DSL validator to check those declarations before any connector opens.