Lesson 1 of 7

Your first job

Read a CSV, keep the large orders, write the result. Four files, one command.

1. What to Expect

Objectives

You will see the four files that make up every Hydra job.
You will read a CSV, convert a column, filter rows and write the result.
You will run the job and read what the engine reports.

Description

We start from orders.csv, thirty rows of a small shop. Amounts are stored as text, as they always are in a CSV. We convert them to numbers, keep the orders above fifty, drop the columns we do not need, and write what remains.

This project exists in the repository under examples/tutorial/01-first-job. Every block below is copied from it, and the output shown is the output it produced.

Processing Steps

Read orders.csv as a CSV source.
Cast amount from text to a number.
Keep the rows above fifty, then select four columns.
Write the result to big_orders.csv.

2. Implementation

Walk the four files with Next. The highlighted line is the one being explained.

sources.yaml 10 steps

What the engine produced

Thirty rows in, six out. These are the real rows, not an illustration.

big_orders.csv6 of 30 rows

order_id	customer_id	amount	status

✓ Pipeline completed · 30 read · 6 written

3. Execution

From the root of the repository:

hdrctl run examples/tutorial/01-first-job

The engine reports what it did:

  ▶  Pipeline  01-first-job  — examples/tutorial/01-first-job

  ✅ Pipeline completed successfully in 3.3s
  Rows read   : 30
  Rows written: 6

Before running anything, hdrctl validate reads the four files and reports what it finds without touching the data. It does not type-check expressions, so a comparison between a text column and a number passes validation and fails at run time — that is exactly why cast comes before filter here.

4. Next Lesson

Lesson 2 chains more operations on the same project: computing a column, renaming it, and sorting the result.