Hydra ETL
Build your first job
Guide›Sources›csv
Workshop 1 · 20 minutes

Ingesting the daily sales export

A shop receives one CSV a day and opens it by hand. Nine steps to automate the reading, the typing and the filtering — then read what the run tells you.

Context — a shop that closes its day in Excel

A retailer exports its orders every evening from the point-of-sale software. The file lands in a folder, someone opens it in a spreadsheet, converts the amounts, keeps the large orders and mails the result to the sales manager. Fifteen minutes a day, every day, done by hand.

Where things stand

  • One file a day — sales_2025-03-04.csv, seven orders, six columns.
  • Everything is text. A CSV has no types: 120.50 is a string until someone converts it.
  • No trace. Nobody can say afterwards which file produced which figure.
  • Manual filtering. The rule — keep orders of 100 and above — lives in a person's head.

The question

How do you read that file automatically, type its columns, apply the rule and write the result — without opening a spreadsheet, and in a way that runs the same tomorrow?

  • Read the file where it sits, without copying or moving it
  • Turn the amounts into numbers before comparing them
  • Keep only the orders of 100 and above
  • Write a file that can be regenerated without piling up

The solution in one line

A Hydra job: four small YAML files in one folder, one command to check them, one command to run.

data/sales_2025-03-04.csv9 steps

      

    

    

      
      
      
    

  


  

    

      
      
What you do
Create the folder and drop the export inside it.

    

    

    

    

      Step 1 · nothing to run yet
      
    

  



 

  
Steps


  
1. Create the folder and drop the export

  
    
    
  1. Create a folder sales/ wherever you work.
    2. 
    
  2. Inside it, create data/ and out/.
    3. 
    
  3. Put the export in sales/data/sales_2025-03-04.csv.
    4. 
  

  
mkdir -p sales/data sales/out

  
Check — the file is at sales/data/sales_2025-03-04.csv and has a header line.

  
Tip — Hydra never moves or copies your file. It reads it where it sits, so the layout is yours to choose.


  
2. Declare the source

  
    
    
  1. Create sales/sources.yaml.
    2. 
    
  2. Name the entry src_sales — you choose this name.
    3. 
    
  3. Set type: csv, then base_path to the folder and extract.table to the file.
    4. 
  

  
version: "1.0"
sources:
  src_sales:
    type: csv
    connection:
      base_path: "sales/data"
    extract:
      table: sales_2025-03-04.csv
      batch_size: 10000

  
Trap — base_path is resolved from the folder you run the command in, not from the job folder. Run everything from the folder that contains sales/, or write an absolute path.


  
3. Declare the destination

  
    
    
  1. Create sales/destinations.yaml.
    2. 
    
  2. Name the entry dest_large_orders.
    3. 
    
  3. Set mode: replace.
    4. 
  

  
version: "1.0"
destinations:
  dest_large_orders:
    type: csv
    connection:
      base_path: "sales/out"
    load:
      table: large_orders.csv
      mode: replace

  
Tip — the default is append. With replace the output stays at three rows however many times you run; with append it would grow every evening.


  
4. Wire the two ends

  
    
    
  1. Create sales/pipeline.yaml.
    2. 
    
  2. Write the two identifiers declared above — not paths.
    3. 
  

  
version: "1.0"
pipeline:
  from: src_sales
  to: dest_large_orders

  
Check — three files now, and the job is already runnable. It would copy the seven rows unchanged.


  
5. Type the amounts

  
    
    
  1. Create sales/transformations.yaml.
    2. 
    
  2. Write steps at the root, then one cast entry.
    3. 
  

  
version: "1.0"
steps:
  - cast:
      mapping:
        amount: float

  
Trap — hdrctl test only counts the steps of this short shape. Written under a transformations key it reports 0 step(s) valid and says nothing about it.


  
6. Apply the business rule

  
    
    
  1. Add a second entry after the cast: keep the orders of 100 and above.
    2. 
  

  
  - filter:
      expr: "amount >= 100"

  
Trap — the order of the two steps is the whole point. A CSV holds text, so the comparison only makes sense after the cast has produced numbers.


  
7. Check the manifest

  
    
    
  1. Run hdrctl test from the folder containing sales/.
    2. 
  

  
hdrctl test sales

  
  Sources
  ok  sources.yaml         — DSL valid
  ok  src_sales            — csv (local file)

  Destinations
  ok  destinations.yaml    — DSL valid
  ok  dest_large_orders    — mode replace

  Transformations
  ok  transformations.yaml — 2 step(s) valid
  Operations: cast, filter

  ✅ All tests pass — ready to execute.

  
Check — 2 step(s) valid and the two operations named. No data has been read at this point.


  
8. Run the job

  
    
    
  1. Run hdrctl run on the same folder.
    2. 
  

  
hdrctl run sales

  
  ✅ Pipeline completed successfully
  Rows read   : 7
  Rows written: 3


  
9. Read the result

  
    
    
  1. Open sales/out/large_orders.csv.
    2. 
  

  
order_id,customer,city,amount,status,order_date
A-1001,Alice Martin,Paris,120.5,paid,2025-03-04
A-1003,Chloe Ruiz,Madrid,310.75,paid,2025-03-04
A-1006,Farid Ziani,Nantes,260.0,paid,2025-03-04


  
Expected result

  
    
    
  • Seven rows read, three written. The four orders below 100 were dropped by the filter.
    • 
    
  • The amounts are numbers. 120.50 is written 120.5 and 260.00 becomes 260.0 — the trailing zero is a text artefact, not a value.
    • 
    
  • The file can be regenerated. Run it twice: still three rows.
    • 
  


  
Reading the results


  
Why seven and three?

  

    The two counters bracket the job. Rows read is what the source produced, Rows written
    what reached the destination. The gap is the work of the steps — here a filter that kept 3 of 7,
    or 43 % of the orders, which is the share of the day above 100.
  


  
What would happen without the cast?

  

    The filter would compare text to a number and the run would stop at step 1 of 2, before writing
    anything. Nothing would be lost — the destination is only touched once every step has passed.
  


  
What the manifest does not know

  

    Nothing here says today's file. The name is written in extract.table, so the
    job reads that one file. Making the date a parameter is the subject of the next workshop.
  


  
Did we answer the question?

  

    
objectiveresultwhere

    
      
Read without copyingyesbase_path points at the folder in place

      
Type the amountsyescast, first step

      
Keep orders ≥ 100yesfilter, second step

      
Regenerable outputyesmode: replace

      
Read today's file automaticallynot yetthe date is written in the manifest

    
  


  
Before and after

  
    
    
  • Before — open the file, convert a column, sort, filter by hand, copy the result out. Around
      fifteen minutes, every evening, and the rule exists only in the head of whoever does it.
    • 
    
  • After — one command. The rule is written in transformations.yaml, readable by
      anyone, versioned with the rest, and it produces the same result every time.
    • 
    
  • What is really gained is not the fifteen minutes. It is that the rule left the spreadsheet
      and became text a colleague can read, review and correct.
    • 
  


  
Close by

  


 
  
 
 
      
  0 / 0 on this page