Hydra ETL
Build your first job
Lesson 3 of 14 · Read the canvas

Recognize nodes, ports and edges

Take a guided tour of first-workflow: two jobs, two actions, three containers, and the YAML each of them produces.

Jobs configurationabout 20 minutesAPI online

1. Objective

Goal

Open first-workflow and read everything it holds without changing anything. Tell Jobs configuration, where a job runs source to destination, from Workflow configuration, where jobs and actions form a graph. Recognize the three containers and what each writes into the manifest.

Why this matters

One screen hosts two different graphs. Confusing them is the most common beginner mistake: connector nodes belong inside a job, Job and Action nodes belong to orchestration. Reading before editing is what makes the next nine lessons quick.

Course project

Keep one cumulative Studio project named studio-course, one manual workflow named first-workflow, and one job named first-job. Each lesson builds on the state saved by the previous one.

What you are looking at

PieceWhere it livesWhat it does
first-jobJobs configurationReads a CSV, keeps some rows, writes a CSV
second-jobJobs configurationReads the first job's output, keeps some columns, writes a CSV
Log actionWorkflow configurationWrites a message to the run log
PowerShell actionWorkflow configurationRuns a shell command on the host
Three containersBoth scenesGroup nodes, and two of them apply a policy

Prerequisites

  • Hydra API started with hdrctl serve.
  • Studio started from Hydra/studio with npm run dev.
  • A project holding first-workflow with its two jobs — or the bundle below.
Follow along with the real workflow The workflow read in this lesson, with both jobs and their sample data. Unzip it, then use Import workflow and pick the import_bundle folder. hdrctl workflow run works on it too.
Download the bundle 4 KB

2. Steps

01Open the workflow

Objective

Read what the card announces before entering the editor.

Actions

  1. Open the project from Overview.
  2. Read the line under the workflow name.
  3. Select Open in editor.
Overview → project → first-workflow → Open in editor
What you should see

The card summarises the workflow in one line: its trigger and how many jobs it orchestrates. Draft and never run describe the file and its history, not its validity.

first-workflow
Manual · 2 jobs
Draft   never run
The project page showing the first-workflow card, marked Manual with two jobs, Draft and never run, with a Trigger selector and an Open in editor link.
Counting 2 jobs is not the same as counting nodes — actions and containers are not jobs.

02Read the workflow graph

Objective

Identify the four executable nodes and the direction of the edges between them.

Actions

  1. Stay on the Workflow configuration tab.
  2. Read the counters beside the workflow name.
  3. Follow the arrows from left to right, then select the PowerShell node.
Workflow configuration → select a node
What you should see

Five nodes for four steps: the fifth is the container, which never executes. Each node carries a blue handle on the left and on the right — an edge always leaves a right handle and enters a left one, so the graph reads in one direction only.

first-workflow   5 nodes · 3 edges   Valid
first-job → second-job → action_log_1
                      → action_powershell_2
The workflow canvas showing an Error Scope container around first-job and second-job, with a Log action and a PowerShell action to the right, and the inspector panel open on the PowerShell node.
Selecting a node opens the inspector: step name, node type, on failure and the read-only node ID.

03Read the containers

Objective

Understand what a container is, and what it is not.

Actions

  1. Look at the header of the red container around the two jobs.
  2. Open its On child failure list.
  3. Close it again without changing the value.
Error Scope → On child failure

A container never executes. Hydra's runner never sees one. What it does is copy a policy onto the nodes it encloses when the YAML is written — so you set the rule once, in one place, instead of on every step.

ContainerOption on its headerWhat each child receives
Sequencenonenothing — it only groups the canvas
Error ScopeOn child failure: skip · continue · failon_failure
Retry ScopeRetries N × · delay N sretry
What you should see

Three values, and only three. fail stops the workflow, skip marks the step skipped, continue carries on. A child that sets its own value keeps it — the container fills in the blanks, it does not overrule.

On child failure:
  skip
  continue
  fail
The Error Scope container header with the On child failure list open, showing the three values skip, continue and fail.
Deleting the container does not delete its children — but the policy it was applying goes with it.

04Open the first job

Objective

Switch scenes and read a job as a straight line from source to destination.

Actions

  1. Select the Jobs configuration tab.
  2. Pick first-job in the tab bar at the bottom.
  3. Read the header of the container around the first two nodes.
Jobs configuration → first-job
What you should see

Three executable nodes in a row — source, transformation, destination — plus the container. The palette also changes: Sources, Transformations and Destinations replace Actions and Jobs, because those have no meaning inside a job.

first-workflow / first-job   4 nodes · 2 edges   Valid
orders.csv → transform_filter_3 → firtered_orders.csv
Retry Scope   Retries 4 ×, delay 5 s
The first-job canvas with a Retry Scope container set to four retries and five seconds delay, holding a CSV source and a Filter transformation, with the CSV destination outside the container.
The destination sits outside the Retry Scope: only what the container encloses receives the policy.

05Open the second job

Objective

Confirm the shape repeats, and meet the container that changes nothing.

Actions

  1. Select second-job in the tab bar.
  2. Read the three nodes and the two edges.
  3. Look at the container header — it has no option.
Jobs configuration → second-job
What you should see

The same source-transformation-destination line, this time reading the file the first job wrote. The Sequence container carries no setting because it applies no policy: it is there to make a long chain readable.

first-workflow / second-job   4 nodes · 2 edges   Valid
src_orders_clean → transform_select_4 → dst_sales_by_region
Sequence   (no option)
The second-job canvas with an orange Sequence container holding a CSV source, a Select transformation and a CSV destination.
Two jobs, one shape. Every Hydra job is one source, N transformations, one destination.

06Read what a node is configured with

Objective

Open the four connector nodes and see that each holds only what its role needs.

Actions

  1. Double-click the source of first-job.
  2. Close it, then open the Filter node, then the destination.
  3. Switch to second-job and open its source.
double-click a node → read → Cancel

Every dialog shows the same three parts: a step name you choose, the fields required by that node type, and a read-only node ID. The name is what appears in the YAML; the ID is what the canvas uses internally and never changes.

The CSV source dialog, showing a step name, a required file path with a Browse button, and a read-only node identifier.
Source. A CSV needs one thing: a file path. Browse avoids typing it.
The Filter transformation dialog, with a step name and a required expression field holding amount greater than 100.
Transformation. Filter keeps the rows for which the expression is true.
The CSV destination dialog, with a step name and a required output file path.
Destination. Same dialog as the source, opposite direction.
The CSV source dialog of the second job, whose file path points at the output written by the first job.
The link between the two jobs. This source reads what the first job wrote — nothing else connects them.
What you should see

No dialog asks for anything it does not need. That is the whole idea of the node types: the form is the contract.

CSV  SOURCE        step name · file path · node ID
Filter  TRANSFORMATION   step name · expression · node ID
CSV  DESTINATION   step name · file path · node ID

07Read the two actions

Objective

See what an action is, and why one of them is tied to the operating system.

Actions

  1. Return to Workflow configuration.
  2. Open the Log node, read its message, cancel.
  3. Open the PowerShell node and read its command, working directory and timeout.
Workflow configuration → double-click an action
The Log action dialog with a step name, an on-failure list, an optional run condition and a message field.
Log always succeeds. Useful to mark a milestone in the run log.
The PowerShell action dialog with a step name, an on-failure list, a run condition, the required command field, a working directory and a timeout in seconds.
PowerShell runs on the host. The timeout is what stops a command that never returns.

Both dialogs carry a Run condition (when). Left empty, the step always runs; filled in, it runs only when the expression is true. That is how a Condition node upstream steers the rest of the graph.

Trap — the shell action does not adapt to the host

Hydra does not swap PowerShell for Bash when it runs on Linux. Each action refuses the wrong platform outright: a PowerShell step fails on Linux and macOS, a Bash step fails on Windows. If a workflow has to run on both, replace the node yourself, or guard each one with a Run condition. A workflow that is portable everywhere is a workflow with no shell action in it.

What you should see

The same frame around a different payload: a message for Log, a command for PowerShell. Only on failure and run condition are shared by every action.

Log         ACTION   message
PowerShell  ACTION   command · working directory · timeout (60 s)

08Read the DSL the canvas produces

Objective

Connect what you have been reading on screen to the four files on disk.

Actions

  1. With a job open, select Hydra DSL in the top bar.
  2. Walk through the four tabs in order.
  3. Close the panel.
Jobs configuration → Hydra DSL

A job is not one file but four, and the panel gives one tab to each. Read them in this order and the job explains itself: where the data comes from, what happens to it, where it goes, and how the three are wired together.

The Hydra DSL panel on the Sources tab, showing a csv source with its connection and extract table and batch size.
sources.yaml — the source node, with the file path under extract.table.
The Hydra DSL panel on the Transformations tab, showing a steps list with a select operation and its columns.
transformations.yaml — an ordered list. The order on the canvas is the order here.
The Hydra DSL panel on the Destinations tab, showing a csv destination with its load table and replace mode.
destinations.yaml — note mode: replace, which empties the target before writing.
The Hydra DSL panel on the Pipeline tab, showing from and to fields referring to the declared source and destination.
pipeline.yamlfrom and to hold the identifiers, never the file paths.
What you should see

Identifiers, not paths, in pipeline.yaml. That indirection is why a source can be repointed at another file without touching the wiring.

sources:          source id  → type, connection, extract
transformations:  steps      → one operation per entry
destinations:     dest id    → type, connection, load
pipeline:         from: <source id>   to: <dest id>

09Read the workflow the canvas describes

Objective

See the whole orchestration as one file, and find each thing you read on screen inside it.

Actions

  1. Compare each name below with a node on the canvas.
  2. Compare each depends_on with an edge.
  3. Look for the containers — and notice they are absent.
hdrctl workflow validate ./workflow.yaml

Four steps, in the order the graph imposes. first-job starts alone, second-job waits for it, and the two actions both wait for second-job — so they run in parallel, because nothing links them to each other.

version: "1.0"
workflow:
  name: first-workflow
  trigger:
    type: manual
  steps:
    - name: first-job
      type: job
      job: ./jobs/first-job
      depends_on: []
      on_failure: skip

    - name: second-job
      type: job
      job: ./jobs/second-job
      depends_on: ["first-job"]
      on_failure: skip

    - name: action_log_1
      type: action
      action: log
      params:
        message: "test the log"
      depends_on: ["second-job"]

    - name: action_powershell_2
      type: action
      action: powershell
      params:
        command: Get-Process
        timeout: 60
      depends_on: ["second-job"]

The Error Scope is nowhere in this file, and neither are the other two containers. All that survives is the on_failure: skip it wrote on each of its children. That is the whole contract: you draw a container, Hydra writes a policy.

On the canvasIn the file
A nodean entry under steps
An edgea name inside depends_on
A Job nodetype: job and a job path
An Action nodetype: action, its action and its params
An Error Scopeon_failure on each child
A Retry Scoperetry on each child
A Sequence Containernothing at all
What you should see

Two jobs, two actions, no container. The step count matches the four executable nodes, not the five drawn on screen.

✅ Workflow valid — no errors detected.
Tips and traps

Sources and transformations live in Jobs configuration. Jobs and actions live in Workflow configuration.

A container never runs. Five nodes on screen can mean four steps in the YAML.

An edge leaves a right handle and enters a left handle. Direction is never ambiguous.

The Valid badge means the manifest parses. Only a run proves the job works.

A policy reaches only what sits inside the container. A node beside it gets nothing.

Studio can be ahead of the disk. Open the file to be sure.

Counts and durations are live values. Never quote them as thresholds.

An animation is not an outcome. Wait for a terminal state on the Runs page.

3. Checklist

VerificationExpected result
Workflow counters5 nodes · 3 edges for four executable steps
Scene switchThe palette changes with the tab
ContainersSequence, Error Scope and Retry Scope identified
Error Scope optionsskip, continue, fail — three values
Job shapeSource, transformation, destination in both jobs
Hydra DSLFour tabs: sources, transformations, destinations, pipeline
Nothing changedThe workflow is still marked Valid

4. Troubleshooting

SymptomCauseFix
The palette has no SourcesYou are on Workflow configurationSwitch to Jobs configuration
No Actions in the paletteYou are inside a jobActions belong to orchestration; switch scenes
A node ignores its containerIt sits beside the container, not inside itDrag it into the frame until the header highlights
Node count exceeds step countContainers are counted as nodesExpected — they are not steps
A PowerShell step fails on LinuxThe action is bound to WindowsReplace it with a Bash action, or guard it with a run condition
An edge refuses to connectYou started from a left handleDrag from a right handle to a left one

5. Next lesson

You can now read a workflow without touching it. The next lesson opens the node palette properly — what each category holds, what the counts mean, and how to find the node you need.

Close by

0 / 0 on this page