Hydra ETL
Build your first job
Lesson 4 of 14 · Use the palette

Everything Hydra can do, in one panel

The palette is not a list of icons. It is the complete vocabulary of the product: seven connectors, eighteen transformations, nine actions, three containers — and your own jobs.

Node paletteabout 20 minuteseditor open

1. Objective

Goal

Open both palettes, read every category, and leave knowing what Hydra can do without writing a line of YAML. By the end you should be able to answer one question for any task: is there a node for this, and in which scene does it live?

Course project

Keep one cumulative Studio project named studio-course, one manual workflow named first-workflow, and one job named first-job. Each lesson builds on the state saved by the previous one.

Prerequisites

  • Hydra API started with hdrctl serve.
  • Studio started from Hydra/studio with npm run dev.
  • A workflow open in the editor.

2. Steps

01Discover the palette

Objective

See that there are two catalogues, not one, and read what each offers.

Actions

  1. Open the Workflow configuration tab and read the four categories.
  2. Switch to Jobs configuration and read them again.
  3. Note the number beside each category name.
Workflow configuration ⇄ Jobs configuration

The palette follows the scene. Orchestration offers what a workflow can do; a job offers what data can go through. Nothing is hidden — what a scene does not show, it cannot do.

The node palette in the workflow scene, showing four collapsed categories: Actions, Control Flow, Containers and Jobs, each with a count.
Workflow scene. Actions, Control Flow, Containers, Jobs.
The node palette in the job scene, showing four collapsed categories: Sources, Transformations, Destinations and Containers.
Job scene. Sources, Transformations, Destinations, Containers.
What you should see

Only Containers appears in both. Every other category belongs to one scene and one only.

Workflow    Actions (9)  Control Flow (2)  Containers (3)  Jobs (2)
Job         Sources (7)  Transformations (18)  Destinations (6)  Containers (3)

02Read what a workflow can do

Objective

Discover that a workflow is not limited to running jobs.

Actions

  1. Expand Actions.
  2. Read the nine entries from top to bottom.
  3. Leave it expanded for the next steps.
Actions (9)

This is where a data pipeline becomes an operational one. A job moves rows; an action tells someone, calls something, runs a command, or remembers a value.

ActionWhat it does
LogWrites a message to the run log. Always succeeds.
WebhookSends an HTTP request — notify a service, trigger a build.
EmailSends a message when a load finishes or fails.
Bash · PowerShellRuns a shell command on the host that carries Hydra.
PythonRuns Python — inline or a .py file. Works on every platform.
SSHRuns a command on another machine.
Set Param · Assign ParamCreate and update values that later steps can read.
What you should see

Nine entries, one of them dimmed. That dimming is the subject of the next step.

The Actions category expanded, listing Log, Webhook, Email, Bash, PowerShell, Python, SSH, Set Param and Assign Param, with Bash dimmed and carrying a Unix badge while PowerShell carries a Win badge.
Nine actions. Bash is greyed out here, and it carries a badge.

03Tell someone, or tell something

Objective

Meet the three actions that carry news out of the workflow.

Actions

  1. Open Log and read its subtitle.
  2. Open Webhook and read the method list.
  3. Open Email and note which fields carry a red asterisk.
Actions → Log · Webhook · Email

These three answer the same question — who needs to know? — at three distances. Log speaks to whoever reads the run afterwards. Webhook speaks to another system. Email speaks to a person who is not watching.

ActionRequiredWorth knowing
LognothingThe only action that always succeeds. Safe to place anywhere, including after a failure.
Webhooka URLPOST by default, with a JSON body. Choose another method from the list when the service expects one.
Emailrecipient and subjectSends over SMTP. The server, port and credentials belong to the step, so each workflow can use its own.
The Log action dialog, whose subtitle states that it writes a message to the run logs and always succeeds, with a free-text message field.
Log. Even the message is optional — the step is a marker as much as a sentence.
The Webhook action dialog, described as an outgoing HTTP call, with a required URL, a method list set to POST and a JSON body field.
Webhook is the outgoing HTTP call — not to be confused with the incoming webhook trigger.
The Email action dialog, with required recipient and subject fields and a body.
Email. Recipient and subject are required; the body is not.
What you should see

Three subtitles that say exactly what each one does, and no more.

Log       ACTION   Writes a message to the run logs (always succeeds)
Webhook   ACTION   Outgoing HTTP call
Email     ACTION   Send email

04The palette knows which machine it runs on

Objective

Read the platform badges, and understand why one shell is always unavailable.

Actions

  1. Find Bash and PowerShell in the list.
  2. Read the badge on the right of each: Unix or Win.
  3. Note which of the two is dimmed.
Actions → Bash · PowerShell

Hydra runs the command on the machine that hosts it, so only one of the two shells can ever work. The palette does not hide the other — it shows it, dimmed, with the platform it needs. You always know the whole vocabulary, and what your host is missing.

The Actions list on a Windows host: PowerShell is active with a blue Win badge, Bash is dimmed with a grey Unix badge.
Hydra on Windows. PowerShell is available, Bash is dimmed.
The same Actions list on a Linux host: Bash is active with a blue Unix badge, PowerShell is dimmed with a grey Win badge.
Hydra on Linux or macOS. The pair is simply reversed.
Trap

Hydra never substitutes one shell for the other. A workflow carrying a PowerShell step fails on Linux, and a Bash step fails on Windows — by design, with a clear message. If a workflow must run on both, use Python instead: it is the only interpreter available everywhere.

What you should see

Both dialogs are the same, only the command differs. The subtitle states the platform outright.

Bash        ACTION   Runs a Bash script (Linux / macOS)
PowerShell  ACTION   Runs a PowerShell script (Windows)
The Bash action dialog, with a step name, an on-failure list, a run condition, the required Bash command field and a working directory.
Beyond the command, every action shares the same frame: on failure, run condition, working directory.

05Run Python, on any host

Objective

Find the one interpreter that never depends on the platform, and its two ways of receiving code.

Actions

  1. Open the Python action.
  2. Select Inline script and read the editor.
  3. Select .py file and read the fields that replace it.
Actions → Python → Inline script | .py file

Python carries no platform badge, because there is nothing to warn about: it runs on Windows, Linux and macOS alike. Where a shell action ties a workflow to one host, this one keeps it portable — which is why it is the answer whenever a workflow has to travel.

ModeWhat you provideWhen to choose it
Inline scriptThe code, typed in the dialogA few lines that belong to this workflow and nowhere else. Everything stays in one file.
.py fileA path, plus a working directoryReal code — one you test, review and version separately. The workflow references it instead of copying it.
The Python action dialog in Inline script mode, with a code editor holding an import of sys and a print of the version.
Inline script. The code lives in the workflow file itself.
The same Python action switched to .py file mode, showing a required file path with a Browse button and a working directory.
.py file. The editor gives way to a path and a working directory.
What you should see

One toggle, two shapes, the same action. Parameters set earlier reach the script as environment variables, so you rarely need to hard-code a value.

Python  ACTION  Runs a Python script
  Inline script   →  PYTHON SCRIPT *
  .py file        →  .PY FILE PATH *   WORKING DIRECTORY

06Reach outside the machine, and carry values across steps

Objective

Meet the three actions that make a workflow more than a sequence of loads.

Actions

  1. Open SSH and read the fields it requires.
  2. Open Set Param, then Assign Param.
  3. Compare the two subtitles word for word.
Actions → SSH · Set Param · Assign Param

SSH runs a command on another machine — restart a service, move a file, trigger a job on a server Hydra does not live on.

Set Param creates a value; Assign Param changes one that already exists. The distinction is deliberate: creating twice is a mistake, and Hydra refuses it rather than overwriting something you forgot about. Later steps read the value with {{ param:name }}, and shell and Python actions also receive it as an environment variable.

The SSH action dialog, requiring a host, a port defaulting to 22 and a user.
SSH. Host and user are required; the port defaults to 22.
The Set Param dialog, whose subtitle reads that it creates a runtime parameter and fails if it already exists, with a parameter name, a value and a type.
Set Param creates — and fails if the name already exists.
The Assign Param dialog, whose subtitle reads that it assigns a value to an existing runtime parameter.
Assign Param updates a parameter that already exists.
What you should see

Two dialogs that look alike and mean the opposite. Read the subtitle, not the shape.

Set Param      Creates a runtime parameter (fails if it already exists)
Assign Param   Assigns a value to an existing runtime parameter

07Steer the graph, and group what it holds

Objective

Find the two categories that shape execution rather than perform it.

Actions

  1. Expand Control Flow and read its two entries.
  2. Expand Containers in the workflow scene.
  3. Switch to Jobs configuration and expand Containers there too.
Control Flow (2) · Containers (3)

Condition evaluates an expression and lets downstream steps decide whether to run; Delay simply waits. Containers do not execute at all — they group nodes and hand a policy down to whatever they enclose.

The Control Flow category expanded, showing Condition and Delay.
Control Flow. Two entries, and both change when things run rather than what runs.
The Containers category expanded in the workflow scene, showing Sequence Container, Error Scope and Retry Scope.
Containers, workflow scene. Sequence, Error Scope, Retry Scope.
The same three containers offered in the job scene, below Sources, Transformations and Destinations.
Containers, job scene. The same three — this is the only category shared by both.
What you should see

Three containers in both scenes, and the same names. Their effect differs: in a workflow they write a policy onto steps, in a job they only tidy the canvas.

Control Flow   Condition   Delay
Containers     Sequence Container   Error Scope   Retry Scope

08Your own jobs become palette entries

Objective

See that the palette is not fixed — it grows with the project.

Actions

  1. Expand Jobs in the workflow scene.
  2. Read the names listed there.
  3. Look at the tab bar at the bottom of the job scene.
Jobs (2)

Every job you design in this project appears here, ready to be dropped into any workflow. That is what makes a job reusable: written once in Jobs configuration, orchestrated as many times as needed. The count is the size of your own library, not a product limit.

The Jobs category expanded in the workflow palette, listing first-job and second-job.
The two jobs of the project, offered as nodes.
The job tab bar at the bottom of the editor, with a tab per job and a plus button to add one.
The tab bar switches between jobs. + creates one, and it joins the palette.
What you should see

The job names you created, nothing else. An empty project shows Jobs (0) and says so in plain words.

Jobs (2)
  first-job
  second-job

09Read the connectors, in and out

Objective

Learn what Hydra can read from and write to, and spot the one asymmetry.

Actions

  1. In Jobs configuration, expand Sources.
  2. Expand Destinations.
  3. Compare the two lists entry by entry.
Sources (7) · Destinations (6)

Six connectors work in both directions: three file formats and three databases. The seventh, Web API, reads only — Hydra pulls from an HTTP endpoint but does not push to one. To send data out over HTTP, use the Webhook action in the workflow scene instead.

The Sources category expanded, listing CSV, JSON, Parquet, MySQL, PostgreSQL, MongoDB and Web API.
Sources. Three file formats, three databases, one HTTP API.
The Destinations category expanded, listing CSV, JSON, Parquet, MySQL, PostgreSQL and MongoDB.
Destinations. The same six, without Web API.
What you should see

Seven on the way in, six on the way out. The difference is one entry, and it is not an oversight.

Sources        CSV  JSON  Parquet  MySQL  PostgreSQL  MongoDB  Web API
Destinations   CSV  JSON  Parquet  MySQL  PostgreSQL  MongoDB

10Eighteen ways to reshape data

Objective

Read the full transformation list once, so you stop reinventing what already exists.

Actions

  1. Expand Transformations.
  2. Scroll to the end of the list.
  3. Note the last entry, Script.
Transformations (18)

They fall into four families. Knowing which family you need is usually faster than remembering a name.

FamilyOperations
ChooseFilter · Select · Deduplicate
ChangeRename · Cast · Derive · Fill Null · Trim · Clean
ReshapeSort · Aggregate · Pivot · Unpivot · Transpose
CombineJoin · Merge · Union
The Transformations category expanded, showing Filter, Select, Rename, Cast, Aggregate, Sort, Deduplicate, Derive and Join.
The first nine.
The rest of the transformation list, showing Fill Null, Trim, Clean, Pivot, Unpivot, Transpose, Merge, Union and Script.
And the last nine, ending on Script.
What you should see

Seventeen named operations, then one that is not an operation at all.

Filter  Select  Rename  Cast  Aggregate  Sort  Deduplicate  Derive  Join
Fill Null  Trim  Clean  Pivot  Unpivot  Transpose  Merge  Union  Script

11Write the transformation that does not exist

Objective

Find the way out when none of the seventeen operations fits.

Actions

  1. Drop a Script node onto the job canvas.
  2. Read the two execution modes.
  3. Use Preview to run it against real rows before saving.
Transformations → Script

Every declarative tool eventually meets a rule its authors did not foresee. Hydra answers with Script: you declare which columns come in and which go out, then write Python between the two. The contract stays readable in the YAML even though the logic is code.

Execution modeWhat your code receivesWhen to choose it
VectorizedEach input column as a pandas SeriesThe default. Operates on whole columns, so it stays fast on large volumes.
Row-by-rowEach input column as a single valueEasier to reason about, much slower. Keep it for genuinely per-row logic.
What you should see

np, pd, math and re are available, and Ctrl Space completes as you type. Preview runs the code on real rows so you find out here, not during a run.

The Script transformation dialog, offering a Vectorized or Row-by-row execution mode, a Python code editor with Preview and Expand buttons, and a note listing the available libraries.
A custom transformation, declared like any other node — and testable before it is saved.

12Give the canvas room

Objective

Recover screen space once you know where everything is.

Actions

  1. Select the < control at the top right of the palette.
  2. Work on the canvas with the palette collapsed.
  3. Select it again to bring the panel back.
NODE PALETTE → ‹

Collapsed, the palette keeps its icons: you can still drag a node onto the canvas without giving up the width. The vertical toolbar beside it stays put in both states — pan, select, undo, redo, minimap and terminal.

What you should see

The panel narrows to a strip of icons. Nothing is lost, and the canvas gains a third of the window.

The palette beside the vertical canvas toolbar, highlighted in red, holding pan, select, undo, redo, minimap and terminal controls.
The toolbar highlighted here belongs to the canvas, not to the palette — it stays whatever the panel does.
Tips and traps

The palette follows the scene. If a node is missing, you are probably in the other tab.

A dimmed entry is not a bug. It names what your host cannot run, and tells you which platform it needs.

Only Python runs everywhere, inline or from a .py file. Prefer it when a workflow must travel.

Remember that Set Param creates and Assign Param updates. The wrong one fails on purpose.

A Web API node reads only. To send data out over HTTP, use the Webhook action.

Reach for Script last. Seventeen named operations are read by everyone; your Python is read by you.

Counts beside Jobs grow with the project. Every job you write becomes a reusable node.

The Webhook action calls out; the Webhook trigger listens in. Same word, opposite direction.

A container in a job tidies the canvas. Only in a workflow does it write a policy.

3. Checklist

VerificationExpected result
Workflow paletteActions, Control Flow, Containers, Jobs
Job paletteSources, Transformations, Destinations, Containers
Shared categoryContainers, and it alone
Platform badgesOne shell active, the other dimmed
ConnectorsSeven in, six out — Web API reads only
TransformationsEighteen, the last one being Script
Jobs categoryLists the jobs of this project
CollapseIcons remain, and stay draggable

4. Troubleshooting

SymptomCauseFix
No Sources in the paletteYou are in the workflow sceneSwitch to Jobs configuration
No Actions in the paletteYou are inside a jobActions orchestrate; switch to Workflow configuration
Bash or PowerShell cannot be draggedYour host does not provide that shellUse the other one, or Python for portability
Jobs (0) in the paletteThe project has no job yetCreate one from the tab bar in Jobs configuration
No HTTP destinationWeb API is a source onlySend data out with the Webhook action
Set Param fails at runtimeThe parameter already existsUse Assign Param to change an existing value

5. Next lesson

You now know the whole vocabulary and where each word lives. The next lesson puts it to use: dragging a source onto the canvas and pointing it at a real file.

Close by

0 / 0 on this page