Hydra DSL

What is the Hydra DSL

The language in which a Hydra job is written. Four files declare where the data comes from, what happens to it, and where it goes. The engine reads them; nothing else is needed.

What it is used for

• Move data between files, databases and HTTP APIs.
• Clean, reshape and aggregate tables along the way.
• Order several jobs by dependency, on a schedule or on a webhook.

Notable features

• 18 operations, one YAML key each. No expression language to learn first.
• Nine connector types, same block shape — changing the source touches four lines.
• Secrets by reference. ${SECRET:NAME}, never a literal value.
• Explicit dependencies between jobs. depends_on is always a list.
• Validation before execution — hdrctl validate reads the manifest without touching data.
• Extendable through plugins: auth, cache, retry, pagination, circuit breaker.

Where to go from here

Follow the tutorial

One project, built lesson by lesson. Every block is copied from a job that runs.

• Lesson 1 — Your first job
• Lesson 2 — Transformationssoon
• Lesson 3 — Connectorssoon
• Lesson 4 — Joinssoon
• Lesson 5 — Aggregatessoon
• Lesson 6 — Workflowssoon
• Lesson 7 — Parameters and secretssoon

Start with a job

One source, a few operations, one destination. That is the whole unit.

• version
• sources.yaml
• transformations.yaml
• destinations.yaml
• pipeline.yaml
• workflow.yaml
• from and to

Transform the data

The 18 operations, grouped by what they change.

• steps
• Columns
• Rows
• Compute
• Aggregate
• Combine
• Reshape

Connect to real data

The shape of a source and of a destination block.

• type
• connection
• extract
• schema
• type
• connection
• load

Order several jobs

Dependencies, triggers, failures.

• name and version
• trigger
• steps
• retry

The fine print

Expressions, and what is substituted before parsing.

• filter.expr
• calculate.expr
• step.when
• script.code
• ${ENV} and ${SECRET}
• {{ param }} and {{ env }}

Beyond the language

Installing, connecting, running and watching belong to the platform.

• Guide — install, connect, run
• Playground — a full job on real data
• Build — the DSL written for you as you choose
• MigrateVS Code extension — coming from Airflow, dbt, Dagster, Prefect, Databricks or Airbyte

31 of 31 pages are published. The rest are marked soon rather than hidden.