Hydra ETL changelog

Start a project when a file arrives, a table changes, a webhook is called or another project ends — and let one project run another — without an external scheduler or message broker.

pip install -U hydra-etl

New features

Project triggers — triggers.yaml

• A triggers.yaml file at the project root, seven types:

  Type	Starts the project when…	Parameters received
  cron	the schedule matches (cron, IANA timezone)	—
  interval	every every	—
  file	a file matching path (glob, **) is added or modified	trigger.file, trigger.files
  marker	a marker file such as inbox/READY appears (removed unless consume: false)	—
  source_change	a job source changes (check: count | max | checksum)	trigger.value, trigger.previous
  webhook	POST …/triggers/{name}/webhook with a valid token	none (nothing from the request)
  after_project	another project ends (on: success | failure | any)	trigger.project, trigger.status

• Common keys: env, workflows (default: the thread), params (no secret), overlap (skip by default), timeout, retry: {max, delay}, enabled, description. A trigger always runs the project in its own process, under .hydra/run.lock.

• hdrctl watch [PROJECT]... [--env] runs the triggers of one or more projects. It prints each trigger and its next run, then every start and end.

• hdrctl serve runs the same engine for every project of the Studio list.

• hdrctl trigger list | validate | fire <name> | history | token <name> [--rotate]. validate checks workflows, called projects, sources, webhook secrets and refuses after_project loops. Every run is recorded in .hydra/triggers/history.jsonl.

• Guarantees:

  • a trigger still running is not started again (the missed run is recorded as skipped); runs missed while the engine was stopped are not replayed;
  • triggers.yaml is reloaded without restart; an invalid file keeps the previous triggers running;
  • a file is taken only once unchanged for debounce (no half-copied file); a file arriving during a run starts the next run;
  • a source_change measure runs apart from the engine (a slow database does not hold up other triggers); on PostgreSQL, MySQL/MariaDB and SQL Server count and max are a single SELECT COUNT(*) / SELECT MAX(col);
  • webhook tokens are ${SECRET:…} references, compared in constant time (401 / 404 / 503); keep hdrctl serve on 127.0.0.1 unless an HTTPS reverse proxy stands in front;
  • first Ctrl+C on hdrctl watch waits for running projects; a second stops at once.

• Cron expressions are parsed by Hydra itself (no extra package); tzdata is installed on Windows.

• Studio — Project settings › Triggers: triggers with event, environment, next run, running state and last run; a form per type; enable/disable, delete, Run now; create or rotate a webhook token (shown once, with PowerShell and curl calls); run history. An invalid change is not written; the previous file is backed up to ~/.hydra/_backups.

• API: GET|PUT /api/projects/{id}/triggers, …/triggers/history, POST …/triggers/{name}/fire, …/{name}/webhook, …/{name}/token.

Sub-projects — type: project step

• A workflow of a "master" project runs another project: its thread.yaml, or only the workflows listed. Keys: project (relative path, or ref: a Studio project), env (inherit by default), params, variables, workflows, outputs, isolation (inprocess or subprocess), timeout, overlap (wait, skip, fail).
• Nothing is passed implicitly except the environment; the called project keeps its own secrets, and ${SECRET:…} is refused in params and variables.
• Checked before anything runs (hdrctl run, --dry-run, workflow validate, Studio Check): project not found or ambiguous, cycles (a → b → a), more than 5 levels. workflow validate prints the tree of called projects.
• Results readable by the next steps: {{ project:STEP.status }}, .rows, .duration, .param.NAME.
• A called project runs once at a time (.hydra/run.lock, also between processes; a stale lock is taken over).
• hdrctl run <project> -w <workflow> (repeatable): run only these workflows.
• Studio — Project node: a Projects section in the palette; a cycle is refused with its path; one window sets path, environment, workflows, parameters, outputs, process, timeout and overlap.

Tools

• VS Code extension 0.1.8 (type: project) and 0.1.9 (triggers.yaml: the seven types, token as a secret reference only, 8 snippets).
• MCP: hydra_project_calls and hydra_project_triggers (read-only).
• API: POST /api/projects/{id}/project-step.

Patches and fixes

• Workflow actions lost the values of environments/<env>.yaml when a required parameter was given at run time (-P, or params of a project step): {{ param:X }} fell back to the defaults while the jobs of the same run used the environment.
• hdrctl workflow run refuses project steps that form a cycle, call a missing project or nest more than 5 levels, before anything runs.
• An optional {{ env:NAME }} or a parameter without value inside a text becomes an empty string instead of None.
• Parameter errors are in English: missing required parameter, unresolved parameter, missing environment variable.

Documentation

• White paper (FR/EN): 1.4 Sub-projects and 1.5 Project triggers. README and 03-workflows.md updated.

Full changelog: v0.18.0...v0.19.0

One project, several environments: global variables shared by the machine, parameter values and secrets per environment, managed from the CLI or Studio.

pip install -U hydra-etl

New features

• Global environment variables, shared by every project of the machine: ~/.hydra/env (all environments) and ~/.hydra/env.<env> (one environment), selected by --env / HYDRA_ENV; location HYDRA_HOME. Order, weakest first: env < env.<env> < project .env < job .env < --env-file. When a run switches environment (Studio, API), the variables of the previous one are removed. Workflow actions see these variables and the project .env from the first step.
• Secrets per environment: .hydra/secrets.<env> (plain or SOPS-encrypted) wins over .hydra/secrets with --env <env>; sops.file accepts {env}. The .gitignore written by hdrctl init ignores .hydra/secrets.* but keeps secrets.yaml and encrypted files.
• --env-file FILE on hdrctl run and hdrctl workflow run: extra variables for one run, above every other layer.
• hdrctl workflow run accepts --env and -P/--param.
• New commands:
  • hdrctl env list|show|create|rename|delete|set|unset — environments and global variables; rename and delete also move parameter values and secrets, after a backup to ~/.hydra/_backups/;
  • hdrctl param list|set|unset — parameters, default value and value per environment;
  • hdrctl secret list|set|delete — the value is typed hidden or read from stdin, never passed as an argument; list shows names only.
• Studio:
  • active environment selector on the project page and in the editor, used by every run and by the Expression Assistant;
  • Global settings › Environments: create, rename, delete environments and edit their global variables;
  • Project settings: tabs Execution order, Parameters (one column per environment) and Secrets (names only).
• parameters.yaml accepts the short form name: value (it stopped the run with an internal error).
• VS Code extension 0.1.7: schemas and snippets for parameters.yaml, environments/<env>.yaml and .hydra/secrets.yaml.
• MCP: hydra_project_settings (read-only; never a secret or variable value); hydra_run_workflow accepts env.
• API: /api/global-env; /api/secrets accepts env; inline job and action runs accept env.

Patches and fixes

• hdrctl run jobs/<job> reads the .env at the project root, not only the one of the job's parent folder.
• bash, powershell and python actions without working_dir run in the project root; a relative working_dir is resolved from there. Scheduled and webhook runs started by hdrctl serve used to run in the server's folder, so relative paths failed.
• Studio: a project whose .hydra/project.json had no id got the id ""; the id is now added to the file.
• Studio: the schedule workflows of a project opened in Studio are planned at once, not at the next start of hdrctl serve.
• Studio: runs with no environment selected sent env: "default"; "default" now means no environment.

Full changelog: v0.17.0...v0.18.0

New features

Secrets

• Secrets are masked everywhere Hydra shows or keeps text: logs (every logger, step logs, API), error messages, tracebacks, the output of bash, powershell, python and ssh actions, webhook and email log lines, the rollback report and the runs stored by the API. URL-encoded and JSON-escaped forms are masked too. Values sent to connectors, scripts and webhooks are unchanged.
• Visibility, as in GitLab: plain (debugging, with a warning), masked (default, [MASKED]) or hidden (lines containing a secret are dropped and counted). Set it in .hydra/secrets.yaml, with HYDRA_SECRETS_VISIBILITY or --secrets-visibility. Values shorter than 4 characters are not masked; a warning names them.
• .hydra/secrets.yaml: project secrets settings, with no secret inside, safe to commit (visibility, provider). Unknown keys and providers are reported, never ignored.
• SOPS provider (provider: sops; age, PGP, cloud KMS): secrets decrypted in memory on first use; nested keys read as ${SECRET:db.password}. A SOPS-encrypted .hydra/secrets or --secrets-file is detected and decrypted as well.
• Provider interface (SecretProvider, register_provider) for external vaults.

Transformations

• join with how: cross (cartesian product), guarded by max_rows (default 10,000,000 rows produced): the run stops before the batch that would exceed it.
• text_to_rows: one row per value of a text column ("a;b;c" → 3 rows). Options sep, regex, trim, keep_empty, index_column. No input row is lost.
• window: window functions that add columns and keep every row in order — rank (row_number, rank, dense_rank), offset (lag, lead), running (cumsum, cumcount, cummin, cummax) and whole-partition (sum, mean, min, max, count, first, last). Runs in DuckDB with the large-volume engine, same results as pandas.

Studio and tools

• Expression Assistant: lists the project's parameters (with their value in the active environment), environment variables and, on reference fields, secrets. Double-click or drag an item; Insert writes it into the field. Opens from fx on expressions and from a new {x} button on fields such as paths, host, SQL query, URL or timeouts. Numeric fields accept a reference.
• Text to Rows and Window nodes; the Join node offers cross with a Max rows field.
• DSL specification: 20 operations. MCP: hydra_list_operations lists them, hydra_explain_error explains the new errors, hydra_check_job warns when a request needs window, text_to_rows or join.
• VS Code extension 0.1.6: schema and snippets for the additions.

Patches and fixes

• Studio: the Script node showed a generic icon in the palette and its dialog.
• Studio: the MySQL and PostgreSQL nodes show the official logos; the SQL Server node shows a database icon.

Full changelog: v0.16.1...v0.17.0

New features

• None.

Patches and fixes

• MongoDB source: schema.drift_policy now applies. The schema block never reached the connector, and documents were normalised before drift detection, so fail / warn / adapt / ignore had no effect. Now:

  • fail raises SchemaDriftError naming the field and the document;
  • warn logs each new field once, plus a summary;
  • adapt extends the schema;
  • ignore stays silent.

  schema.sample_size is honoured. Regression tests: tests/test_mongodb_drift.py.

Full changelog: v0.16.0...v0.16.1

New features

• engine: auto | pandas | duckdb in transformations.yaml (at the root or under transformations:) chooses the engine of a job's global operations; HYDRA_ENGINE still takes precedence. Studio: Engine selector on a job's canvas, kept on round-trips with the YAML files. JSON schemas (Hydra DSL, VS Code extension 0.1.5) updated.
• Right and outer join run in DuckDB in the large-volume engine, with the same rows, columns, order and types as pandas; only the columns the following aggregation needs are copied. 20 million rows, right join then aggregation (2 CPUs): 71.8 s / 3.4 GB → 27.7 s / 2.6 GB.

Patches and fixes

• sort is stable: rows with equal sort keys keep their input order (it already was for several columns, not always for one).

Full changelog: v0.15.0...v0.16.0

New features

• Large-volume engine (optional: pip install "hydra-etl[duckdb]", which now also installs pyarrow). Above HYDRA_LARGE_VOLUME_ROWS estimated rows (2,000,000 by default), a job with a global operation reads its CSV source with pyarrow and runs aggregate and sort in a temporary on-disk DuckDB database. pandas keeps every other step and stays the default below the threshold.
• HYDRA_ENGINE=auto | pandas | duckdb forces the choice.
• No silent fallback: a global step DuckDB cannot translate goes back to pandas with a log message; the fast reader hands over to the Python reader with a warning.
• Measured on 20 million rows (2 CPUs): 112.8 s / 5.4 GB → 34.3 s / 1.4 GB, identical results.
• --profile shows the engine chosen and why.

Patches and fixes

• DuckDB sums are byte-identical to pandas (compensated, ordered summation), so switching engine never changes a result.

Full changelog: v0.14.0...v0.15.0

New features

• Fallback after a failure (fallback.on_failure: true): a workflow that fails is replaced by its first backup, only if it left nothing published — no load published and no outside action done (except actions marked undo: none), or everything undone by rollback: true. If the backup fails too, the next one in use runs. Otherwise the project stops and says why: ⛔ load_crm: fallback load_crm_backup not started — step 'load1' had already published its data. The replaced run is reported as recovered (CLI ↯, Studio badge replaced by …); the project succeeds if the backup does.
• timeout per workflow in thread.yaml (30s, 5m, 1h): beyond it the workflow is stopped at the next step or batch (an all-or-nothing load is cancelled; an action already started finishes) and counts as failed, which can trigger the fallback. Shown by --dry-run, in Studio, in the VS Code schema and in the DSL reference.
• hdrctl run --dry-run and Studio's Check warn when a workflow with on_failure publishes in several steps without rollback: true.
• Studio — wait field (delay_before) on each workflow of the execution order; the Preview shows the wait above the block, the run page a waited … badge. The run API returns waited on each step.
• Studio — Settings has two tabs: Global settings (editor preferences) and Project settings (execution order, fallbacks, loops, written to workflows/thread.yaml).
• Studio — the project page lists the workflows with Project settings and Run project (the project is checked first; errors link to Project settings).

Patches and fixes

• Studio: the run page of a workflow kept by its condition, then replaced after a failure, is shown correctly.
• Studio: a workflow opened from the Preview opens on its Workflow configuration tab and returns to Project settings.
• Studio asks before leaving the execution order with unsaved changes.
• CLI: durations of whole minutes or hours are shown as 5m, 1h.

Full changelog: v0.13.0...v0.14.0

A failed or risky run no longer leaves half-written data behind: each job loads all or nothing, a workflow can undo what it did, and a backup workflow can take over before the start.

New features

Declarative fallback (thread.yaml)

• A workflow can declare backup workflow(s): use (tried in order), when (condition checked just before the start) and on_probe_error (keep by default, or replace).
• when uses a closed language analysed by Hydra, never Python eval: source_available, destination_available, file_exists, file_size, file_age, url_ok, secret_defined, db.load, previous.status, previous.rows, and/or/not, comparisons, durations (24h) and sizes (5MB).
• At run time, quick probes with a time limit (HYDRA_PROBE_TIMEOUT, 15 s) read no data. True → the first backup runs in place (↪ Fallback: a → a_backup — <reason>).
• hdrctl run <project> --use-fallback <workflow> forces a fallback to rehearse a recovery.
• hdrctl workflow add <name> --fallback-for <workflow>; workflow list shows backups; workflow remove cleans the references.
• --dry-run and Studio's Check check backups, their jobs and the sources/destinations named in when, and warn when a backup writes elsewhere.
• Studio: fallback editor in Define (condition checked as you type, helper to insert functions and project resources), backups drawn as dashed blocks in Preview, the decision and its reason on the run page.

All-or-nothing load per job

• A job writes its destination through one load session (begin → stage → publish | abort), common to every connector. Nothing is visible before the end of the job; a failure or Ctrl+C leaves the destination as it was.
• On by default for replace (PostgreSQL / SQL Server: TRUNCATE in one transaction; MySQL / MariaDB: DELETE in one transaction, InnoDB; CSV / Parquet: temporary file then swap) and for CSV / Parquet append.
• load.atomic: true | false in destinations.yaml; true on a connector that cannot honour it is an error before anything is read.
• Two jobs of the same run writing the same destination take turns.

Workflow rollback

• rollback: true in workflow.yaml: if the workflow fails, the steps already done are undone, last first. Files restored, CSV append truncated back, replaced tables restored, rows added by append/upsert removed and changed rows restored (PostgreSQL, MySQL / MariaDB, SQL Server).
• Actions declare undo: (an action, or none), required for webhook, email, bash, powershell, ssh, python.
• A destination changed by someone else since the run is not undone: the rest is kept in .hydra/undo/<workflow>.json and the workflow refuses to run until hdrctl workflow undo <workflow> finishes it.
• Studio: Rollback on/off in the toolbar, Undo section on actions, undone steps on the run page.

Also

• delay_before on a workflow: wait before starting it (staggered starts in a parallel stage).
• API: GET /api/projects/{id}/conditions, POST /api/projects/{id}/conditions/check; project run steps carry a fallback object.
• VS Code schema and DSL reference updated.

Patches and fixes

• Studio: a job whose operations were written as top-level steps: in transformations.yaml (the form of hdrctl init and the CLI) lost them when saved. Both forms are now read.
• A job keeps one connection / one open file for all its batches; Parquet no longer copies the whole file for every batch.
• Without all-or-nothing, a failed job logs that the destination keeps the batches written before the failure.

Full changelog: v0.12.3...v0.13.0

New features

• Studio and the job files stay in sync, both ways. Studio keeps a job and its jobs/<name>/*.yaml files in step, and the latest change wins, made in Studio or outside (editor, hdrctl, git).
  • The sync button (Hydra DSL panel, or Sync with disk in the toolbar) compares canvas and files: only the disk changed → the canvas is rebuilt (node positions kept); only Studio changed → the files are written; both changed → the most recent change wins.
  • Automatic sync of the open job every few seconds (Settings, default 5 s, 0 = off), and when a job is opened.
  • Formatting and comments are not changes: Studio compares the content.
  • API: GET /api/jobs/files also returns hashes, hash and mtime.
• The Hydra DSL panel buttons show three states: ready, busy (spinning) and unavailable.
• /api/health returns studio_build; a tab running an older Studio shows New Studio version — reload.

Patches and fixes

• Studio no longer loses what the canvas does not edit. A job rebuilt from its YAML and saved again reset batch_size to 1000, dropped fields such as delimiter or encoding, and renamed source/destination keys, which changed pipeline.from/to.
• hdrctl validate printed "0 error(s) detected" when the only errors were in pipeline.yaml, in strict checks or a missing file. The total now counts every error shown.
• pipeline.yaml is checked the way the engine reads it: old nested format, missing or empty from/to, unknown keys. Same check in hdrctl run <job> --dry-run.
• hdrctl run <project> (and --dry-run, Studio's Check and Run project) checks pipeline.yaml of every job used by an active workflow before starting: a broken pipeline.from no longer passes as valid.
• hdrctl serve sends index.html with Cache-Control: no-cache, so an update is not hidden by the browser cache.
• Connector messages (CSV, JSON, Parquet, PostgreSQL, MySQL/MariaDB, SQL Server) and Studio templates are in English.

Full changelog: v0.12.2...v0.12.3

New features

• Studio: the workflow cards of a project page can be rearranged by drag and drop. Grab a card by its title and drop it on another card (dashed outline) to swap them. Display only: the order is kept in .hydra/card-order.json and does not change thread.yaml or hdrctl run. API: PUT /api/projects/{id}/card-order.

Patches and fixes

• None.

Full changelog: v0.12.1...v0.12.2

New features

• Project secrets in .hydra/secrets: one name=value per line, read by ${SECRET:name} in jobs and in workflow action parameters. An environment variable wins over the file (${SECRET:db.password} first reads DB_PASSWORD). Keep the file elsewhere with HYDRA_SECRETS_FILE or --secrets-file on hdrctl run and hdrctl workflow run. Parse errors give the line, never the value.
• hdrctl init (and a project created in Studio) writes a commented secrets template and a .gitignore that excludes .hydra/secrets, .env and .env.*.
• Studio — Secrets tab in the workflow editor: names only (values never leave the server), add, replace, delete, copy the ${SECRET:name} reference, and see when an environment variable overrides a secret. A plain-text password field offers Move to Secrets. API: GET|PUT /api/secrets, DELETE /api/secrets/{name}.
• Studio — Execution order panel on the project page, editing workflows/thread.yaml: stages, sequential/parallel, delay_after, active or skipped workflows, order. Check runs the same checks as hdrctl run --dry-run; Run project runs every active workflow in that order, followed live. API: GET|PUT /api/projects/{id}/thread, POST /api/projects/{id}/check.
• Loops in thread.yaml: loops: [{from, to, times, delay}] replays the part of the thread from to to from, times passes in total, with an optional pause. hdrctl run shows (pass i/N); --dry-run lists the loops; the VS Code schema validates them.
• Studio — the Execution order panel has two tabs: Define (stages and loops, checked as you type) and Preview (the thread drawn left to right, parallel workflows stacked, delays as diamonds, loops as dashed return arrows ×N).
• The demo project has the same layout as hdrctl init, and hdrctl run . runs it whole.

Patches and fixes

• Studio no longer overwrites changes made outside it. A job attached with hdrctl job attach, or a file edited by hand, was not shown and was erased by the next save. Studio now records a fingerprint of every file it writes: a workflow whose files changed is rebuilt from disk with a notice; a save over a file changed meanwhile is refused (HTTP 409, nothing written), auto-save pauses, and Studio offers Reload from disk or Overwrite with Studio's version.
• init no longer creates an empty .hydra/workflows/.
• Studio's example paths for CSV, JSON and Parquet destinations point to the project's outputs/.

Full changelog: v0.12.0...v0.12.1

A project is now the unit you create and run: one command checks it whole, then runs its workflows in a declared order.

New features

• workflows/thread.yaml: the order in which the project's workflows run. Stages run one after the other; each stage is sequential or parallel, takes an optional delay_after (30s, 5m, 1h), and a workflow can be set active: false to skip it while debugging.
• hdrctl run <project> (or hdrctl run .) runs every workflow in the order of thread.yaml. Before starting anything it checks the thread, every active workflow and the job folders they use, and lists every problem found. A failed workflow stops the project (exit code 1); workflows not reached are reported as not run. Without thread.yaml, workflows/*.yaml run in alphabetical order. --dry-run prints the execution plan; --env, -P and the --on-success/--on-failure/--on-finish hooks apply to the whole project.
• hdrctl workflow add | remove | list: create a workflow and add it to the thread, delete it (jobs are kept, confirmation unless -y), list workflows in thread order with stage, mode, state and step count, plus workflows missing from the thread and entries without a file.
• hdrctl job add | attach | detach | list: create a job from the init templates and attach it to a workflow at a position (-p N), attach an existing job, detach it (the following steps inherit its dependencies, the folder is kept), list free jobs, the workflows that use each job and missing files. Every change is validated before the workflow file is written.
• hdrctl workflow run and workflow validate accept a workflow name as well as a file path, and --project.
• hdrctl init <path> creates the full project layout: .hydra/, Data/, environments/, jobs/job1/, outputs/ (was output/), parameters.yaml, README.md, scripts/, and workflows/ with workflow1.yaml and thread.yaml.
• Studio keeps thread.yaml in step when a workflow is created or deleted.
• API: POST /api/jobs/workflow/add, /workflow/remove, /job/add, /job/attach, /job/detach (Postman collection updated).

Patches and fixes

• A workflow stored in workflows/ did not see the project's parameters.yaml and environments/, so {{ param:X }} stayed unresolved. The runner now finds the project root (the folder holding .hydra/); a python action's file_path is also looked up from there.
• The output of a failing python, bash, powershell or ssh action is shown under the step (it was dropped). The error reads Python script exited with code 3: <last stderr line>.
• A failed workflow's log line is no longer printed twice on stderr.
• Workflow engine messages are in English, like the rest of the CLI.
• hdrctl list on a project folder lists its jobs (it answered "No jobs found"), and no longer prints "OK" twice.
• hdrctl --help gives each command a one-line description; help texts and hints no longer mention the old hydra command.
• Removed: hdrctl workflow init (use hdrctl workflow add), and the incomplete CLI translations that were never loaded. The Spanish translation is now complete.

Full changelog: v0.11.4...v0.12.0

New features

• hdrctl init now scaffolds a project, not a bare job folder: jobs/job1, workflows/workflow1.yaml, Data/, output/ and .hydra/project.json, which marks the folder as a Hydra project. The six job templates are unchanged (--template). Run a job with hdrctl run my_project/jobs/job1, or the workflow with hdrctl workflow run my_project/workflows/workflow1.yaml.
• A demo project ships inside the package (hydra_etl/samples/hydra-demo), with an endpoint that installs it: a fresh install has something to run without writing any YAML.
• The Studio file picker tells a Hydra project from an ordinary folder (is_project), so an existing project opens directly.

Patches and fixes

• mode: upsert now works on PostgreSQL. Every PostgreSQL upsert failed with KeyError: 0, even with a PRIMARY KEY on exactly the key columns. Two defects in the pre-flight constraint check:

  • array_agg over a sql_identifier column came back as raw text {col}, so columns were compared character by character;
  • the RealDictCursor row was unpacked as a tuple, which hid the real error.

  The aggregate is now cast to text and the row read by key. New unit tests run without a database (tests/test_postgresql_upsert_key_validation.py).

• "Driver missing" errors name the extra to install: pip install "hydra-etl[postgres]", pip install "hydra-etl[mysql]", and the same for MongoDB.

Full changelog: v0.11.3...v0.11.4

New features

• None.

Patches and fixes

• ${SECRET:NAME} now resolves. The CLI built its resolver with an empty secret store, so every ${SECRET:…} placeholder failed with Secret manquant, although the README and the DSL reference documented it. A secret is now looked up in the injected store first, then in the process environment: the name is upper-cased and dots and dashes become underscores, so ${SECRET:db.password} reads DB_PASSWORD. When a secret is missing, the error names the environment variable to define.

Full changelog: v0.11.2...v0.11.3

Hydra ETL 0.11.2 — validation no longer lets mistakes through

pip install -U hydra-etl

This release fixes a class of bugs that contradicted the core promise of Hydra ETL: validate before you run. Several mistakes in a workflow.yaml were silently ignored, and the run could report success for work that never happened.

Fixed

• Unknown actions are rejected. A typo such as action: powersheII used to make the step do nothing while the run reported success. hdrctl workflow validate now fails and suggests the closest action (did you mean 'powershell'?), and hdrctl workflow run refuses to start.
• Unknown keys are rejected at every level. depend_on: [fetch] (instead of depends_on) used to validate, and the step ran without its dependency. Unknown keys at the top level, under workflow, in steps, trigger and retry are now refused with a suggestion.
• Action parameters are checked. Unknown parameters (mesage: under log) and missing required ones (command for bash, url for webhook, script or file_path for python…) are reported by validate, not at 2 a.m.
• webhook sends headers. Studio offered the field but the runner ignored it, so an Authorization header was never sent. A text body is now sent as written.
• hdrctl workflow run output: no Python traceback on an invalid file, steps that never ran after a failure are listed as skipped, and validation errors are printed one per line.

Behaviour change

A workflow that relied on a misspelled or misplaced key being ignored now fails validation. The error names the key to fix or remove:

❌ Invalid workflow:
  - steps.1: Step 'b': unknown key 'depend_on' — did you mean 'depends_on'?

Full details: CHANGELOG.md

pip install hydra-etl==0.11.1

Added

• Hydra Studio offers SQL Server as a source and as a destination. The engine
  had the connector since 0.11.0, but the visual editor gave no way to pick it,
  so a SQL Server pipeline could not be built there at all.

  The form carries host, instance, port, database, schema, user and
  password, then table or query as a source, table and mode as a destination.

  The port is deliberately not defaulted to 1433 the way MySQL and PostgreSQL
  default to 3306 and 5432. An explicit port short-circuits the SQL Browser
  lookup, so writing one by default would make named instances unreachable from
  Studio — that is, SQL Express, whose instance is named by construction. The
  field label says so: leave it empty for a named instance.

  Proven end to end: a SQL Server source resolved through SQL Browser, no port
  given, loading into a CSV destination.

Full history: CHANGELOG.md

pip install hydra-etl==0.11.0

Added

• Microsoft SQL Server connector, aliases sqlserver and mssql. Extract with
  a table or a query, load in append, replace or upsert mode, with
  auto-creation of the target table. Install it with
  pip install "hydra-etl[mssql]".

  The driver is pymssql, which ships FreeTDS inside its wheels, rather than
  pyodbc, which needs the msodbcsql system driver installed separately.
  One command, nothing to deploy.

  Upsert runs UPDATE then INSERT ... WHERE NOT EXISTS inside a
  transaction. MERGE is deliberately not used: its concurrency problems are
  documented and many SQL Server teams ban it outright.

  Auto-created tables use NVARCHAR rather than VARCHAR and DATETIME2
  rather than DATETIME, so accented text and timestamps behave identically
  whatever the server's collation.

  A named instance is resolved by Hydra itself. pymssql embeds FreeTDS,
  which unlike the Microsoft drivers never queries the SQL Browser service, so
  MACHINE\\SQLEXPRESS would reach port 1433 and fail — the default setup of
  SQL Express, that is, of the audience this connector targets. Hydra sends
  the SSRP datagram itself over UDP 1434 and connects on the port it gets
  back. An explicit port short-circuits the lookup; a silent SQL Browser falls
  back to an error message that says which of the two to fix.

  Tested end to end against two live servers: SQL Server 2022 on Windows with
  a named instance on a static TCP port, and
  mcr.microsoft.com/mssql/server:2022-latest in Docker on Linux. Same major
  version on both, so the operating system was the only variable. 74 unit
  tests need no server; 11 end-to-end tests are skipped unless
  HYDRA_E2E_MSSQL is set.

Known limitation

• Azure AD authentication and Always Encrypted are out of reach with
  pymssql. They need pyodbc and the msodbcsql system driver, which will
  be added as an option the day a user asks for it.

Full history: CHANGELOG.md

pip install hydra-etl==0.10.3

Changed

• The MCP server now speaks English. The instructions of the server and the
  descriptions and annotation titles of its 17 tools were in French; they are
  what an AI client reads to decide which tool to call, and what MCP
  registries publish.
• serverInfo reported an empty version. It now carries the package version,
  and adds a title and the website URL when the installed MCP SDK accepts
  them. The server name is hydra-etl, matching the package.
• hydra-mcp --help is in English.

Full history: CHANGELOG.md

pip install hydra-etl==0.10.2

Fixed

• hdrctl validate --strict now recognises the steps of a transformations.yaml
  written as transformations: steps:; it previously reported "none" on the
  manifests shipped in examples/.
• hdrctl validate --strict checks the syntax of filter and calculate
  expressions with pandas' own parser, without evaluating them. An invalid
  expression now fails validation (exit code 1) instead of failing at run time.

Added

• hdrctl workflow run prints what a python, bash or powershell action
  wrote on stdout/stderr, under the corresponding step (capped at 20 lines).
  StepResult carries it in a new output field.

Full history: CHANGELOG.md

[0.10.1] — 2026-09-17

Fixed

hdrctl run no longer ends with "Critical error: TypeError" on Python 3.9 (click 8.1 passed None as verbosity when no -v flag was given).
The mcp extra now installs only on Python 3.10+, so pip install "hydra-etl[all]" works again on Python 3.9.

Changed

httpx and requests-mock are declared in the dev extra.
CI now runs on Linux, Windows and macOS, Python 3.9 to 3.13.