Hydra ETL
Build your first job
Write manifests faster

VS Code extension

Context-aware completion, live validation and ready-to-fill skeletons for the five Hydra manifests. The extension runs nothing: it speeds up authoring and flags structural mistakes. Cross-file consistency stays the job of hdrctl validate.

What you get

Three mechanisms

COMPLETION

Keys that match the context

The offered keys depend on where the cursor sits. A MySQL source offers host and port; a web_api source offers base_url and pagination. Enumerated values come with their explanation.

VALIDATION

Mistakes shown as you type

Unknown key, wrong type, value outside the allowed set, missing required field, unsatisfied conditional rule. Errors are underlined in the editor and listed in the Problems panel.

SNIPPETS

Skeletons instead of blank files

32 prefixes, all starting with h. Type hsrc-pg and press Tab to get a complete PostgreSQL source, then Tab through the fields. Fixed-value fields open a dropdown.

Before you start

Visual Studio Code 1.75 or laterCheck under Help then About
Python 3.9 or laterUsed to build the package, not to run the extension
The Hydra repositoryThe extension lives in VS Code Extension/

Pick a manifest

What the editor knows about each file

jobs/orders/sources.yaml
sources:
  src_orders:
    type: mysql
    connection:
      host: ${ENV:DB_HOST}
      port: 3306
      database: ${ENV:DB_NAME}
    extract:
      table: orders
      batch_size: 10000   # 10 to 100000

Type first. Once type is set, the keys offered under connection change with it — host and port for a database, base_url and pagination for an API.

Install

Three steps, about two minutes

01

Install the YAML engine

30 SEC

The Red Hat YAML extension provides the language server that reads the Hydra schemas. VS Code installs it on its own, since it is declared as a dependency, but you can add it first.

$ code --install-extension redhat.vscode-yaml
What you should see

The extension appears under Ctrl+Shift+X, or a message saying it is already installed.

02

Build the package

10 SEC

A .vsix file is a ZIP archive. The build script assembles it with the Python standard library, so neither Node, npm nor vsce is needed.

# from the VS Code Extension directory
$ python build_vsix.py
What you should see

The list of packaged files, then the path to install.

Package built: hydra-etl-0.1.2.vsix  (42.0 KB, 12 entries)
  extension/schemas/sources.schema.json
  extension/snippets/hydra.json
  ...
The extension directory in Windows Explorer, listing hydra-etl-0.1.2.vsix next to the schemas, snippets and examples folders.
The file appears in the extension directory. Take the highest version — earlier builds are kept but only the latest matters.
03

Install the package

30 SEC

Install it from VS Code, never from the file explorer. Two routes lead to the same result — pick either.

A From the command line

$ code --install-extension hydra-etl-0.1.2.vsix
  1. Run it from the extension directory.
  2. If code is not recognised, run Shell Command: Install 'code' command in PATH from the command palette first.

B From the Extensions panel

  1. Open Ctrl Shift X.
  2. Click the ... menu at the top of the panel.
  3. Choose Install from VSIX… and pick the file.
The Extensions panel overflow menu open in VS Code, with the Install from VSIX entry at the bottom of the list.
Install from VSIX… sits at the bottom of the overflow menu.

Whichever route you took, reload with Ctrl Shift P, then Developer: Reload Window.

Trap

Do not double-click the .vsix in the file explorer: Windows hands it to the Visual Studio installer, which refuses it. Do not copy the folder into the extensions directory either — since VS Code 1.74 a directory missing from the registry is ignored at startup, silently.

What you should see

A notification confirming the installation, and the extension in the installed list.

04

Confirm it is active

15 SEC
  1. Open Ctrl Shift X and search for Hydra.
  2. Check the name, the version and the publisher on the card.
What you should see

HYDRA ETL, version 0.1.2, published by Bechir Bejaoui. The full identifier is Bechir Bejaoui.hydra-etl — it contains a space, so quote it when passing it to a shell.

The HYDRA ETL extension card in the VS Code Extensions panel, showing version 0.1.2 and the publisher name, with the extension README open beside it.
The card carries the red hydra mark. The pane on the right is this README, rendered by VS Code.

Test it

Five checks, about five minutes

Open the examples/ folder of the extension in VS Code. It ships correct manifests and a set of deliberately faulty ones, each error documented in a comment.

T1

Errors are caught

1 MIN
  1. Open examples/invalid_examples/destinations.yaml.
  2. Open the Problems panel with Ctrl Shift M.
  3. Repeat with workflow.yaml and transformations.yaml.
What you should see

Sixteen entries in total, each pointing at the right line. The offending keys are underlined in the editor, and the file tab carries the error count.

VS Code showing destinations.yaml with underlined keys, and the Problems panel below listing four schema errors with their line and column.
Each entry names the schema that raised it and gives a line and column. Missing property "key" is the conditional rule on upsert firing.
Right-click menu on a Problems panel entry, offering Fix and Explain.
Right-clicking an entry offers Fix and Explain — these come from VS Code, not from the extension.
T2

Correct files stay silent

30 SEC
  1. Open the four files under examples/valid_job/.
  2. Open examples/valid_workflow/workflow.yaml.
  3. Look at the Problems panel again.
What you should see

No entry at all. A warning here means the schema is stricter than the engine — worth reporting.

T3

Completion follows the context

1 MIN

This is the check that matters: it proves the conditional schemas are wired correctly.

  1. In examples/valid_job/sources.yaml, replace type: mysql with type: web_api.
  2. Delete the contents of the connection: block, keeping the key itself.
  3. Put the cursor on an indented empty line below it and press Ctrl Space.
What you should see

Typing web on the type line already offers web_api. Once it is set, the keys under connection switch from host, port, database, user and password to base_url, auth and pagination.

The sources.yaml editor with type set to web, and a completion popup offering web_api.
Values are completed too, not just keys.
The same file after switching to web_api, with the connection block now holding url, auth, type and key.
The connection block now carries API keys instead of database ones.
T4

Snippets expand

1 MIN
  1. Create an empty file named sources.yaml.
  2. Type sources:
  3. On a new line, press Esc, then type hsrc-pg and press Tab.
  4. Press Tab again to move between fields.
Tip

The Esc matters. On an empty line the schema opens its own key suggestions, and that popup captures the Tab before the snippet does. Dismissing it first lets the prefix expand. Typing h then Ctrl Space lists all 32 prefixes.

What you should see

A complete PostgreSQL source, cursor on the source name. On the table or query field, a dropdown opens.

A sources.yaml file with the prefix hsr typed, and the suggestion list showing the Hydra snippets, hsrc-pg highlighted as the PostgreSQL source.
Every prefix starts with h, and each carries the manifest it belongs to.
T5

Schemas match the engine

1 MIN

The same checks, run without opening the editor. Useful in continuous integration.

$ pip install jsonschema pyyaml
$ python examples/check_schemas.py
What you should see

Every schema valid, correct manifests clean, faulty ones caught.

=== 1. Schema validity ===
[  OK  ] destinations.schema.json
...
PASSED — every check succeeded.

Expected errors

What the faulty examples advertise

FileErrorsOne of them
destinations.yaml4Missing property "key" — upsert needs one
transformations.yaml7"integer" is not one of int, float, str, bool, date, datetime
workflow.yaml5"cron" is a required property

Troubleshooting

The extension is not listed after installing

Copying the folder into the extensions directory no longer works. Since VS Code 1.74, extensions.json acts as the registry, and a directory missing from it is ignored at startup without any message.

Install through the .vsix instead, then remove the stray copy.

The VSIX installer refuses the file

The window titled VSIX Installer belongs to Visual Studio, not to VS Code. Windows opens it on double-click.

Close it and install from inside VS Code, through Install from VSIX or the code command.

No completion and no errors at all

The YAML language server has not taken over. Check the language mode in the status bar reads YAML, then reload the window once more — the server sometimes needs a second cycle to pick up schemas from a freshly installed extension.

Suggestions disappear while filling in a snippet

The extension sets editor.suggest.snippetsPreventQuickSuggestions to false by default. A workspace or user setting can override it.

A connector type is underlined but the job runs

Connectors supplied by a plugin are not part of the offered list. They are accepted without error, so this should not happen — report the case with the manifest.

Known limitations

Version 0.1.0

Next

Read the grammarEvery manifest block the schemas are built from, field by field. Use the CLIValidate, test and run what you just wrote in the editor. Build your first jobCSV in, two operations, CSV out — the complete guided example.
0 / 0 on this page