Delivering records to a Web API

Context — a partner accepts completed orders through an API

The task is currently manual and its assumptions are not recorded. Hydra turns it into files that can be reviewed and rerun.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you load records into web_api with explicit, testable rerun semantics?

• Keep credentials outside YAML
• Choose append, replace or upsert deliberately
• Validate before touching the target
• Verify the target after a second run

The solution in one line

A Hydra web_api destination wired to a deterministic three-row fixture.

Steps

1. prepare a deterministic input

1. Start from three known records.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

id,name,amount
1,Amina,120.5
2,Lucas,83.0
3,Sofia,49.9

2. declare the input

1. Read the fixture as CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  input:
    type: csv
    connection:
      base_path: "data"
    extract:
      table: input.csv

3. declare the web_api destination

1. Describe the target and write mode.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  target:
    type: web_api
    connection:
      base_url: "https://api.partner.example"
      timeout_read: 30
    load:
      table: /v1/completed-orders
      mode: append
      batch_size: 100

4. wire the load

1. Connect input to target.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
pipeline:
  from: input
  to: target

5. validate the manifest

1. Catch shape and identifier mistakes.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test web_api-destination

ok  sources.yaml      — csv
ok  destinations.yaml — web_api
ok  pipeline.yaml     — input → target

✅ All tests pass — ready to execute.

6. run and inspect the target

1. Execute once, then verify rows and mode.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run web_api-destination

✅ Pipeline completed successfully
Rows read   : 3
Rows written: 3

POST requests : 3
Accepted rows : 243
Rejected rows : 0

Expected result

• Three rows reach the target.
• The chosen mode defines the second run.
• Credentials remain outside the manifest.

Reading the results

Rows written is the connector acceptance count. Confirm the physical target as well: files, tables, collections and APIs can impose constraints beyond the DSL.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

The Web API destination is beta: confirm that its writer plugin is installed before relying on this load.

Did we answer the question?

Before and after

• Before — a partner accepts completed orders through an API requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops