Hydra ETL
Build your first job
Guide›Sources›web_api
Workshop 7 · 28 minutes

Collecting every page of a REST API

A partner API returns one hundred orders at a time and protects the endpoint with a bearer token.

Context — a partner exposes paginated orders over HTTPS

A partner API returns one hundred orders at a time and protects the endpoint with a bearer token.

Where things stand

  • The current result is fragile. Its assumptions are split between tools, clicks and memory.
  • Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
  • Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you authenticate, paginate and turn API responses into Hydra batches?

  • Never write the token in YAML
  • Follow pagination until completion
  • Bound request and batch sizes
  • Extract only the records array

The solution in one line

A Hydra web_api source, a local inspection destination and one pipeline command.

HTTP response6 steps
GET /v1/orders?offset=0&limit=100

{
  "items": [{"id": "O-1", "amount": 72.5}],
  "next_offset": 100
}
What you do

Identify the records and the contract you need.

Step 1 · input understood

Steps

1. inspect the input

  1. Identify the records and the contract you need.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
GET /v1/orders?offset=0&limit=100

{
  "items": [{"id": "O-1", "amount": 72.5}],
  "next_offset": 100
}
Check — input shape identified

2. declare the web_api source

  1. Describe the connection and extraction.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
version: "1.0"
sources:
  partner_orders:
    type: web_api
    connection:
      base_url: "https://api.partner.example"
      timeout_read: 30
      auth:
        type: bearer
        token: ${ENV:PARTNER_API_TOKEN}
      pagination:
        strategy: offset
        page_size: 100
    extract:
      table: /v1/orders
      batch_size: 100
Check — partner_orders is the pipeline-facing identifier
Trap — Pagination belongs to the connector. A transform never needs to know which HTTP page produced a row.

3. declare a reviewable output

  1. Write the extracted rows to a local CSV.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace
Check — replace keeps the inspection file stable

4. wire source to destination

  1. Connect the two identifiers.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
version: "1.0"
pipeline:
  from: partner_orders
  to: result
Check — the job now has one input and one output

5. validate without reading data

  1. Check the four manifest files.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
$ hdrctl test web_api-source

ok  sources.yaml      — web_api
ok  destinations.yaml — csv, replace
ok  pipeline.yaml     — partner_orders → result

✅ All tests pass — ready to execute.
Check — manifest valid; source not consumed yet

6. run and read the counters

  1. Execute the extraction once.
  2. Compare the file with the explanation in the workbench.
  3. Record the shown check before moving to the next step.
$ hdrctl run web_api-source

✅ Pipeline completed successfully
Requests      : 4
Rows read     : 327
Rows written  : 327
Last page     : 27 records
Check — 327 orders · four requests

Expected result

  • 327 orders · four requests
  • The source remains unchanged.
  • The connection and extraction contract are versioned.

Reading the results

The source counter proves what the connector emitted. Compare it with the expected table, file or API count before adding business transforms.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

Pagination belongs to the connector. A transform never needs to know which HTTP page produced a row.

Did we answer the question?

objectiveresultwhere
Configuration explicityesthe relevant YAML block
Safe rerunyesthe destination or workflow policy
Observable resultyesthe command output and counters
Hidden manual ruleremovedthe rule now lives in versioned text

Before and after

  • Before — a partner exposes paginated orders over HTTPS requires a person to remember the order, options and checks.
  • After — one reviewed manifest and one command produce the same observable result.
  • What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

0 / 0 on this page