Turning application events into rows

Context — an application exports nested events as JSON

The support team receives a JSON export containing one object per event. They need a flat, reproducible CSV for incident analysis.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you read JSON records in batches without writing a one-off conversion script?

• Keep the original JSON untouched
• Read large files in bounded batches
• Make nested-field handling explicit
• Produce a regenerable CSV

The solution in one line

A Hydra json source, a local inspection destination and one pipeline command.

Steps

1. inspect the input

1. Identify the records and the contract you need.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

[
  {"id": 101, "kind": "login", "user": {"id": "U-7"}},
  {"id": 102, "kind": "purchase", "user": {"id": "U-9"}}
]

2. declare the json source

1. Describe the connection and extraction.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  events:
    type: json
    connection:
      base_path: "data"
    extract:
      table: events.json
      batch_size: 1000
      flatten_depth: 1

3. declare a reviewable output

1. Write the extracted rows to a local CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace

4. wire source to destination

1. Connect the two identifiers.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
pipeline:
  from: events
  to: result

5. validate without reading data

1. Check the four manifest files.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test json-source

ok  sources.yaml      — json
ok  destinations.yaml — csv, replace
ok  pipeline.yaml     — events → result

✅ All tests pass — ready to execute.

6. run and read the counters

1. Execute the extraction once.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run json-source

✅ Pipeline completed successfully
id,kind,user.id
101,login,U-7
102,purchase,U-9

Expected result

• 2 rows read · 2 written
• The source remains unchanged.
• The connection and extraction contract are versioned.

Reading the results

The source counter proves what the connector emitted. Compare it with the expected table, file or API count before adding business transforms.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

`flatten_depth: 1` flattens one object level; arrays remain values and may need a later transform.

Did we answer the question?

Before and after

• Before — an application exports nested events as JSON requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops