Reading an analytics extract efficiently

Context — analytics delivers a typed Parquet snapshot

A reporting job produces a compact Parquet file every night. The downstream team needs selected rows without losing the column types.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you ingest a Parquet snapshot while preserving its typed, columnar data?

• Preserve numeric and date types
• Avoid converting the source to CSV first
• Read by batches
• Keep the output reproducible

The solution in one line

A Hydra parquet source, a local inspection destination and one pipeline command.

Steps

1. inspect the input

1. Identify the records and the contract you need.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

Parquet schema
order_id: string
amount: double
ordered_at: timestamp

row groups: 2 · rows: 12,480

2. declare the parquet source

1. Describe the connection and extraction.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  orders:
    type: parquet
    connection:
      base_path: "data"
    extract:
      table: orders.parquet
      batch_size: 5000

3. declare a reviewable output

1. Write the extracted rows to a local CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace

4. wire source to destination

1. Connect the two identifiers.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
pipeline:
  from: orders
  to: result

5. validate without reading data

1. Check the four manifest files.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test parquet-source

ok  sources.yaml      — parquet
ok  destinations.yaml — csv, replace
ok  pipeline.yaml     — orders → result

✅ All tests pass — ready to execute.

6. run and read the counters

1. Execute the extraction once.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run parquet-source

✅ Pipeline completed successfully
Rows read   : 12480
Rows written: 12480
Types kept  : string, float, datetime

Expected result

• 12,480 rows read · types preserved
• The source remains unchanged.
• The connection and extraction contract are versioned.

Reading the results

The source counter proves what the connector emitted. Compare it with the expected table, file or API count before adding business transforms.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

Parquet already carries a schema. Add a `cast` only when the business type must differ from the stored type.

Did we answer the question?

Before and after

• Before — analytics delivers a typed Parquet snapshot requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops