Flattening customer documents from MongoDB

Context — customer profiles are nested MongoDB documents

Operations needs a tabular customer list, but addresses are nested and documents can drift over time.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you read a MongoDB collection and make the expected fields explicit?

• Keep the URI outside source control
• Select one collection
• Flatten named nested fields
• Warn when documents drift

The solution in one line

A Hydra mongodb source, a local inspection destination and one pipeline command.

Steps

1. inspect the input

1. Identify the records and the contract you need.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

{
  "customer_id": "C-14",
  "name": "Amina Diallo",
  "address": {"city": "Lille"},
  "active": true
}

2. declare the mongodb source

1. Describe the connection and extraction.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  customers:
    type: mongodb
    connection:
      uri: ${ENV:MONGODB_URI}
      database: crm
    extract:
      collection: customers
      filter:
        active: true
      batch_size: 1000
    schema:
      mode: manual
      fields:
        - {name: customer_id, path: customer_id, type: string, required: true}
        - {name: city, path: address.city, type: string}
      drift_policy: warn

3. declare a reviewable output

1. Write the extracted rows to a local CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace

4. wire source to destination

1. Connect the two identifiers.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
pipeline:
  from: customers
  to: result

5. validate without reading data

1. Check the four manifest files.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test mongodb-source

ok  sources.yaml      — mongodb
ok  destinations.yaml — csv, replace
ok  pipeline.yaml     — customers → result

✅ All tests pass — ready to execute.

6. run and read the counters

1. Execute the extraction once.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run mongodb-source

✅ Pipeline completed successfully
customer_id,city
C-14,Lille
C-18,Paris

Expected result

• 2 active profiles flattened
• The source remains unchanged.
• The connection and extraction contract are versioned.

Reading the results

The source counter proves what the connector emitted. Compare it with the expected table, file or API count before adding business transforms.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

A manual schema is a contract: unknown fields follow `drift_policy`, while required missing fields are validation events.

Did we answer the question?

Before and after

• Before — customer profiles are nested MongoDB documents requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops