Exporting the customer table from MySQL

Context — a customer directory is stored in MySQL

The CRM team needs a daily customer snapshot. The table is too large to load into memory in one pass.

Where things stand

• The current result is fragile. Its assumptions are split between tools, clicks and memory.
• Reruns are uncertain. The write mode or orchestration rule is not visible beside the data.
• Evidence is missing. A colleague cannot compare a declared rule with a concrete before and after state.

The question

How do you stream a MySQL table through Hydra with no password in the manifest?

• Use environment-backed credentials
• Read a bounded batch at a time
• Select a stable table
• Produce a replaceable snapshot

The solution in one line

A Hydra mysql source, a local inspection destination and one pipeline command.

Steps

1. inspect the input

1. Identify the records and the contract you need.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

MYSQL_HOST=localhost
MYSQL_PORT=3306
MYSQL_USER=hydra_reader
MYSQL_PASSWORD=replace-me
MYSQL_DATABASE=crm

2. declare the mysql source

1. Describe the connection and extraction.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
sources:
  customers:
    type: mysql
    connection:
      host: ${ENV:MYSQL_HOST}
      port: ${ENV:MYSQL_PORT}
      user: ${ENV:MYSQL_USER}
      password: ${ENV:MYSQL_PASSWORD}
      database: ${ENV:MYSQL_DATABASE}
    extract:
      table: customers
      batch_size: 5000

3. declare a reviewable output

1. Write the extracted rows to a local CSV.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
destinations:
  result:
    type: csv
    connection:
      base_path: "out"
    load:
      table: result.csv
      mode: replace

4. wire source to destination

1. Connect the two identifiers.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

version: "1.0"
pipeline:
  from: customers
  to: result

5. validate without reading data

1. Check the four manifest files.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl test mysql-source

ok  sources.yaml      — mysql
ok  destinations.yaml — csv, replace
ok  pipeline.yaml     — customers → result

✅ All tests pass — ready to execute.

6. run and read the counters

1. Execute the extraction once.
2. Compare the file with the explanation in the workbench.
3. Record the shown check before moving to the next step.

$ hdrctl run mysql-source

✅ Pipeline completed successfully
Rows read   : 15320
Rows written: 15320
Batches      : 4

Expected result

• 15,320 customers · four batches
• The source remains unchanged.
• The connection and extraction contract are versioned.

Reading the results

The source counter proves what the connector emitted. Compare it with the expected table, file or API count before adding business transforms.

What the counters do—and do not—prove

They prove how many records entered and left this run. They do not replace checking the target schema, the business meaning of values or the reason a workflow step was skipped.

The rule to carry forward

`batch_size` controls memory pressure; it does not add a SQL `LIMIT` to the full extraction.

Did we answer the question?

Before and after

• Before — a customer directory is stored in MySQL requires a person to remember the order, options and checks.
• After — one reviewed manifest and one command produce the same observable result.
• What is really gained — The durable gain is the contract: a colleague can read the configuration, reproduce the run and challenge the assumptions.

Close by

• sources.yaml
• transformations.yaml
• destinations.yaml
• workflow.yaml
• Tutorial — your first job
• All Guide workshops