Module 7 – Orchestrate & Operate

DAGs and Orchestration

Dependency management for data pipelines

What is a DAG?

▼

What it is

A Directed Acyclic Graph defines the execution order of tasks in a pipeline. Each node is a task (extract, transform, load, test), and edges define dependencies (task B runs after task A). "Acyclic" means no circular dependencies: the graph always moves forward.

What the orchestrator does

Scheduling: trigger DAGs on a cron schedule, event trigger, or manual run
Dependency resolution: only run a task when all upstream dependencies succeed
Retry logic: automatically retry failed tasks with configurable backoff
Alerting: notify teams on failure via Slack, email, or PagerDuty
Backfill: re-run historical DAG runs for past dates
Observability: logs, run history, task duration, success/failure rates

Interactive: Compare orchestration tools

Select a tool to understand its philosophy and trade-offs.

Apache Airflow: The industry standard

Philosophy: DAGs-as-code in Python. Strengths: Massive community, 1000+ provider packages, battle-tested at scale, managed options (MWAA, Cloud Composer, Astronomer). Best for: Teams that need a proven orchestrator with extensive integrations. Trade-off: Configuration complexity, task-centric (not data-aware), UI can be overwhelming for beginners.

Idempotency & Reliability

Safe to re-run, every time

Idempotency: the most important pipeline property

▼

What it is

A pipeline is idempotent if running it multiple times with the same input produces the same result. No duplicates, no side effects, no corruption. This is critical because in production, pipelines will fail and need to be re-run. If re-running creates duplicates or double-counts, your data is wrong.

How to achieve it

MERGE / upsert: use INSERT ON CONFLICT or MERGE to handle duplicates gracefully
Partition overwrite: delete and replace the partition for the run date rather than appending
Deterministic keys: compute surrogate keys from business keys, not auto-increment sequences
Functional transforms: same input always produces same output (no random IDs, no wall-clock timestamps)

Handling late data and backfills

▼

What it is

Late data arrives after its expected processing window. Backfills re-run historical pipeline runs to fix data, apply new logic, or onboard new sources. Both are routine in production, not exceptions.

Strategies

Parameterize by date: every DAG run processes a specific execution_date, making backfills a loop over past dates
Idempotent writes: enables safe re-runs without manual cleanup
Late-arrival windows: keep a grace period for stragglers before marking data as final
Incremental models: dbt incremental materializations process only new/changed rows

CI/CD, Git, and Testing

Software engineering practices for data

Git workflows for data pipelines

▼

Best practices

Version control everything: DAG definitions, dbt models, schema migrations, config files
Branch strategy: feature branches, pull requests, code review before merge to main
CI checks: linting (sqlfluff), unit tests, dbt compile/test on PR
CD deployment: merge to main triggers deployment to staging, then production
Infrastructure as Code: Terraform or Pulumi for warehouse, buckets, IAM roles

Testing data pipelines

▼

Test type	What it checks	Tool examples
Unit tests	Individual transform logic in isolation	pytest, dbt unit tests
Data quality tests	Completeness, uniqueness, validity of output	dbt tests, Great Expectations, Soda
Integration tests	End-to-end pipeline on sample data	CI pipeline with fixture data
Contract tests	Schema compatibility between producer and consumer	Schema Registry, dbt contracts
Performance tests	Query time, resource usage under load	Benchmarks, warehouse profiling

Knowledge Check: Module 7

1. What does it mean for a pipeline to be idempotent?

Idempotency means re-running a pipeline on the same input produces identical output: no duplicates, no side effects, no corruption. This is essential because production pipelines will be re-run after failures.

2. What is a DAG in the context of orchestration?

A DAG (Directed Acyclic Graph) defines the dependency structure of pipeline tasks. Each task is a node, edges define order, and "acyclic" ensures no circular dependencies.

3. Which strategy ensures idempotent writes when processing daily data?

Partition overwrite deletes and replaces the entire partition for a given execution date, so re-running the pipeline for the same date produces identical results. Appending without dedup creates duplicates.

4. What is the primary advantage of Dagster over Airflow?

Dagster is built around "software-defined assets" (the data products) rather than tasks. This makes lineage, testing, and observability native concepts, whereas Airflow is task-centric and requires extra tooling for data awareness.

5. What should happen in CI when a developer opens a pull request for a dbt model change?

CI should automatically lint SQL (sqlfluff), compile dbt models, and run tests against a staging environment. This catches errors before they reach production and provides confidence for code reviewers.