Interactive learning journey · 10 modules
One system.
Many moving parts.
Data engineering moves data from operational reality to trusted decisions: securely, reliably, and at a sustainable cost. Tools change; the responsibilities endure.
ENGINEERINGconnect · transform · trust
Your learning contract
By the end, you can explain the system, not just name tools.
Prerequisites: basic SQL, familiarity with files and databases, and curiosity about how data reaches a dashboard.
- Trace a record from source to consumption.
- Compare batch, CDC, and streaming patterns.
- Separate storage, file, and table formats.
- Explain Bronze, Silver, and Gold by purpose.
- Design for reruns, late data, and failures.
- Choose an architecture from requirements and trade-offs.
- Compare Inmon, Kimball, and Data Vault modeling.
- Explain Lambda vs. Kappa streaming architectures.
The constant pattern
Tools change.
The data journey does not.
Switch platforms to see the same responsibilities expressed through different services. Select a layer to zoom from the architecture down to engineering decisions.
04 · Trust
Silver · Conformed
Make data consistent, valid, and reusable across domains. Silver is where engineering quality becomes organizational trust.
- Deduplicate, validate, and quarantine
- Standardize dates, currencies, units, and codes
- Resolve identities and conform shared entities
Architect's question: Is this data trustworthy enough to reuse without relearning every source?
Learning companion · 10 modules
Architecture is the map.
Practice makes the engineer.
Each module is a self-contained deep dive. Work through them in order for the best learning progression, or jump to any topic you need to review.
Foundations & Requirements
- The data engineer's role & responsibilities
- OLTP vs. OLAP systems
- SQL and data profiling fundamentals
- Latency, volume, SLAs & constraints
- Understanding data consumers
Sources & Ingestion
- Batch, incremental & full loads
- Change Data Capture (CDC)
- Event-driven ingestion
- Schema drift & data contracts
- Tools: Airbyte, Fivetran, dlt
Storage & Formats
- Row-based vs. columnar storage
- Parquet, JSON, Avro, ORC
- Data lake, lakehouse & warehouse
- Delta Lake, Iceberg, Hudi
- Partitioning & the small files problem
Transform & Quality
- Bronze → Silver → Gold progression
- Profile, deduplicate, validate, standardize
- Data quality & testing frameworks
- Libraries: Pandas, Polars, DuckDB, PySpark
- Tools: dbt, Great Expectations, Soda
Data Modeling
- Bill Inmon: 3NF top-down approach
- Ralph Kimball: Star schema, facts & dimensions
- Data Vault 2.0: Hubs, links & satellites
- ETL vs. ELT, schema on read vs. write
- Slowly changing dimensions (SCD)
Streaming & Real-Time
- Event streams & event brokers
- Lambda vs. Kappa architecture
- Event time vs. processing time
- Watermarks & delivery semantics
- Tools: Kafka, Flink, Kafka Streams
Orchestrate & Operate
- DAGs, retries & backfills
- Idempotency & SLIs/SLOs
- Git, CI/CD & version control
- Docker & Infrastructure as Code
- Tools: Airflow, Dagster, Prefect
Govern, Secure & Observe
- PII, masking & encryption
- MDM & identity resolution
- Lineage, catalog & data contracts
- RBAC, ABAC & least privilege
- Observability, SLIs & compliance
Tools & Technology Landscape
- Databricks & Snowflake deep dive
- Apache Spark / PySpark ecosystem
- dbt, Airflow & the modern data stack
- Cloud platform comparison
- When to use what: decision frameworks
Scale, Cost & Architecture
- Architecture decision framework
- Volume, velocity, variety trade-offs
- FinOps & total cost of ownership
- The purpose: analytics, ML & AI
- Data engineers and the people bridge
The modern data stack
Where each tool fits
in the pipeline.
Tools change constantly, but responsibilities don't. This map shows leading tools organized by what they do in the data journey. Highlighted blue tools represent primary industry standards covered in detail throughout the 10 modules.
Learning companion · Connected glossary
Find the term.
Reconnect it to the system.
Definitions are more useful when students can place a concept in the architecture and explain what breaks when it fails.