Module 4 – Transform & Quality

The Medallion Architecture

Bronze, Silver, and Gold

Progressive data refinement

Bronze
Raw, as-landed

→

Silver
Conformed, trusted

→

Gold
Business-ready

Bronze: the raw safety net

▼

What it is

Bronze stores an auditable copy of source data with minimal transformation. The goal is preserving original fidelity plus ingestion metadata (load timestamp, source system, batch ID). Bronze is your safety net: if anything downstream breaks, you can always rebuild from here.

What happens at Bronze

Land data in its original structure (or close to it)
Add ingestion metadata: load_timestamp, source_system, file_name
Store as Parquet in object storage, partitioned by ingestion date
Quarantine corrupt or malformed records (don't silently drop them)
Keep history: append-only or versioned for replay capability

Silver: conformed and trusted

▼

What it is

Silver is where engineering quality becomes organizational trust. Data is cleaned, deduplicated, validated, and conformed so that multiple teams can reuse it without relearning every source.

What happens at Silver

Deduplication: remove exact duplicates and resolve near-duplicates
Validation: type checks, range checks, null constraints, referential integrity
Standardization: unify date formats, currency codes, country names, casing
Identity resolution: conform customer IDs, product codes across sources
Schema enforcement: apply schema on write, reject non-conforming records

Gold: business-ready models

▼

What it is

Gold organizes data around decisions, KPIs, products, and business questions. It is not merely clean data; it carries shared analytical meaning. Gold tables are the ones analysts query, dashboards consume, and ML models train on.

What happens at Gold

Build dimensional models (facts and dimensions)
Create domain-specific aggregates and metrics
Materialize joined views for query performance
Apply business logic: calculated fields, fiscal calendars, segments
Publish to consumption layer with clear documentation

Processing Libraries

Choose the right engine for the data volume

Interactive: Compare processing libraries

Select a library to understand when to use it and what it excels at.

Pandas: The Python data workhorse

Type: Single-machine DataFrame library. Best for: Data under ~10 GB, prototyping, ad-hoc analysis, data science workflows. Strengths: Huge ecosystem, familiar API, integrates with everything. Limitation: Single-threaded by default, loads everything into memory, does not scale to large datasets. For bigger data, consider Polars or PySpark.

Library	Scale	Language	Best for	Key strength
Pandas	GBs	Python	Prototyping, small data	Ecosystem, familiarity
Polars	10s of GBs	Rust (Python API)	Fast single-machine work	Speed, memory efficiency
DuckDB	10s of GBs	SQL (Python API)	Local OLAP, Parquet queries	SQL on files, zero setup
PySpark	TBs to PBs	Python / Scala	Distributed processing	Horizontal scale
dbt	Warehouse scale	SQL + Jinja	Warehouse transforms	Testing, docs, modularity

Data Quality

Measure trust, don't assume it

The six dimensions of data quality

▼

What to measure

Completeness: Are expected values present? What is the null rate per column?
Uniqueness: Are primary keys actually unique? Are there duplicate records?
Validity: Do values conform to defined rules? (e.g., age > 0, email matches pattern)
Accuracy: Do values match reality? (hardest to measure without a reference source)
Consistency: Do related datasets agree? (same customer count in orders and CRM?)
Freshness: Is the data current enough for its intended use case?

Quality tools: Great Expectations, Soda, dbt tests

▼

Tools for enforcement

Great Expectations: Python framework for defining, running, and documenting data expectations. Write tests like "expect_column_values_to_not_be_null" and generate data docs.
Soda: SodaCL language for checks like "row_count > 0", freshness checks, anomaly detection. Integrates with orchestrators.
dbt tests: Built-in and custom SQL tests (unique, not_null, accepted_values, relationships). Run as part of the dbt build process.
Monte Carlo / Elementary: Data observability platforms that detect anomalies automatically (schema changes, volume drops, distribution shifts).

Knowledge Check: Module 4

1. What is the primary purpose of the Bronze layer?

Bronze is the safety net: it preserves source data in its original form plus ingestion metadata. If anything downstream breaks, you can always rebuild from Bronze.

2. Which library would you choose for processing a 500 GB dataset distributed across a cluster?

PySpark is designed for distributed processing across clusters and can handle terabytes to petabytes. Pandas and DuckDB are single-machine tools and would struggle with 500 GB.

3. What distinguishes Silver from Gold in the medallion architecture?

Silver makes data trustworthy (deduplicated, validated, standardized). Gold shapes that trusted data into business models (facts, dimensions, metrics) designed for specific consumption use cases.

4. Which of these is NOT one of the six dimensions of data quality?

The six dimensions are: Completeness, Uniqueness, Validity, Accuracy, Consistency, and Freshness. Velocity is a big data concept (volume, velocity, variety) but not a quality dimension.

5. What is dbt primarily used for in a data pipeline?

dbt (data build tool) is a SQL-first transformation framework. It brings software engineering practices (testing, documentation, version control, modularity) to warehouse transformations. It does not handle ingestion, orchestration, or streaming.