Module 10 · Mastery

Scale, Cost & Architecture

Architecture is the sum of all trade-offs you chose not to revisit. This final module synthesizes everything: architectural paradigms, cost optimization, emerging patterns like Data Mesh, and how to design systems that evolve with changing requirements.

70 to 90 minAdvanced5 concepts · 1 interactive · 5 quiz questions

Architecture Paradigms

Data Mesh, Data Fabric, and centralized patterns

Interactive: Compare architecture paradigms

Select a paradigm to understand its philosophy, strengths, and when it fits.

Centralized: One team, one platform

Philosophy: A dedicated data team owns the entire data platform: ingestion, storage, transformation, and serving. Strengths: Consistent standards, economies of scale, clear ownership, simpler governance. Best for: Small to medium organizations, or companies in early data maturity stages. Challenge: The central team becomes a bottleneck as the number of data consumers and sources grows.

Data Mesh: domain-oriented, decentralized ownership

▼
What it is

Proposed by Zhamak Dehghani, Data Mesh is an organizational and architectural paradigm that treats data as a product owned by domain teams, not a centralized data team. It has four principles:

  • Domain ownership: each business domain (sales, marketing, supply chain) owns and serves its own data products
  • Data as a product: data is treated with the same rigor as software products (SLAs, documentation, discoverability)
  • Self-serve platform: a central platform team provides infrastructure and tooling that domain teams use independently
  • Federated computational governance: shared policies enforced through automation, not manual review
Trade-offs
  • Pro: Eliminates central bottleneck, scales with organizational growth, domain expertise embedded in data products
  • Con: Requires mature engineering culture across all domains, risk of inconsistency without strong governance automation
  • Reality: Most organizations adopt hybrid approaches, centralizing infrastructure while distributing ownership

Cost Optimization & FinOps

Cloud costs are engineering decisions

Where data engineering costs come from

▼
Cost categories
  • Storage: object storage (cheap), warehouse storage (moderate), hot vs. cold tiers
  • Compute: warehouse credits, Spark cluster hours, serverless function invocations
  • Data transfer: egress fees between regions, zones, or cloud providers (often the hidden cost)
  • Tooling: SaaS licenses (Fivetran, Monte Carlo, Atlan), managed services
  • People: engineering time is usually the largest cost, not infrastructure

Cost optimization strategies

▼
Practical tactics
  • Right-size compute: auto-scaling clusters, suspend idle warehouses, use spot/preemptible instances
  • Partition and prune: well-designed partitions reduce scanned data volume by 10 to 100x
  • Compress effectively: Parquet + ZSTD compression reduces storage and scan costs
  • Materialize wisely: pre-compute expensive joins as materialized views; avoid recomputing on every query
  • Lifecycle policies: move cold data to cheaper storage tiers, set retention policies, delete unused tables
  • Monitor and alert: set budget alerts, track cost per pipeline, identify expensive queries
  • Choose the right tool: DuckDB for 5 GB is free; Spark for 5 GB is wasteful

Designing Your First Production Pipeline

Putting it all together

The checklist for production readiness

▼
Before you ship, verify
  • Requirements documented: what decision does this data support? What is the SLA?
  • Idempotent: safe to re-run without duplicates or corruption
  • Tested: unit tests, data quality checks, integration tests with sample data
  • Monitored: alerting on failures, freshness checks, volume anomaly detection
  • Documented: schema descriptions, lineage, business logic, runbook for failures
  • Governed: access controls, PII handling, retention policies defined
  • Cost-aware: estimated monthly cost, cost monitoring in place
  • Version controlled: all code in Git, CI/CD pipeline configured, deployment automated
  • Recoverable: can backfill from Bronze, can roll back bad deployments

Knowledge Check: Module 10

1. What are the four principles of Data Mesh?

Data Mesh (Zhamak Dehghani) has four principles: domain ownership of data, treating data as a product, a self-serve data platform, and federated computational governance.

2. Which cloud cost is often underestimated by data engineers?

Data transfer (egress) costs are the most commonly overlooked expense. Moving data between regions, availability zones, or cloud providers can cost $0.01 to $0.09 per GB, which adds up quickly at scale.

3. What is the most effective way to reduce query costs in a columnar warehouse?

Partitioning enables the engine to skip irrelevant data (partition pruning), and selecting only needed columns leverages columnar storage. Together, they can reduce scanned data by 10 to 100x, directly reducing cost in pay-per-scan warehouses.

4. Why do most organizations adopt a hybrid approach rather than pure Data Mesh?

Data Mesh requires every domain team to have the engineering capability to own data products. Most organizations centralize infrastructure and tooling while distributing ownership gradually as teams mature.

5. Which of these is NOT part of a production readiness checklist for data pipelines?

Production readiness requires balancing performance with cost. "Maximum throughput regardless of cost" is the opposite of good engineering. Production pipelines should be right-sized, cost-aware, and optimized for the actual SLA requirements.

You have completed all 10 modules.

You now have a comprehensive foundation in data engineering: from requirements and source systems, through storage, transformation, modeling, streaming, orchestration, governance, tools, and architecture. The next step is to build.

← Return to the module hub