Module 9 – Tools & Technology Landscape

Platform Deep Dives

Databricks, Snowflake, and cloud services

Interactive: Compare platforms

Select a platform to understand its architecture, strengths, and ideal use case.

Databricks: The Lakehouse Platform

Architecture: Built on Apache Spark. Unified lakehouse platform with Delta Lake as the default table format. Runs on AWS, Azure, and GCP. Strengths: Excellent Spark performance (Photon engine), Unity Catalog for governance, integrated ML (MLflow), SQL analytics, streaming, and notebooks. Best for: Teams that need Spark at scale, ML pipelines, and a unified platform for data engineering + data science. Key feature: Delta Live Tables for declarative ETL, Unity Catalog for cross-workspace governance.

Feature	Databricks	Snowflake	BigQuery	Redshift
Architecture	Lakehouse (Spark)	Cloud warehouse	Serverless warehouse	MPP warehouse
Default format	Delta Lake	Proprietary + Iceberg	Capacitor (internal)	Proprietary columnar
Compute model	Clusters	Virtual warehouses	Serverless	Clusters
Multi-cloud	Yes	Yes	GCP only	AWS primary
ML integration	Native (MLflow)	Snowpark ML	BigQuery ML	SageMaker
Streaming	Structured Streaming	Snowpipe	Streaming inserts	Kinesis integration

Key Tools Deep Dive

Spark, dbt, Airflow, and more

Apache Spark and PySpark

▼

What it is

Apache Spark is a distributed computing framework for processing large datasets across clusters. PySpark is its Python API. Spark supports batch processing (DataFrames, Spark SQL), streaming (Structured Streaming), machine learning (MLlib), and graph processing (GraphX).

Key concepts

Lazy evaluation: transformations build a plan; actions trigger execution
Partitioning: data is distributed across executors for parallel processing
Catalyst optimizer: optimizes query plans automatically
DataFrame API: structured, SQL-like operations on distributed data
Runs on: Databricks, EMR, Dataproc, standalone clusters, Kubernetes

dbt (data build tool)

▼

What it is

dbt transforms data inside the warehouse using SQL + Jinja templating. It brings software engineering practices to analytics: version control, modular SQL models, automated testing, and auto-generated documentation. dbt does not extract or load data; it only transforms.

Core features

Models: SQL SELECT statements that define transformations
Tests: built-in (unique, not_null, accepted_values, relationships) and custom SQL tests
Documentation: auto-generated data catalog with descriptions and lineage
Materializations: table, view, incremental, ephemeral
Packages: reusable macros and models (dbt_utils, dbt_expectations)
Runs on: Snowflake, BigQuery, Redshift, Databricks, PostgreSQL, DuckDB

Cloud platform comparison: AWS vs. Azure vs. GCP

▼

Service category	AWS	Azure	GCP
Object storage	S3	ADLS Gen2	Cloud Storage (GCS)
Warehouse	Redshift	Synapse / Fabric	BigQuery
Spark	EMR	HDInsight / Synapse Spark	Dataproc
Streaming	Kinesis / MSK	Event Hubs / HDInsight Kafka	Pub/Sub / Dataflow
Orchestration	MWAA (Airflow) / Step Functions	Data Factory / Synapse Pipelines	Cloud Composer (Airflow)
Serverless functions	Lambda	Azure Functions	Cloud Functions

Decision Frameworks

How to choose the right tool

When to use what: practical guidelines

▼

Decision criteria

Data volume: GBs (Pandas/DuckDB) vs. TBs (Spark/warehouse) vs. PBs (Spark cluster)
Latency requirement: Batch (Airflow+dbt), near-real-time (Spark Streaming), real-time (Flink+Kafka)
Team skills: SQL-first teams (dbt+Snowflake), Python teams (Spark/Dagster), mixed (Databricks)
Budget model: Pay-per-query (BigQuery), reserved capacity (Redshift), usage-based (Databricks/Snowflake credits)
Vendor lock-in tolerance: Open-source stack (Iceberg+Spark+Airflow) vs. managed platform (Databricks/Snowflake)
Existing infrastructure: Cloud-native services align with your primary cloud provider

Knowledge Check: Module 9

1. What is the primary purpose of dbt in a data pipeline?

dbt handles only the T in ELT: it transforms data already loaded into the warehouse using SQL models with testing, documentation, and version control. It does not extract, load, orchestrate, or stream.

2. What differentiates Databricks from Snowflake architecturally?

Databricks is built on Apache Spark with Delta Lake as its lakehouse format. Snowflake is a purpose-built cloud warehouse with a proprietary storage engine. Both are multi-cloud, both support Python (Databricks natively, Snowflake via Snowpark).

3. For a 2 GB CSV file that needs quick analysis, which tool is the best fit?

DuckDB and Polars excel at fast, single-machine analytics on small to medium datasets. Deploying Spark, Flink, or Kafka for a 2 GB file adds unnecessary complexity and infrastructure cost.

4. What does "lazy evaluation" mean in Apache Spark?

Spark's lazy evaluation means transformations (filter, map, join) are recorded as a plan but not executed until an action (count, collect, write) triggers it. This allows the Catalyst optimizer to optimize the entire plan before executing.

5. Which factor should NOT drive your primary tool selection?

GitHub stars measure popularity, not fit. Tool selection should be driven by requirements (volume, latency), team capabilities, budget, existing infrastructure, and vendor lock-in tolerance.