Where bytes live determines how fast engines read them, how much you pay, and how flexible your architecture can be. This module covers the fundamental storage decisions every data engineer makes.
The same logical table can be physically stored in two fundamentally different ways. Row-based storage keeps all columns of a single record together on disk. Columnar storage keeps all values of a single column together. This seemingly simple difference has massive implications for query performance, compression, and use case fit.
Choosing the wrong layout for your workload means either reading far more data than needed (row store for analytics) or paying a heavy penalty for single-record mutations (column store for transactions). Understanding this trade-off is foundational to every storage and format decision you will make.
| Property | Row-Based | Columnar |
|---|---|---|
| Best for | Transactions (OLTP), CRUD | Analytics (OLAP), aggregations |
| I/O pattern | Read/write full rows | Read only needed columns |
| Compression | 1.5 to 3x | 5 to 30x |
| Single-row update | Fast (localized) | Slow (scattered across files) |
| Analytical scan | Reads unused columns | Reads only queried columns |
| Concurrency | High (thousands of users) | Lower (analyst workloads) |
| Examples | PostgreSQL, MySQL, Oracle | Snowflake, BigQuery, ClickHouse, Parquet |
Parquet is an open-source columnar file format designed for efficient analytical workloads. It stores data column-by-column, enabling engines to read only the columns needed for a query. It uses row groups (128 MB default) for parallelism and per-column statistics (min/max) for predicate pushdown.
| Format | Layout | Schema | Best for | Compression |
|---|---|---|---|---|
| Parquet | Columnar | Embedded | Analytics, data lakes | Excellent |
| Avro | Row-based | Embedded (JSON) | Streaming, Kafka, schema evolution | Good |
| ORC | Columnar | Embedded | Hive ecosystem | Excellent |
| JSON | Row-based | Implicit | APIs, logs, flexibility | Poor (verbose) |
| CSV | Row-based | None | Simple interchange | Poor |
Select an architecture to understand its strengths and trade-offs.
What: Object storage (S3, ADLS, GCS) holding files in many formats (Parquet, JSON, CSV, images, video). No built-in table semantics or transaction support. Strengths: Cheapest storage option, supports any data type, decouples storage from compute. Limitation: No ACID transactions, no schema enforcement, quality depends entirely on discipline. Without governance, it becomes a "data swamp."
| Feature | Data Lake | Data Warehouse | Lakehouse |
|---|---|---|---|
| Storage | Object storage (cheap) | Proprietary (expensive) | Object storage (cheap) |
| ACID | No | Yes | Yes (via table format) |
| Schema | On read | On write | Both |
| SQL performance | Variable | Optimized | Good (improving) |
| Data types | Any (structured, semi, unstructured) | Structured only | Any |
| Vendor lock-in | Low | High | Low (open formats) |
A table format is a metadata layer on top of files (usually Parquet) that adds ACID transactions, schema evolution, time travel, and partition management to plain object storage. This is what transforms a data lake into a lakehouse.
Partitioning divides data into directories based on column values (e.g., date, region). When a query filters on the partition column, the engine skips irrelevant partitions entirely (partition pruning). Well-chosen partitions can reduce scan volume by 10 to 100x.
Frequent writes (especially streaming) create thousands of tiny files. Each file requires metadata overhead and a separate I/O operation. The result: queries slow down dramatically as the number of files grows, even if total data volume is small. Solutions include compaction (merge small files into larger ones), bin-packing, and auto-optimize features in Delta Lake and Iceberg.
1. Why is columnar storage better for analytical queries?
2. Which file format is the de facto standard for analytical data in modern data lakes?
3. What does a table format (Delta Lake, Iceberg, Hudi) add to plain files in object storage?
4. What is the "small files problem" in data lakes?
5. What distinguishes a lakehouse from a traditional data lake?