Module 6 – Streaming & Real-Time

Batch vs. Streaming

Two paradigms for processing data

Property	Batch	Streaming
Data model	Bounded dataset (finite)	Unbounded stream (infinite)
Latency	Minutes to hours	Seconds to milliseconds
Complexity	Lower	Higher
State management	Simple (read all, process, write)	Complex (windowing, watermarks, checkpoints)
Error recovery	Re-run the job	Checkpointing, exactly-once semantics
Cost	Pay per run	Always-on infrastructure
Use cases	Reports, training data, ETL	Fraud detection, IoT, real-time dashboards

Lambda vs. Kappa Architecture

Two competing philosophies for handling batch and stream

Lambda Architecture

Nathan Marz (2011)

Two parallel paths: batch layer + speed layer
Batch layer: processes all data periodically for accuracy
Speed layer: processes recent events for low latency
Serving layer: merges both for queries
Pro: Accurate results + low latency
Con: Two codebases to maintain (batch + stream logic)

Kappa Architecture

Jay Kreps (2014)

Single path: everything through the stream
All data treated as events in a log
Reprocess by replaying the event log
No separate batch layer
Pro: One codebase, simpler operations
Con: Requires a replayable log (Kafka). Complex aggregations may be harder.

Lambda Architecture: dual path

Source Events

↓

Batch Layer
Spark / Hive

↓

Speed Layer
Flink / Kafka Streams

↓

Serving Layer (merged view)

Kappa Architecture: single path

Source Events

→

Event Log
Kafka

→

Stream Processor
Flink

→

Serving
DB / API

Core Streaming Concepts

Time, windows, and guarantees

Event time vs. processing time

▼

What it is

Event time: when the event actually happened in the source domain (e.g., the timestamp on a user click). This is the truth you usually care about.
Processing time: when the event reaches your stream processor. This can differ from event time due to network delays, buffering, or out-of-order arrival.

Key insight

Building on processing time is simpler but can produce incorrect results when events arrive late or out of order. Building on event time gives correct results but requires watermarks to handle late arrivals: a watermark is a declaration that says "I believe all events with event time < W have now arrived."

Windowing: grouping unbounded data

▼

What it is

Since streams are infinite, you need boundaries to compute aggregates. Windows define finite chunks of time over the stream.

Window type	How it works	Example
Tumbling	Fixed, non-overlapping intervals	Count clicks per 5-minute window
Sliding (hopping)	Fixed size, overlapping by a slide interval	Average over 10 minutes, updated every 1 minute
Session	Dynamic, based on activity gaps	User session ends after 30 minutes of inactivity
Global	Single window over entire stream	Running total (requires trigger)

Delivery guarantees: at-most-once, at-least-once, exactly-once

▼

Guarantee	Risk	Use case
At-most-once	Data loss possible	Logging, metrics where loss is tolerable
At-least-once	Duplicates possible	Most pipelines (with dedup downstream)
Exactly-once	No loss, no duplicates	Financial transactions, billing

Key insight

"Exactly-once" is really "effectively-once": it is achieved through checkpointing plus idempotent writes. The processor may retry, but the combination of checkpoints and idempotent sinks ensures the effect is as if each event was processed exactly once.

Streaming Tools

Brokers and processors

Interactive: Compare streaming tools

Select a tool to understand its role and when to use it.

Apache Kafka: The distributed event log

Role: Event broker (message bus). Not a processor. What it does: Receives events from producers, stores them in durable, partitioned, replayable topics, and delivers them to consumers. Key features: High throughput, fault tolerance, replay capability, configurable retention. Best for: The backbone of event-driven architectures. Every streaming pipeline starts with a broker, and Kafka is the de facto standard.

Knowledge Check: Module 6

1. What is the main difference between Lambda and Kappa architecture?

Lambda runs batch and speed layers in parallel (dual codebase). Kappa simplifies by using a single streaming path for everything, replaying the event log when reprocessing is needed.

2. What is a watermark in stream processing?

Watermarks tell the stream processor when it is safe to close a window and emit results. A watermark at time W means "I believe all events with event_time < W have arrived." Events arriving after the watermark are called late data.

3. Which window type creates non-overlapping, fixed-duration intervals?

Tumbling windows divide the stream into fixed, non-overlapping intervals (e.g., every 5 minutes). Each event belongs to exactly one window.

4. How is "exactly-once" semantics actually achieved in practice?

True exactly-once is "effectively-once": the processor may retry events, but checkpointing (saving progress) plus idempotent sinks (writes that produce the same result regardless of how many times applied) ensure the net effect is exactly one processing per event.

5. What is the role of Apache Kafka in a streaming architecture?

Kafka is an event broker (message bus), not a processor. It receives, stores, and delivers events. Processing is done by consumers like Flink, Kafka Streams, or Spark Structured Streaming.