Stream processing from first principles — computing on unbounded data in motion, the companion to the Event-Driven Architecture with Kafka series. What stream processing is (batch vs stream; unboundedness as the source of all difficulty; when latency justifies streaming over simpler batch), events/streams/time (the core dataflow model and the crucial event-time vs processing-time distinction), watermarks (reasoning about late, out-of-order events; the completeness-vs-latency tradeoff made explicit), windowing (turning the infinite stream into finite answers — tumbling, sliding, session windows; triggers), state and stateful processing (why remembering across events is streaming's hardest problem — bounding, speed, keyed partitioning, surviving failure), fault tolerance and exactly-once (at-most/at-least/exactly-once guarantees; checkpointing, replayable sources, and transactional sinks), stream processing systems (Flink, Spark Structured Streaming, Kafka Streams, cloud services, and the Beam/Dataflow model; how to choose; streaming SQL), and stream processing in practice (designing pipelines, the operational realities of a forever-running stateful service, and reaching for streaming only when latency justifies it). Grounded in Wikipedia (stream processing, batch processing, event, event-driven architecture, complex event processing, state, Apache Flink, Apache Kafka).
Most software processes data at rest: you store it, then query it. Stream processing flips that — it computes on data in motion, as each event arrives, continuously and forever. This shift from "store then process" to "process as it flows" underpins fraud detection, real-time analytics, monitoring, and the event-driven systems that react to the world as it happens. This series builds stream processing from first principles, starting with why computing on data in motion is a fundamentally different model.
Most software processes data at rest; stream processing computes on data in motion, as each event arrives, continuously and forever. Batch vs stream, why unboundedness makes streaming hard, and when latency justifies its complexity over simpler batch.
Stream processing has a small, elegant core model: everything is an event, events form streams, and time is a first-class concern. Getting these three ideas precise — especially the subtle and crucial distinction between when an event happened and when you process it — is what makes the rest of stream processing (windowing, state, correctness) comprehensible. This post builds the core vocabulary that everything else depends on.
The core model: events, streams, and time as first-class. The crucial, most-misunderstood distinction between event time (when it happened) and processing time (when you handle it), why correct answers depend on event time, and why that makes time the central problem of streaming.
Post 2 ended on a hard problem: you need event-time results, but events arrive late and out of order, so you never know when you've seen everything for a given time. Watermarks are stream processing's elegant answer — a running estimate of how far event time has progressed, letting you decide when it's safe to produce a result. They're the mechanism that makes correct event-time computation possible over a messy, unbounded stream.
Watermarks are stream processing's answer to late, out-of-order events: a running estimate of how far event time has progressed, letting you decide when it's safe to emit a result. How they make the completeness-vs-latency tradeoff explicit and tunable, and how to handle late events.
You can't average an infinite stream, or count "all" the events in something that never ends. Windowing is how stream processing carves the unbounded flow into finite, computable chunks — the last minute, each hour, each user session — so that aggregations become meaningful. It's the single most-used concept in practical stream processing, and understanding the window types is understanding how most real streaming logic is expressed.
You can't aggregate an infinite stream, so windowing carves it into finite chunks. Tumbling, sliding, and session windows; triggers that control when a window emits; and how windowing sits on event time and watermarks to produce correct, finite results — the dominant pattern in real stream processing.
Simple stream operations — filter this event, transform that one — need no memory. But the interesting computations — counts, aggregations, joins, windows, pattern detection — require remembering things across events, and that memory is state. Managing state correctly over an unbounded stream, at scale, with fault tolerance, is the hardest engineering problem in stream processing. This post is about state: why it's needed, what makes it hard, and how stream processors manage it.
Interesting stream computations — counts, aggregations, joins, windows, patterns — need to remember across events, and that memory is state. Why state is hard on an unbounded stream (bounding, speed, partitioning, surviving failure), and how engines provide managed keyed state.
A stream-processing job runs forever, which means it will experience failures — nodes crash, networks partition, processes restart. The question isn't whether failure happens but what happens to your results when it does: do you lose events, double-count them, or get exactly the right answer anyway? Fault tolerance, and the famous "exactly-once" guarantee, are what separate a toy stream processor from one you can trust with money. This post is about surviving failure correctly.
A forever-running job will fail, so the delivery guarantee matters: at-most-once (may lose), at-least-once (may duplicate), or exactly-once (correct as if neither). How checkpointing and replayable sources achieve exactly-once semantics, and choosing the guarantee by need — with idempotency as the pragmatic route.
The concepts of stream processing — events, time, windows, state, fault tolerance — are realized by a handful of systems, and choosing among them is one of the main practical decisions in building a streaming pipeline. This post maps the landscape: the engines, the models they embody, and how to think about which fits a problem. It's the bridge from understanding stream processing to actually running it.
The engines that implement stream processing: Flink (the heavyweight true stream processor), Spark Structured Streaming (micro-batch), Kafka Streams (library on Kafka), cloud-managed services, and the unifying Beam/Dataflow model. How to choose by state/correctness, ecosystem, ops capacity, and latency — plus the rise of streaming SQL.
Knowing the concepts and the engines is one thing; building a stream-processing pipeline that's correct, reliable, and operable in production is another. This closing post assembles the series into practical judgment: how to design a streaming pipeline, the operational realities that trip teams up, and when to reach for streaming at all. It turns the theory of data-in-motion into something you can actually run.
Designing a streaming pipeline (source, processing, sink) and the operational realities — it runs forever, state is the crux, backpressure and lag, reprocessing, hard testing — plus the honest judgment: use streaming only when latency genuinely justifies its complexity over simpler batch.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.