Building event-driven systems on Apache Kafka — why EDA, the log abstraction, producers, consumers and consumer groups, delivery semantics and exactly-once, schemas and event design, the core patterns (event sourcing, CQRS, outbox, saga), and running Kafka in production.
Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.
Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.
Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.
Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.
A producer looks trivial — send a record to a topic — but the three decisions it makes (which partition, how durably, how safely on retry) determine your ordering, your durability, and whether retries create duplicates.
A producer looks trivial, but the three decisions it makes — which partition, how durably, how safely on retry — determine your ordering, your durability, and whether retries create duplicates.
A single consumer reading a topic is easy; the elegant part is how Kafka lets a group of consumers share the work automatically, rebalance when members come and go, and each remember exactly where it left off.
The elegant part of Kafka is how a group of consumers shares the work automatically, rebalances when members come and go, and each remembers exactly where it left off.
"Exactly-once" is the most misunderstood phrase in streaming — it is real in Kafka, but only within a specific boundary, and outside that boundary the honest and usually-correct answer is at-least-once plus idempotent consumers.
'Exactly-once' is the most misunderstood phrase in streaming — it's real in Kafka, but only within a specific boundary, and outside it the honest answer is at-least-once plus idempotent consumers.
In an event-driven system your events are a public API that outlives every service that reads them, so how you shape them and how you evolve them without breaking consumers is not a detail — it is the contract the whole architecture rests on.
In an event-driven system your events are a public API that outlives every service that reads them, so how you shape and evolve them without breaking consumers is the contract the whole architecture rests on.
Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration — the vocabulary of real event-driven systems.
Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration.
Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.
Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.