#Kafka

Apache Kafka provides distributed event streaming for high-throughput data pipelines. Articles cover Kafka consumer group design in Kubernetes, payment settlement stream processing, and the operational patterns for running Kafka-backed architectures with exactly-once delivery guarantees.

11 posts tagged with kafka. ← All posts

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Agents (335)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (282)#AI Evaluation (9)#AI Financial Engineering (14)#AI Gateway (8)#AI Governance (40)#AI Red Teaming (9)#AI Research (9)#AI Safety (12)#AI Security (29)#AI in Production (12)#AML (3)#API Design (10)#API Security (8)#APIs (59)#AWS (17)#Accountability (8)#Accounting (9)#Agent Skills (11)#Agentic AI (25)#Agentic Commerce (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (42)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (11)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (310)#Benchmarks (3)#Best Practices (4)#BigQuery (6)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (16)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (25)#Causal Inference (8)#Cellular Automata (8)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (9)#Coding Agents (8)#Collaboration (5)#Communication (9)#Complexity (8)#Compliance (54)#Computer Networking (9)#Computer Science (40)#Computer Vision (15)#Computer-Use Agents (8)#Concurrency (40)#Consulting (5)#Containers (10)#Context Engineering (14)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (16)#Credit Risk (16)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (24)#Data Science (8)#Data Structures (10)#Databases (41)#Deep Learning (20)#Deployment (5)#Design (3)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (5)#Developer Tools (5)#Differential Privacy (3)#Distributed Systems (100)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (14)#Engineering Culture (3)#Engineering Judgment (7)#Engineering Practices (16)#Error Handling (4)#Evaluation (68)#Event-Driven Architecture (8)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (129)#Fine-Tuning (12)#Fintech (131)#Flutter (8)#Forecasting (10)#Foreign Exchange (5)#Forward Deployed Engineer (22)#Forward Deployed Engineering (8)#Fraud (10)#Function Tools (5)#Functional Programming (3)#Fundamentals (23)#Fundraising (8)#GCP (5)#GPU (8)#GUI Automation (8)#Gemma (4)#Generative AI (3)#Git (8)#Go (220)#Go-to-Market (13)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (34)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (8)#Hugging Face (8)#Human-in-the-Loop (9)#IBM watsonx (8)#Identity (11)#Inference (8)#Information Retrieval (8)#Integration (3)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (11)#Knowledge Graphs (10)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (8)#LLMs (192)#LangChain (8)#LangGraph (11)#Leadership (29)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (22)#MLOps (36)#Machine Learning (109)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (15)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Mixture of Experts (5)#Model Risk (3)#Monitoring (5)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (73)#Multimodal (3)#Multimodal AI (11)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (50)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenTelemetry (5)#Operating Systems (9)#Operations (13)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (54)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (9)#Policy as Code (3)#PreSales (8)#Privacy (13)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (8)#Production (22)#Programming (10)#Programming Languages (48)#Prompt Engineering (80)#Prompt Injection (15)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (22)#RAG (70)#RBI (3)#REST (5)#Rails (16)#Real-Time (8)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (3)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (56)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (14)#Risk Management (11)#Rust (32)#SLSA (3)#SRE (23)#Sales (9)#Scalability (6)#Search (10)#Security (102)#Security Engineering (8)#Self-Evolving Agents (16)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (152)#Spanner (4)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Stream Processing (8)#Streaming (36)#Structured Output (4)#Supply Chain Security (9)#Sustainability (8)#System Design (32)#Systems Programming (56)#Testing (66)#Threat Modeling (3)#Time Series (9)#Tool Use (23)#Tooling (5)#Tools (4)#Trading (8)#Treasury (6)#Trust (3)#Type System (3)#Type Systems (11)#TypeScript (8)#Validation (3)#Valuation (3)#Vector Databases (22)#Vector Search (11)#Venture Capital (8)#Version Control (9)#Voice AI (9)#Web Development (6)#Workflow (4)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·5 min read

Kafka in Production

Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.

Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.

#Kafka#Distributed Systems#Backend Engineering#Observability#Event-Driven Architecture#Reliability
Pratik Dhanave · ·5 min read

Event-Driven Patterns

Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration — the vocabulary of real event-driven systems.

Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration.

#Kafka#Event-Driven Architecture#Distributed Systems#Software Architecture#Backend Engineering#Multi-Agent Systems
Pratik Dhanave · ·5 min read

Schemas and Event Design

In an event-driven system your events are a public API that outlives every service that reads them, so how you shape them and how you evolve them without breaking consumers is not a detail — it is the contract the whole architecture rests on.

In an event-driven system your events are a public API that outlives every service that reads them, so how you shape and evolve them without breaking consumers is the contract the whole architecture rests on.

#Kafka#Event-Driven Architecture#API Design#Backend Engineering#Distributed Systems#APIs
Pratik Dhanave · ·5 min read

Delivery Semantics and Exactly-Once

"Exactly-once" is the most misunderstood phrase in streaming — it is real in Kafka, but only within a specific boundary, and outside that boundary the honest and usually-correct answer is at-least-once plus idempotent consumers.

'Exactly-once' is the most misunderstood phrase in streaming — it's real in Kafka, but only within a specific boundary, and outside it the honest answer is at-least-once plus idempotent consumers.

#Kafka#Distributed Systems#Backend Engineering#Event-Driven Architecture#Streaming
Pratik Dhanave · ·5 min read

Consumers and Consumer Groups

A single consumer reading a topic is easy; the elegant part is how Kafka lets a group of consumers share the work automatically, rebalance when members come and go, and each remember exactly where it left off.

The elegant part of Kafka is how a group of consumers shares the work automatically, rebalances when members come and go, and each remembers exactly where it left off.

#Kafka#Backend Engineering#Distributed Systems#Event-Driven Architecture
Pratik Dhanave · ·5 min read

Producers: Writing Events

A producer looks trivial — send a record to a topic — but the three decisions it makes (which partition, how durably, how safely on retry) determine your ordering, your durability, and whether retries create duplicates.

A producer looks trivial, but the three decisions it makes — which partition, how durably, how safely on retry — determine your ordering, your durability, and whether retries create duplicates.

#Kafka#Backend Engineering#Distributed Systems#Event-Driven Architecture
Pratik Dhanave · ·6 min read

The Log: Kafka's Core Abstraction

Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.

Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.

#Kafka#Distributed Systems#Backend Engineering#Event-Driven Architecture
Pratik Dhanave · ·5 min read

Why Event-Driven Architecture?

Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.

Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.

#Kafka#Event-Driven Architecture#Distributed Systems#Backend Engineering
Pratik Dhanave · ·16 min read

Asynchronous Processing and Messaging

How queues, pub/sub, and log-based streaming let systems stay responsive under load — the delivery semantics, ordering rules, backpressure, and outbox patterns that decide whether async saves you or sinks you.

Decoupling with queues and events: message queues vs pub/sub vs log-based streaming, delivery semantics (why exactly-once delivery is a myth — do idempotent processing), ordering, backpressure, the transactional outbox, and dead-letter queues.

#System Design#Messaging#Event-Driven#Kafka#Distributed Systems#Streaming
Pratik Dhanave · ·7 min read

Stream Processing Systems — The Engines That Run It

The concepts of stream processing — events, time, windows, state, fault tolerance — are realized by a handful of systems, and choosing among them is one of the main practical decisions in building a streaming pipeline. This post maps the landscape: the engines, the models they embody, and how to think about which fits a problem. It's the bridge from understanding stream processing to actually running it.

The engines that implement stream processing: Flink (the heavyweight true stream processor), Spark Structured Streaming (micro-batch), Kafka Streams (library on Kafka), cloud-managed services, and the unifying Beam/Dataflow model. How to choose by state/correctness, ecosystem, ops capacity, and latency — plus the rise of streaming SQL.

#Stream Processing#Apache Flink#Apache Kafka#Data Engineering#Real-Time#Kafka#Streaming#Databases

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.