#Distributed Systems

Distributed systems spread computation across multiple machines to achieve fault tolerance, scalability, and geographic reach. These posts examine saga orchestration, idempotency guarantees, consensus patterns, and the Go concurrency primitives that make distributed architectures tractable in practice.

94 posts tagged with distributed systems. ← All posts

#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Agents (311)#AI Architecture (10)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (201)#AI Evaluation (8)#AI Governance (29)#AI Red Teaming (9)#AI Research (9)#AI Safety (8)#AI Security (21)#AI in Production (12)#AML (3)#API Design (10)#API Security (8)#APIs (55)#AWS (17)#Accounting (9)#Agent Skills (3)#Agentic AI (24)#Agentic Commerce (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (36)#Audit (3)#Authentication (11)#Authorization (3)#Automation (6)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (310)#Benchmarks (3)#Best Practices (3)#BigQuery (6)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (8)#Caching (10)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (24)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (8)#Collaboration (5)#Communication (9)#Compliance (52)#Computer Networking (9)#Computer Science (32)#Computer Vision (5)#Concurrency (37)#Consulting (3)#Containers (10)#Context Engineering (10)#Cost Optimisation (5)#Credit (14)#Credit Risk (14)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (8)#Custody (9)#DSPy (8)#Data (13)#Data Engineering (12)#Data Structures (9)#Databases (37)#Deployment (3)#Design Patterns (10)#DevOps (16)#DevSecOps (11)#Developer Experience (5)#Developer Tools (5)#Distributed Systems (94)#Documentation (3)#Edge AI (8)#Embeddings (15)#Emotional Intelligence (8)#Energy (8)#Engineering (11)#Engineering Culture (3)#Engineering Practices (16)#Error Handling (4)#Evaluation (53)#Event-Driven Architecture (8)#FREE-AI (8)#FX (5)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Systems (129)#Fine-Tuning (11)#Fintech (131)#Flutter (8)#Foreign Exchange (5)#Forward Deployed Engineer (8)#Forward Deployed Engineering (8)#Fraud (10)#Function Tools (5)#Functional Programming (3)#Fundraising (8)#GCP (5)#Gemma (4)#Generative AI (3)#Git (8)#Go (220)#Go-to-Market (8)#Google ADK (36)#Governance (56)#Granite (6)#GraphQL (3)#Growth (3)#Guardrails (25)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (8)#Hugging Face (8)#Human-in-the-Loop (8)#IBM watsonx (8)#Identity (11)#Integration (3)#Intellectual Property (8)#Interfaces (3)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (10)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM-as-Judge (3)#LLMs (135)#LangChain (8)#LangGraph (11)#Leadership (26)#Ledger (12)#Legal (8)#Lending (14)#LlamaIndex (8)#MCP (22)#MLOps (31)#Machine Learning (32)#Marketing (16)#Markets (4)#Memory (15)#Memory Management (5)#Metrics (6)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Monitoring (3)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (73)#Multimodal (3)#Multimodal AI (8)#NIM (5)#NVIDIA (8)#OAuth (3)#OWASP (7)#Observability (44)#On-Device AI (8)#Open Source (7)#OpenTelemetry (5)#Operating Systems (8)#Operations (10)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (54)#People (8)#Performance (43)#Platform Engineering (9)#PreSales (8)#Privacy (5)#Privacy Engineering (3)#Process (4)#Product (27)#Product Management (8)#Production (8)#Programming (10)#Programming Languages (48)#Prompt Engineering (66)#Prompt Injection (6)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#RAG (58)#RBI (3)#REST (5)#Rails (16)#Reasoning Models (8)#Reconciliation (3)#RegTech (8)#Regulation (9)#Reliability (48)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (13)#Rust (32)#SRE (21)#Sales (9)#Scalability (3)#Security (81)#Security Engineering (8)#Self-Evolving Agents (16)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Engineering (144)#Spanner (4)#Startups (30)#Strands (8)#Streaming (28)#Structured Output (4)#Sustainability (8)#System Design (32)#Systems Programming (48)#Testing (49)#Tool Use (22)#Tooling (4)#Tools (3)#Trading (8)#Treasury (6)#Type System (3)#Type Systems (11)#TypeScript (8)#Vector Databases (21)#Vector Search (11)#Venture Capital (8)#Version Control (8)#Web Development (6)#Workflows (14)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·5 min read

Remotes, Refspecs, and How Fetch and Push Work

Collaboration in Git is the same object model, stretched across two repositories. A remote is just a named URL; a remote-tracking branch is just a local pointer that remembers where the other side was; fetch and push are just object transfers plus ref updates, governed by a small syntax called the refspec. Nothing new is invented for the network — it's the graph and the pointers, reaching across a wire.

Collaboration in Git is the same object model stretched across two repositories. A remote is a named URL; a remote-tracking branch is a pointer remembering where the other side was; fetch and push are object transfers plus ref updates governed by refspecs. The network adds almost no new concepts.

Pratik Dhanave · ·6 min read

Controllers and the Reconciliation Loop

Controllers are where Kubernetes's core idea — declarative desired state plus reconciliation — becomes machinery. A controller is a loop that watches "what you want" versus "what exists" and acts to close the gap, forever. Self-healing, scaling, and zero-downtime rollouts are all just controllers doing that one thing. This is the engine of Kubernetes.

Controllers are where Kubernetes's core idea — declarative desired state plus reconciliation — becomes machinery. A controller is a loop that watches what you want versus what exists and acts to close the gap, forever. Self-healing, scaling, and rollouts are all just that.

Pratik Dhanave · ·6 min read

Failure and Resilience

In a distributed system, failure is not an exception to handle — it's the steady state. Nodes are always crashing, recovering, slowing down, and being partitioned somewhere in your cluster. Resilience is not preventing failure; it's designing so that the failures happening right now don't become the outage your users see.

In a distributed system, failure is not an exception to handle — it's the steady state. Resilience is designing so the failures happening right now don't become the outage your users see: detection, safe retries, isolation, and graceful degradation.

Pratik Dhanave · ·5 min read

Kafka in Production

Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.

Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.

Pratik Dhanave · ·7 min read

Load Balancing and Proxies

Almost nothing on the modern internet talks directly to the server that answers it. In between sit proxies and load balancers — the traffic directors that spread load across many servers, terminate TLS, cache responses, and shield your backends. Understanding this layer is understanding how a single domain name serves millions of users from hundreds of machines.

Almost nothing on the modern internet talks directly to the server that answers it. In between sit proxies and load balancers — the traffic directors that spread load, terminate TLS, cache, and shield your backends — turning one domain into a system that scales.

Pratik Dhanave · ·7 min read

Scaling Across GPUs

At some point a model doesn't fit on one GPU, or the traffic doesn't, and you have to spread inference across many. The choices — which kind of parallelism, how to place replicas, when to autoscale — are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication). Get the memory math right and most scaling decisions follow.

At some point a model doesn't fit on one GPU, or the traffic doesn't. The scaling choices are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication) — get the memory math right and most decisions follow.

Pratik Dhanave · ·6 min read

Consensus and Raft

Consensus is the problem of getting a group of unreliable machines to agree on a single value despite crashes, delays, and lost messages. It sounds narrow, but it's the hidden foundation under leader election, distributed locks, configuration, and every "exactly one node is in charge" guarantee. Raft is the algorithm that finally made it understandable.

Consensus is getting unreliable machines to agree on a single value despite crashes and lost messages — the hidden foundation under leader election, distributed locks, and every 'exactly one node is in charge' guarantee. Raft made it understandable.

Pratik Dhanave · ·5 min read

Event-Driven Patterns

Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration — the vocabulary of real event-driven systems.

Kafka gives you a durable log; these patterns are what you build on it — event sourcing, CQRS, the outbox, sagas, and the choice between choreography and orchestration.

Pratik Dhanave · ·6 min read

Transactions and Isolation

A transaction is a promise that a group of operations happens all-or-nothing and doesn't get corrupted by everyone else doing the same thing at once. Most developers know the word ACID; far fewer know that the "I" — isolation — is a dial with several settings, and that the default setting in most databases allows anomalies they've never heard of.

Most developers know ACID; far fewer know that the 'I' — isolation — is a dial with several settings, and that the default in most databases allows anomalies they've never heard of.

Pratik Dhanave · ·6 min read

Partitioning and Sharding

Replication makes copies of the whole dataset; partitioning splits the dataset into pieces so each node holds only some of it. Every large-scale system does both — and the way you choose which piece goes where quietly determines whether your load spreads evenly or one unlucky node melts down under a celebrity's traffic.

Partitioning splits a dataset into pieces so each node holds only some of it. How you choose which piece goes where decides whether load spreads evenly or one unlucky node melts down under a celebrity's traffic.

Pratik Dhanave · ·5 min read

Schemas and Event Design

In an event-driven system your events are a public API that outlives every service that reads them, so how you shape them and how you evolve them without breaking consumers is not a detail — it is the contract the whole architecture rests on.

In an event-driven system your events are a public API that outlives every service that reads them, so how you shape and evolve them without breaking consumers is the contract the whole architecture rests on.

Pratik Dhanave · ·6 min read

Replication

Replication is keeping copies of the same data on multiple nodes, and it's the answer to two different problems at once — surviving failures and serving reads at scale. The hard part is never the copying; it's what happens when the copies disagree, which they always eventually do.

Replication keeps copies of data on multiple nodes to survive failures and scale reads. The hard part is never the copying — it's what happens when the copies disagree, which they always eventually do.

Pratik Dhanave · ·5 min read

Delivery Semantics and Exactly-Once

"Exactly-once" is the most misunderstood phrase in streaming — it is real in Kafka, but only within a specific boundary, and outside that boundary the honest and usually-correct answer is at-least-once plus idempotent consumers.

'Exactly-once' is the most misunderstood phrase in streaming — it's real in Kafka, but only within a specific boundary, and outside it the honest answer is at-least-once plus idempotent consumers.

Pratik Dhanave · ·7 min read

Traces

When a request touches ten services and comes back slow, metrics tell you it's slow and logs tell you what each service did — but neither shows you the one thing you need: where, along that journey, the time actually went. Distributed tracing is the pillar built for exactly this, following a single request across every service it touches and showing you the whole path at once.

When a request touches ten services and comes back slow, metrics say it's slow and logs say what each service did — but neither shows where the time went. Distributed tracing follows a single request across every service and shows the whole path at once.

Pratik Dhanave · ·6 min read

The Write-Ahead Log and Durability

Durability — the promise that a committed transaction survives a crash — comes down to one deceptively simple rule: write down what you're about to do before you do it. The write-ahead log is that rule made concrete, and it's the reason a database can be both fast and crash-safe, two goals that otherwise pull in opposite directions.

Durability comes down to one deceptively simple rule: write down what you're about to do before you do it. The write-ahead log is that rule made concrete — the reason a database can be both fast and crash-safe.

Pratik Dhanave · ·6 min read

Time, Clocks, and Ordering

The most dangerous line of code in a distributed system is the one that trusts a timestamp. Physical clocks on different machines disagree, drift, and jump backward — so "which event happened first?" cannot be answered by comparing wall-clock times. Logical clocks answer it instead, by tracking causality rather than time.

The most dangerous line in a distributed system is the one that trusts a timestamp. Logical clocks — Lamport timestamps and vector clocks — order events by causality instead of unreliable wall-clock time.

Pratik Dhanave · ·5 min read

Consumers and Consumer Groups

A single consumer reading a topic is easy; the elegant part is how Kafka lets a group of consumers share the work automatically, rebalance when members come and go, and each remember exactly where it left off.

The elegant part of Kafka is how a group of consumers shares the work automatically, rebalances when members come and go, and each remembers exactly where it left off.

Pratik Dhanave · ·6 min read

CAP and PACELC

The CAP theorem is the most cited and most misunderstood result in distributed systems. It does not say "pick two of three." It says something narrower and more useful: when the network partitions, you must choose between consistency and availability — and PACELC completes the picture by asking what you trade even when it doesn't.

The CAP theorem doesn't say 'pick two of three.' It says that during a partition you must choose consistency or availability — and PACELC completes it by asking what you trade even when the network is healthy.

Pratik Dhanave · ·5 min read

Producers: Writing Events

A producer looks trivial — send a record to a topic — but the three decisions it makes (which partition, how durably, how safely on retry) determine your ordering, your durability, and whether retries create duplicates.

A producer looks trivial, but the three decisions it makes — which partition, how durably, how safely on retry — determine your ordering, your durability, and whether retries create duplicates.

Pratik Dhanave · ·6 min read

IP and Routing

The internet layer performs a small miracle billions of times a second: it gets a packet from any machine to any other machine on Earth, across networks owned by thousands of independent organizations, with no central controller and no guarantee it'll arrive. Understanding IP — addresses, packets, routing, and why it's deliberately unreliable — is understanding the foundation everything else is built on.

The internet layer performs a small miracle billions of times a second: it gets a packet from any machine to any other on Earth, across networks owned by thousands of organizations, with no central controller and no guarantee it'll arrive.

Pratik Dhanave · ·5 min read

Consistency Models

A consistency model is a contract between a distributed system and its users about what a read is allowed to return. It sounds abstract until you realize that every replication bug, every "why did my write disappear?" incident, and every heated architecture debate is really an argument about which model you're entitled to.

A consistency model is a contract about what a read is allowed to return. Every replication bug and 'why did my write disappear?' incident is really an argument about which model you're entitled to.

Pratik Dhanave · ·6 min read

The Log: Kafka's Core Abstraction

Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.

Almost everything Kafka does follows from one deceptively simple idea — an append-only, ordered, durable log — and once that clicks, topics, partitions, and offsets stop being jargon and become obvious.

Pratik Dhanave · ·6 min read

What Observability Is

Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside. In a world of distributed services where failures are novel and emergent, that difference — between watching known dashboards and investigating unknown problems — is the difference between guessing and knowing.

Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside — the difference between watching known dashboards and investigating unknown problems.

Pratik Dhanave · ·5 min read

Why Distributed Systems Are Hard

A distributed system is one where a machine you've never heard of failing can stop your program from working. That single property — partial failure — is the root of almost everything that makes this field hard, and pretending it away is the most common and most expensive mistake in backend engineering.

A distributed system is one where a machine you've never heard of failing can stop your program from working. That single property — partial failure — is the root of almost everything that makes the field hard.

Pratik Dhanave · ·5 min read

Why Event-Driven Architecture?

Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.

Synchronous request/response quietly welds your services together until a change in one breaks three others; event-driven architecture breaks that weld by making the event — a fact that happened — the thing services share.

Pratik Dhanave · ·5 min read

Tasks and the Task Lifecycle

Delegating real work between agents is rarely a quick round trip, so A2A makes the task a first-class object with an explicit lifecycle that survives long-running, interruptible, asynchronous collaboration.

Delegated work is rarely a quick round trip, so A2A makes the task a first-class object with an explicit lifecycle that survives long-running, interruptible, asynchronous collaboration.

Pratik Dhanave · ·15 min read

Designing a System End to End

The capstone — one problem, a home-timeline feed, designed the whole way through with the method from post one: clarify, estimate, contract, then high-level to deep-dive to bottleneck, naming the trade-off at every step and drawing on all seven earlier posts.

The capstone: one worked design end to end — requirements, estimation, API and data model, high-level architecture, and deep dives applying the whole series (scaling, caching, sharding, consistency, async, reliability) with explicit trade-offs.

Pratik Dhanave · ·17 min read

Reliability and Resilience

How to design a system that keeps serving when its parts fail — the vocabulary of availability, the patterns that contain failure, and the Go primitives that make retries, limits, and fallbacks safe rather than dangerous.

Designing systems that survive failure: the nines and SLI/SLO/error budgets, eliminating single points of failure, timeouts and retries with backoff+jitter made safe by idempotency, circuit breakers, rate limiting and load shedding, and graceful degradation.

Pratik Dhanave · ·16 min read

Asynchronous Processing and Messaging

How queues, pub/sub, and log-based streaming let systems stay responsive under load — the delivery semantics, ordering rules, backpressure, and outbox patterns that decide whether async saves you or sinks you.

Decoupling with queues and events: message queues vs pub/sub vs log-based streaming, delivery semantics (why exactly-once delivery is a myth — do idempotent processing), ordering, backpressure, the transactional outbox, and dead-letter queues.

Pratik Dhanave · ·15 min read

Consistency, Availability, and Consensus

The theory that governs distributed data, made practical — CAP stated correctly, PACELC, the full consistency spectrum with "what the user sees" examples, quorums, and Raft-style consensus without the proofs.

The theory that governs distributed data, stated correctly: CAP as a partition-time choice (not pick-2-of-3), PACELC, the consistency spectrum with what-the-user-sees examples, quorums, and consensus/Raft at an intuition level.

Pratik Dhanave · ·14 min read

Databases and Storage

Choosing and scaling the data layer without cargo-culting: how to pick relational versus NoSQL by access pattern, why every index is a tax on writes, and why your shard key is the highest-stakes decision you will make.

Choosing and scaling the data layer: relational vs NoSQL by access pattern, indexing (B-tree/hash/LSM), normalization vs denormalization, replication, partitioning/sharding and the shard-key decision, and the distributed-transaction trade-off.

Pratik Dhanave · ·15 min read

Caching

The highest-leverage tool for latency and scale — and the source of its hardest problem, invalidation. Where caches live, the patterns for filling them, how they evict, and why keeping them correct is the part that stays hard.

Caching as the highest-leverage latency tool — and its hardest problem: where caches live, the patterns (cache-aside/read-through/write-through/write-behind), eviction, and invalidation including cache stampede, penetration, and hot keys.

Pratik Dhanave · ·15 min read

Scaling Fundamentals

How systems grow under load — vertical vs horizontal scaling, why statelessness is the real enabler, load balancing from L4 to L7, consistent hashing, read/write scaling, the scale cube, and when the honest answer is "don't scale yet."

How systems grow: vertical vs horizontal scaling, statelessness as the enabler of horizontal scale, load balancing (L4/L7, consistent hashing), read/write scaling with replicas — and knowing when not to scale.

Pratik Dhanave · ·12 min read

How to Approach System Design

A repeatable method for designing systems and acing the design interview — clarify requirements, estimate on the back of an envelope, pin down the API and data model, then work high-level to deep-dive to bottleneck, always naming the trade-off.

The opener to a system-design series: a repeatable method rather than a grab-bag of components — clarifying functional vs non-functional requirements, back-of-the-envelope estimation with the latency numbers every engineer should know, and the trade-off-driven design flow.

Pratik Dhanave · ·6 min read

Card Authorization, Capture, and Clearing: One Transaction, Three Money States

How to model the card payment as a two-phase auth-then-capture flow plus a clearing tail — handling incremental auths, partial captures, reversals, expiry, and the auth-vs-settled reconciliation that trips up every ledger.

Teaches how to model the card transaction as a two-phase (auth then capture) plus clearing/settlement state machine, handling partial captures, incremental auths, auth expiry/reversal, and the auth-vs-settled amount reconciliation.

Pratik Dhanave · ·6 min read

Integrating 3-D Secure 2 Without Breaking Your Checkout

Wiring the 3DS Server, Directory Server, and issuer ACS into a handshake that produces a cryptogram your authorization message can carry.

Teaches how to integrate 3-D Secure 2: the 3DS Server, Directory Server and issuer ACS handshake, frictionless vs challenge decisioning, device data collection, and liability-shift outcomes feeding the authorization.

Pratik Dhanave · ·6 min read

Network Tokenization and How a Token Vault Shrinks Your PCI-DSS Scope

How a token vault and scheme-issued network tokens push the raw PAN out of your application systems, so most of your services fall out of PCI-DSS scope entirely.

Teaches how a token vault and network tokens (TR-31/EMV payment tokens) remove PAN from application systems, cutting PCI-DSS scope, and how token provisioning, cryptograms, and detokenization boundaries are architected.

Pratik Dhanave · ·6 min read

Engineering SEPA: Mandates, Sequence Types, and the R-Transaction State Machine

How to engineer SEPA SCT and SDD flows: mandate lifecycle storage, pre-notification timing, FIRST/RCUR sequence types, and the R-transaction taxonomy modeled as an explicit state machine.

Teaches how to engineer SEPA SCT and SDD flows: mandate lifecycle storage, pre-notification timing, FIRST/RCUR sequence types, and the R-transaction taxonomy (reject/return/refund/reversal/revocation) as a state machine.

Pratik Dhanave · ·6 min read

Threading a UETR Through SWIFT MT-to-MX Migration

How to parse legacy MT103/MT202 fields, map them to MX pacs equivalents during coexistence, and thread a gpi UETR end-to-end so a cross-border payment stays trackable across correspondent hops.

Teaches how to parse legacy MT103/MT202 fields, map them to MX pacs equivalents during coexistence, and thread a gpi UETR end-to-end so a cross-border payment is trackable across correspondent hops.

Pratik Dhanave · ·6 min read

Building for Instant Payments: RTP, FedNow, and Request-for-Payment

How to engineer for 24x7 irrevocable instant credit — synchronous ISO 20022 messaging, request-for-payment flows, and idempotent liquidity checks with no batch cutoff.

Teaches how to build for 24x7 irrevocable instant credit: synchronous ISO 20022 request/response, request-for-payment (RfP) flows, credit-transfer timeouts, and idempotent liquidity checks at the rail with no batch cutoff.

Pratik Dhanave · ·6 min read

Modeling ISO 20022 Payment Messages Without Losing Your Mind

How to treat pain, pacs, and camt as one typed, schema-driven domain instead of a pile of XML you concatenate by hand.

Teaches how to model, validate, and generate ISO 20022 XML payment messages (pain.001 initiation, pacs.008 interbank, camt.053 statements) with schema-driven typing, structured references, and idempotent message identifiers.

Pratik Dhanave · ·6 min read

Building a Byte-Level ISO 8583 Codec for Card Authorization

A field-by-field guide to decoding bitmaps, data elements, and MTI so a raw TCP frame becomes a typed auth request you can trust.

Teaches how to build a byte-level ISO 8583 encoder/decoder: primary/secondary bitmaps, data-element (DE) field definitions, MTI parsing, and stan/RRN correlation for card authorization messaging.

Pratik Dhanave · ·7 min read

Building a NACHA ACH File Processor: Records, Hash Totals, and Return Codes

Model the fixed-width record hierarchy, warehouse entries until their effective date, and turn R-series returns into automated re-presentment.

Teaches how to build a NACHA file processor: fixed-width file/batch/entry/addenda record hierarchy, hash totals, effective-entry-date windows, and handling R-series return/NOC codes with automated re-presentment logic.

Pratik Dhanave · ·7 min read

Controls, Access, and Testing a System That Moves Money

The last mile of a financial system isn't code — it's who can do what, who approves it, and how you prove the whole thing is correct.

The last mile isn't code — it's who can do what, who approves it, and how you prove it's correct. Segregation of duties, four-eyes, an auditable change trail, and property tests that assert ledger invariants over thousands of scenarios.

Pratik Dhanave · ·7 min read

Reliable Delivery: The Outbox, CDC, and Reconciliation

Two systems will drift. The transactional outbox stops you losing events; reconciliation is how you find the truth when they disagree anyway.

You can't atomically update your database and publish a message. The transactional outbox (or CDC) stops you losing events; reconciliation is how you find and classify the breaks when two systems drift anyway.

Pratik Dhanave · ·7 min read

Consuming APIs and Handling Webhooks

Talking to a payment rail is the least reliable part of your system. Treat every outbound call as fallible and every inbound webhook as hostile.

Talking to a payment rail is the least reliable part of your system. Treat every outbound call as fallible and reconcile on ambiguity; treat every inbound webhook as hostile — verify, dedupe, ack fast, and go read the truth.

Pratik Dhanave · ·6 min read

Idempotency and Full Resumability

The network will time out mid-transfer. The only safe assumption is that every request runs zero, one, or many times — so make "many" behave like "one."

The network times out ambiguously, so every request runs zero, one, or many times. Idempotency keys make many behave like one; full resumability lets a crashed money flow resume from its last committed step.

Pratik Dhanave · ·7 min read

Executing Money Flows: Invariants, Reservations, and Overdrafts

A transfer is a state machine with money on the line. Model the states explicitly, reserve funds before you commit, and decide up front what an overdraft even means.

A transfer is a state machine with money on the line. Model the states explicitly, reserve funds before you commit, enforce invariants at every transition, and decide up front what an overdraft even means.

Pratik Dhanave · ·7 min read

Audit Trails, Event Sourcing, and the GDPR Problem

In finance you never update and never delete — you only append. Event sourcing gives you a perfect audit trail; the right to be forgotten is where it fights back.

In finance you append, never update or delete. Event sourcing gives a perfect audit trail; the right to be forgotten is where it fights back — and crypto-shredding is how you reconcile the two.

Pratik Dhanave · ·6 min read

The Ledger: Double-Entry Bookkeeping

Every movement of money touches at least two accounts, and the entries always sum to zero. That one invariant is the backbone of a correct financial system.

Every transaction is balanced postings that sum to zero — enforce it at write time. The ledger is append-only, and booking, value, and settlement time are three different clocks you must not conflate.

Pratik Dhanave · ·6 min read

The Card Payment Actors Map: Who Actually Does What

One reference model of every party in a card transaction — cardholder, merchant, gateway, acquirer/processor, scheme, and issuer — and the settlement path that moves the real money back the other way.

One reference map of who does what in a card payment: cardholder, merchant, gateway, acquirer/processor, scheme, issuer, and the settlement return path.

Pratik Dhanave · ·7 min read

AP2 vs ACP vs x402 vs Network Tokens: How the Approaches Compare

A clear-eyed look at trust models, who signs what, rails, credential handling, and settlement — and why these four overlap more than they compete.

A clear-eyed comparison across trust model, who signs what, payment rails supported, credential handling, settlement, and best-fit use cases.

Pratik Dhanave · ·8 min read

Building an Agent-Ready Merchant

The surface a store must expose when the buyer is an AI agent, not a browser — and why it is the fintech reliability playbook wearing a new hat.

What a store must expose to sell to agents: a machine-readable product feed/catalog, agentic checkout endpoints, acceptance of delegated payment tokens, idempotency keys for retried agent calls, webhooks for async status…

Pratik Dhanave · ·7 min read

The Agentic Checkout Flow, End to End

How an AI agent turns a shopper's intent into a settled purchase, and where ACP and AP2 plug into the same eight-stage skeleton.

A complete walkthrough: product discovery via an AI surface → cart assembly → user approval/mandate → delegated payment token → merchant checkout → authorization → fulfillment → receipts/webhooks.

Pratik Dhanave · ·5 min read

Building a Dunning and Retry Engine for Failed Payments

How decline-code classification, backoff scheduling, retry budgets, and network-token refresh recover subscription revenue without hammering the rails.

Teaches how to build a smart retry/dunning system: decline-code classification (hard vs soft), backoff and retry-window scheduling, retry-budget limits, and network-token refresh to recover subscription revenue.

Pratik Dhanave · ·6 min read

Building Marketplace Split Payments and Connected Payouts

Architecting platform balances, commission splits, delayed payouts, and negative-balance recovery without ever losing a cent.

Teaches how to architect marketplace money movement: platform vs connected-account balances, fee/commission splits, delayed payouts, negative-balance handling, and merchant-of-record vs facilitator models.

Pratik Dhanave · ·6 min read

Engineering Escrow: Conditional Hold and Release with Double-Entry Safety

How to build escrow and conditional hold/release — segregated ledger accounts, release conditions and approvals, partial releases, and expiry auto-refund that never strand money.

Teaches how to build escrow and conditional hold/release: segregated escrow ledger accounts, release conditions/approvals, partial releases, and expiry auto-refund with double-entry safety.

Pratik Dhanave · ·7 min read

Card Networks Enter Agentic Commerce: Visa Intelligent Commerce and Mastercard Agent Pay

How Visa and Mastercard are reshaping tokenization so an AI agent can pay on your behalf — with scoped credentials, agent-aware identity, and the network doing what it has always done: authenticate, authorize, tokenize.

How the card networks are adapting tokenization for agents.

Pratik Dhanave · ·7 min read

RTGS vs Deferred Net Settlement

Two ways to move interbank money, and the engineering tradeoff that decides which one you build: settle every payment gross and pay in liquidity, or net at a window and carry settlement risk.

Teaches the engineering tradeoffs between gross real-time settlement and deferred net settlement: liquidity vs finality, queue/gridlock resolution in RTGS, and settlement-risk windows in DNS.

Pratik Dhanave · ·6 min read

Payment-versus-Payment: Settling FX Without Principal Risk

How matched-leg submission, net pay-in scheduling, and conditional simultaneous settlement remove Herstatt risk from cross-currency trades.

Teaches how payment-versus-payment settlement eliminates Herstatt (principal) risk in FX: matched trade submission, pay-in schedules, simultaneous conditional settlement, and pay-out with net funding.

Pratik Dhanave · ·6 min read

Building a Deterministic Interchange Fee Engine

Turn card transaction attributes into interchange, scheme, and markup lines that reconcile to the cent.

Teaches how to build a deterministic fee engine that classifies each transaction into an interchange category (regulated debit, rewards, CPS qualification) and computes interchange + scheme + acquirer markup for merchant statements.

Pratik Dhanave · ·6 min read

Designing a Netting Engine That Collapses Gross Obligations Into Minimal Settlement

How to turn thousands of gross obligations into the fewest net positions per counterparty — with deterministic cutoff snapshots, netting cycles, and net-debit-cap enforcement.

Teaches how to build a netting engine that collapses many gross obligations into minimal net settlement positions per counterparty, handling netting cycles, cutoff snapshots, and net-debit-cap enforcement.

Pratik Dhanave · ·6 min read

Modeling Card Chargebacks as a State Machine

Modeling the full card dispute lifecycle with reason codes, evidence deadlines, representment, arbitration, and provisional-credit ledger entries at every transition.

Teaches how to model the full card dispute lifecycle as a state machine with reason codes, evidence deadlines, representment, pre-arbitration and arbitration, and provisional-credit ledger entries at each transition.

Pratik Dhanave · ·7 min read

Batch vs Streaming

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in large chunks on a schedule — simpler, cheaper, and fine when yesterday's data is good enough. Stream processing handles data continuously as it arrives — more complex and costly, but necessary when you need to know now. Choosing between them (and knowing when each fits) is one of the most consequential architectural decisions in a data platform, and it's driven by real requirements, not by which sounds more impressive.

How fresh does the data need to be? That one question splits data engineering into two paradigms. Batch processing handles data in chunks on a schedule — simpler and cheaper. Stream processing handles data continuously as it arrives — more complex, but necessary when you need to know now.

Pratik Dhanave · ·9 min read

Distributed Caching

A cache inside a single application process is easy — but it doesn't scale, and every instance of your app has its own separate copy. The moment you run many application servers, you want a shared cache they all use, which means a cache that lives across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces the distributed-systems problems that a local cache never had. Understanding distributed caching is understanding how caching works at real scale.

A cache inside a single process is easy — but it doesn't scale, and every instance of your app has its own separate copy. Run many servers and you want a shared cache across the network on its own machines: a distributed cache like Redis or Memcached. This unlocks scale and sharing, but introduces distributed-systems problems.

Pratik Dhanave · ·8 min read

Cache Invalidation

This is the hard one. "There are only two hard things in computer science: cache invalidation and naming things" names it directly — cache invalidation is genuinely, notoriously difficult. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync (or deciding how much drift you'll tolerate) is a problem with no clean, universal solution. Understanding why it's hard, and the strategies for managing it, is the difference between caching that helps and caching that causes baffling bugs.

This is the hard one. 'There are only two hard things in computer science: cache invalidation and naming things' names it directly. The moment you cache a copy of data, you've created a second source of truth that can drift from the first, and keeping them in sync has no clean universal solution.

Pratik Dhanave · ·8 min read

Demand-Side Flexibility

For a century, grid operation had one basic move: adjust supply to follow demand. The renewable era adds a second, transformative move — adjust demand to follow supply. If you can shift when electricity is used to when clean power is abundant, you turn demand from a fixed constraint into a flexible resource that helps balance the grid. Orchestrating that flexibility across millions of devices and distributed resources is a massive coordination problem, and it's one of the most exciting frontiers for AI in energy.

For a century, grid operation had one basic move: adjust supply to follow demand. The renewable era adds a transformative second move — adjust demand to follow supply. If you can shift when electricity is used to when clean power is abundant, demand becomes a flexible resource that helps balance the grid. Orchestrating that across millions of devices is a massive AI coordination problem.

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.