A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — and the smartest move platform engineering makes is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default instead of each team reinventing them. Reliability becomes a paved road too.
A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — the smartest move is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default.
A model-driven agent decides its own path, which means you cannot know what it did without watching — so observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use, Strands treats seeing inside the agent as first-class, because a loop you can't see is a loop you can't trust.
A model-driven agent decides its own path, so you cannot know what it did without watching — observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use.
The gap between a LangChain demo and a LangChain production system is the same gap as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline the rest of your engineering already has. This closing post covers operating LangChain applications and the honest verdict on when to use the framework.
The gap between a LangChain demo and a production system is the same as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline your engineering already has.
Every lever in this playbook shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload. Measurement, and the right unit of measurement, is what turns a list of tactics into an actual reduction.
Every lever shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload, measured as cost per unit of value.
All the theory pays off in a handful of habits that separate resilient backend code from code that falls over the first time the network misbehaves — which it will. Set timeouts on everything, reuse connections, retry idempotently, and know how to read the layers when something breaks. This closing post turns the stack into a working checklist.
All the theory pays off in a handful of habits that separate resilient backend code from code that falls over the first time the network misbehaves — which it will. Set timeouts on everything, reuse connections, retry idempotently, and debug by layer.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you have to manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage. This closing post turns the concepts into a way of working.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you must manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage.
In a distributed system, failure is not an exception to handle — it's the steady state. Nodes are always crashing, recovering, slowing down, and being partitioned somewhere in your cluster. Resilience is not preventing failure; it's designing so that the failures happening right now don't become the outage your users see.
In a distributed system, failure is not an exception to handle — it's the steady state. Resilience is designing so the failures happening right now don't become the outage your users see: detection, safe retries, isolation, and graceful degradation.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe — the work that starts after the demo impresses everyone.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe.
Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.
Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.
An alert that fires when nothing is actually wrong trains people to ignore alerts — and an ignored alert is worse than no alert, because it creates false confidence while the real incident scrolls past unnoticed. Good alerting is ruthlessly selective: page a human only for things that are both urgent and real, alert on what users feel, and treat every noisy alert as a bug to fix.
An alert that fires when nothing is wrong trains people to ignore alerts — and an ignored alert is worse than none. Good alerting is ruthlessly selective: page a human only for things both urgent and real, alert on what users feel, and treat every noisy alert as a bug.
"Is the system reliable?" is a useless question until you define reliability as a number. SLOs turn reliability from a vague aspiration into a measurable target, and the error budget — the small amount of unreliability you're allowed — turns it into a tool that settles the eternal fight between shipping features and keeping things stable. This is where telemetry becomes a way to run engineering, not just watch it.
'Is the system reliable?' is useless until you define reliability as a number. SLOs turn reliability into a measurable target, and the error budget — the unreliability you're allowed — settles the eternal fight between shipping features and keeping things stable.
You cannot manage what you cannot see, and the difference between a team that controls its AI spend and one that is surprised by it is almost always whether they measure cost per feature, per user, and per call.
You cannot manage what you cannot see. The difference between a team that controls its AI spend and one surprised by it is whether they measure cost per feature, per user, and per call.
Before OpenTelemetry, instrumenting a system meant picking a vendor and wiring their proprietary agent into all your code — and switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard for producing metrics, logs, and traces, so you instrument once and send the data anywhere. It's become the default way to make systems observable.
Before OpenTelemetry, instrumenting meant picking a vendor and wiring their agent into all your code — switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard, so you instrument once and send the data anywhere.
You cannot operate what you cannot see, and AI systems fail in ways ordinary monitoring misses — quality silently degrades, cost silently climbs, and inputs silently drift — so observability has to watch the things that don't throw exceptions.
AI fails in ways ordinary monitoring misses — quality degrades, cost climbs, inputs drift, all silently. Phase 8: observability that watches the signals that don't throw exceptions, plus runbooks and a kill-switch.
When a request touches ten services and comes back slow, metrics tell you it's slow and logs tell you what each service did — but neither shows you the one thing you need: where, along that journey, the time actually went. Distributed tracing is the pillar built for exactly this, following a single request across every service it touches and showing you the whole path at once.
When a request touches ten services and comes back slow, metrics say it's slow and logs say what each service did — but neither shows where the time went. Distributed tracing follows a single request across every service and shows the whole path at once.
Logs are the oldest and most detailed telemetry — the granular record of what actually happened. But the log line you write for a human to read with grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m. The shift from text logs to structured logs is the single biggest upgrade most teams can make.
Logs are the most detailed telemetry — the record of what actually happened. But the log line you write for a human to grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m.
Metrics are the cheapest, most efficient telemetry you have — a handful of numbers that summarize millions of events and tell you, at a glance, whether your system is healthy. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies to you.
Metrics are the cheapest, most efficient telemetry — a handful of numbers summarizing millions of events. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside. In a world of distributed services where failures are novel and emergent, that difference — between watching known dashboards and investigating unknown problems — is the difference between guessing and knowing.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside — the difference between watching known dashboards and investigating unknown problems.
How to design a system that keeps serving when its parts fail — the vocabulary of availability, the patterns that contain failure, and the Go primitives that make retries, limits, and fallbacks safe rather than dangerous.
Designing systems that survive failure: the nines and SLI/SLO/error budgets, eliminating single points of failure, timeouts and retries with backoff+jitter made safe by idempotency, circuit breakers, rate limiting and load shedding, and graceful degradation.
Why a consistent, machine-readable error model (RFC 9457 problem+json) and idempotency keys are the difference between an API clients can trust and one that quietly double-charges them.
The two things that make an API safe to build on: a consistent machine-readable error model (RFC 9457 problem+json, stable error codes, no leaked internals) and idempotency (the Idempotency-Key pattern so retries don't double-charge).
A production RPC is more than a request and a response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and it should pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms are how a gRPC system becomes observable, secure, and resilient — and they're the pieces beginners most often skip.
A production RPC is more than a request and response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms make a gRPC system observable, secure, and resilient.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
Two systems will drift. The transactional outbox stops you losing events; reconciliation is how you find the truth when they disagree anyway.
You can't atomically update your database and publish a message. The transactional outbox (or CDC) stops you losing events; reconciliation is how you find and classify the breaks when two systems drift anyway.
Talking to a payment rail is the least reliable part of your system. Treat every outbound call as fallible and every inbound webhook as hostile.
Talking to a payment rail is the least reliable part of your system. Treat every outbound call as fallible and reconcile on ambiguity; treat every inbound webhook as hostile — verify, dedupe, ack fast, and go read the truth.
The network will time out mid-transfer. The only safe assumption is that every request runs zero, one, or many times — so make "many" behave like "one."
The network times out ambiguously, so every request runs zero, one, or many times. Idempotency keys make many behave like one; full resumability lets a crashed money flow resume from its last committed step.
A transfer is a state machine with money on the line. Model the states explicitly, reserve funds before you commit, and decide up front what an overdraft even means.
A transfer is a state machine with money on the line. Model the states explicitly, reserve funds before you commit, enforce invariants at every transition, and decide up front what an overdraft even means.
An agent that burns its turn budget without finishing is neither a success nor an error. Give it a distinct outcome and a typed event stream.
The scariest agent failure is the runaway loop that never finishes. Cap every run at max_turns and make EXHAUSTED a first-class outcome, neither a failure nor a silent success, with a typed event stream you can route on.
Deterministic rules get first refusal at zero model cost. Only the unknown cases escalate to graduated AI agents. A human approves anything that mutates.
Deterministic rules get first refusal at zero cost and still fire when the model is down; only the novel long tail escalates to graduated agents; a human approves anything that mutates. Build a ladder, not a model call.
Instrument auth rate, settlement lag, and decline-reason breakdowns instead of raw uptime.
Instrument the payment SLOs that actually matter: auth rate, settlement lag, and decline-reason breakdowns.
Wrap every run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap model providers behind one agent.Agent.
Wrap every Microsoft Agent Framework run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap Anthropic, OpenAI, Gemini, Copilot, and Azure behind one agent.
How WithTelemetry instruments a whole workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so it behaves like one agent.
WithTelemetry instruments a French-to-English workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so RunText drives the whole pipeline.
Building a matching engine across the internal ledger, the processor report, and the bank statement — with tolerances, break classification, aging, and auto-resolution rules for the items nobody wants to touch.
Teaches how to build a three-way reconciliation engine (internal ledger vs processor vs bank statement): matching keys and tolerances, break classification, aging, and auto-resolution rules for the long tail.
How to coordinate a multi-step payment as an orchestrated saga: compensating actions for partial failures, idempotent steps, and a guarantee that money is never left stranded.
Teaches how to coordinate a multi-step payment across services without distributed transactions: orchestrated saga steps, compensating actions for partial failures, and guaranteeing money is never stranded.
How Visa Account Updater and Mastercard Automatic Billing Updater keep stored credentials alive when cards get reissued, expire, or change numbers.
How VAU/ABU keep stored credentials current, credential-on-file mandates, and reducing involuntary churn from expired/reissued cards.
Turn agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
Turn Microsoft Agent Framework agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
How a network authorizes on the issuer's behalf when the issuer host is unreachable — and how the books get squared afterward.
How the network authorizes on the issuer's behalf during downtime using stand-in rules and limits, then reconciles advices when the issuer returns.
How decline-code classification, backoff scheduling, retry budgets, and network-token refresh recover subscription revenue without hammering the rails.
Teaches how to build a smart retry/dunning system: decline-code classification (hard vs soft), backoff and retry-window scheduling, retry-budget limits, and network-token refresh to recover subscription revenue.
Architecting platform balances, commission splits, delayed payouts, and negative-balance recovery without ever losing a cent.
Teaches how to architect marketplace money movement: platform vs connected-account balances, fee/commission splits, delayed payouts, negative-balance handling, and merchant-of-record vs facilitator models.
How to build escrow and conditional hold/release — segregated ledger accounts, release conditions and approvals, partial releases, and expiry auto-refund that never strand money.
Teaches how to build escrow and conditional hold/release: segregated escrow ledger accounts, release conditions/approvals, partial releases, and expiry auto-refund with double-entry safety.
Agents return a clean 200 OK and still be wrong, unsafe, or expensive. Why agentic AI needs a new observability layer — LLM-as-judge, safety metrics, and the four lifecycle stages.
OpenTelemetry through the Microsoft Agent Framework's configure_otel_providers, custom workflow spans, custom metrics for runs/duration/agent selection, Jaeger + Prometheus + Grafana wiring, and the set-once latch gotcha.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces a whole category of subtle, intermittent, hard-to-debug problems — stale data that appears randomly, a cache that collapses under load at the worst moment, bugs that only happen when the cache is cold or full. This closing post catalogs the pitfalls that bite real systems, and distills the practical wisdom of the series: cache deliberately, expect the failure modes, and remember that the two genuinely hard things are still hard.
Caching giveth performance and taketh away your sanity. The same technique that makes systems fast introduces subtle, intermittent, hard-to-debug problems — stale data appearing randomly, a cache collapsing under load, bugs that only happen when the cache is cold. This catalogs the pitfalls and the practical wisdom.
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Every LLM-backed agent in this platform has a deterministic rule-based fallback. The case always finalises. The fallback isn't a workaround — it's the contract.
Balancing supply and demand keeps the grid running moment to moment — but the grid also runs on millions of physical assets (transformers, lines, substations) that age, wear, and occasionally fail, sometimes catastrophically. Keeping the grid reliable over time means anticipating and preventing those failures, not just reacting to them. This is a data problem — reams of sensor readings hinting at trouble before it happens — and it's where AI helps the grid stay reliable: predicting failures, spotting anomalies, and monitoring the vast physical system.
Balancing keeps the grid running moment to moment — but the grid also runs on millions of physical assets that age, wear, and occasionally fail catastrophically. Keeping the grid reliable means anticipating failures, not just reacting. That's a data problem, and it's where AI helps: predicting failures, spotting anomalies, and monitoring the vast physical system.
Notes from integrating OpenTelemetry into airshipit, an open-source bare-metal Kubernetes lifecycle project with contributions from Ericsson, AT&T, Microsoft, and others. The hard part wasn't OTel; it was making distributed traces useful across foreign code.
How to turn every agent run into an OpenTelemetry span with the SDK's otelprovider middleware.
Wire otelprovider.NewMiddleware into agent.Config.Middlewares to open a gen_ai span around every run, exported to whatever TracerProvider you register globally.
Test coverage and observability are the boring infrastructure that makes the interesting changes safe. Notes on how the Picnic team built both, and the on-call experience they enabled.