When an LLM application gives a bad answer, "the model messed up" is almost never a useful diagnosis. Was it the retrieval that pulled the wrong document? The prompt template? A tool that returned garbage? The model's generation itself? Without observability built for LLM systems, you can't tell — and you're left re-running prompts hoping to reproduce a non-deterministic failure. This post is about instrumenting an LLM app so that when something goes wrong, you can actually see why.
'The model messed up' is never a useful diagnosis. Why ordinary monitoring isn't enough, tracing as the spine (spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/cost), and turning aggregated traces into metrics, quality trends, and drift detection.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — and then handing it over so the customer operates it without you. The forward deployed engineer's goal, in the end, is to make themselves unnecessary: a deployment that only works while you're standing next to it hasn't actually been delivered.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — then handing it over so the customer operates it without you. The FDE's goal, in the end, is to make themselves unnecessary. Cost, latency, drift, observability, and a handover that transfers the eval discipline, not just the code.
Observability is where eBPF first went mainstream, and for good reason: it lets you see almost anything the kernel sees — syscalls, function calls, network events, disk I/O — with very low overhead and, crucially, without changing the code you're observing. That combination broke a long-standing trade-off in tracing and profiling, and it's why modern observability tools are increasingly built on eBPF.
Observability is where eBPF first went mainstream: it lets you see almost anything the kernel sees — syscalls, function calls, network events, disk I/O — with very low overhead and, crucially, without changing the code you're observing. That combination broke a long-standing trade-off in tracing and profiling, and it's why modern observability tools are increasingly built on eBPF.
You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it: what was called, what it cost, how long it took, and whether it was allowed. This post is about turning the gateway into your AI system's source of truth and its governance point.
You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it and govern it: what was called, what it cost, how long it took, whether it was allowed. Turning the gateway into your AI system's source of truth.
A pipeline isn't a build-it-once artifact; it's a production system that needs operating. It degrades — tests get flaky, builds get slow, rollbacks get rusty — and if you can't measure your delivery, you can't improve it. This closing post is about keeping a pipeline healthy over time and using the DORA metrics to know whether your CI/CD is actually working.
A pipeline isn't a build-it-once artifact; it's a production system that needs operating. It degrades — tests get flaky, builds get slow, rollbacks get rusty — and if you can't measure your delivery, you can't improve it. This closing post covers keeping a pipeline healthy, fast recovery over rare failure, observability, and using the DORA metrics to know whether your CI/CD actually works.
eBPF lets you run your own sandboxed programs inside the Linux kernel, safely, without changing kernel source or loading a risky module — and that quietly unlocks a new generation of observability, networking, and security tools. It's one of the most consequential systems technologies of the last decade, and it's worth understanding from first principles. This series builds eBPF up from the problem it solves.
eBPF lets you run your own sandboxed programs inside the Linux kernel, safely, without changing kernel source or loading a risky module — quietly unlocking a new generation of observability, networking, and security tools. It's one of the most consequential systems technologies of the last decade. This series builds eBPF up from the problem it solves.
A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — and the smartest move platform engineering makes is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default instead of each team reinventing them. Reliability becomes a paved road too.
A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — the smartest move is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default.
A model-driven agent decides its own path, which means you cannot know what it did without watching — so observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use, Strands treats seeing inside the agent as first-class, because a loop you can't see is a loop you can't trust.
A model-driven agent decides its own path, so you cannot know what it did without watching — observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use.
The gap between a LangChain demo and a LangChain production system is the same gap as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline the rest of your engineering already has. This closing post covers operating LangChain applications and the honest verdict on when to use the framework.
The gap between a LangChain demo and a production system is the same as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline your engineering already has.
A framework earns its place not in the demo but in production — under real traffic, real failures, and the need to observe, control cost, and evolve. Pydantic AI's typed, testable design carries into production well, and paired with observability and the model-agnosticism it's had all along, it makes agents you can actually operate. This closing post covers taking a Pydantic AI agent live.
A framework earns its place in production — under real traffic, failures, and the need to observe, control cost, and evolve. Pydantic AI's typed, testable design carries into production well, paired with observability and model-agnosticism.
Every lever in this playbook shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload. Measurement, and the right unit of measurement, is what turns a list of tactics into an actual reduction.
Every lever shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload, measured as cost per unit of value.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you have to manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage. This closing post turns the concepts into a way of working.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you must manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage.
In a distributed system, failure is not an exception to handle — it's the steady state. Nodes are always crashing, recovering, slowing down, and being partitioned somewhere in your cluster. Resilience is not preventing failure; it's designing so that the failures happening right now don't become the outage your users see.
In a distributed system, failure is not an exception to handle — it's the steady state. Resilience is designing so the failures happening right now don't become the outage your users see: detection, safe retries, isolation, and graceful degradation.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe — the work that starts after the demo impresses everyone.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe.
Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.
Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.
An alert that fires when nothing is actually wrong trains people to ignore alerts — and an ignored alert is worse than no alert, because it creates false confidence while the real incident scrolls past unnoticed. Good alerting is ruthlessly selective: page a human only for things that are both urgent and real, alert on what users feel, and treat every noisy alert as a bug to fix.
An alert that fires when nothing is wrong trains people to ignore alerts — and an ignored alert is worse than none. Good alerting is ruthlessly selective: page a human only for things both urgent and real, alert on what users feel, and treat every noisy alert as a bug.
"Is the system reliable?" is a useless question until you define reliability as a number. SLOs turn reliability from a vague aspiration into a measurable target, and the error budget — the small amount of unreliability you're allowed — turns it into a tool that settles the eternal fight between shipping features and keeping things stable. This is where telemetry becomes a way to run engineering, not just watch it.
'Is the system reliable?' is useless until you define reliability as a number. SLOs turn reliability into a measurable target, and the error budget — the unreliability you're allowed — settles the eternal fight between shipping features and keeping things stable.
You cannot manage what you cannot see, and the difference between a team that controls its AI spend and one that is surprised by it is almost always whether they measure cost per feature, per user, and per call.
You cannot manage what you cannot see. The difference between a team that controls its AI spend and one surprised by it is whether they measure cost per feature, per user, and per call.
Before OpenTelemetry, instrumenting a system meant picking a vendor and wiring their proprietary agent into all your code — and switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard for producing metrics, logs, and traces, so you instrument once and send the data anywhere. It's become the default way to make systems observable.
Before OpenTelemetry, instrumenting meant picking a vendor and wiring their agent into all your code — switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard, so you instrument once and send the data anywhere.
You cannot operate what you cannot see, and AI systems fail in ways ordinary monitoring misses — quality silently degrades, cost silently climbs, and inputs silently drift — so observability has to watch the things that don't throw exceptions.
AI fails in ways ordinary monitoring misses — quality degrades, cost climbs, inputs drift, all silently. Phase 8: observability that watches the signals that don't throw exceptions, plus runbooks and a kill-switch.
When a request touches ten services and comes back slow, metrics tell you it's slow and logs tell you what each service did — but neither shows you the one thing you need: where, along that journey, the time actually went. Distributed tracing is the pillar built for exactly this, following a single request across every service it touches and showing you the whole path at once.
When a request touches ten services and comes back slow, metrics say it's slow and logs say what each service did — but neither shows where the time went. Distributed tracing follows a single request across every service and shows the whole path at once.
Logs are the oldest and most detailed telemetry — the granular record of what actually happened. But the log line you write for a human to read with grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m. The shift from text logs to structured logs is the single biggest upgrade most teams can make.
Logs are the most detailed telemetry — the record of what actually happened. But the log line you write for a human to grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m.
Metrics are the cheapest, most efficient telemetry you have — a handful of numbers that summarize millions of events and tell you, at a glance, whether your system is healthy. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies to you.
Metrics are the cheapest, most efficient telemetry — a handful of numbers summarizing millions of events. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside. In a world of distributed services where failures are novel and emergent, that difference — between watching known dashboards and investigating unknown problems — is the difference between guessing and knowing.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside — the difference between watching known dashboards and investigating unknown problems.
The DevSecOps series finale — shifting right to runtime, turning compliance into code, closing the incident feedback loop, measuring what matters, and the culture that makes secure the default path.
The capstone: shifting right to continuous security — runtime detection and vulnerability management as a loop, security observability and tamper-evident audit, compliance-as-code with evidence from the pipeline, metrics that matter, and the culture (paved roads, champions).
How APIs are designed, shipped, and kept consistent at scale — the design review, the enforceable style guide, spec linting as policy-as-code, contract tests, an API catalog, and the org model that makes ten teams' APIs feel like one.
The capstone: designing and shipping consistent APIs at scale — the lifecycle, design review and style guides, linting the spec as policy-as-code in CI, contract testing so changes don't break consumers, an API catalog with ownership, and production observability.
Taking a watsonx.ai system from a notebook to production in Python — deployment spaces, reliability with retries and fallbacks, cost and throughput control, observability wired to watsonx.governance, and a hardening checklist.
Run a watsonx system in production from Python: IBM Cloud vs Cloud Pak for Data, project_id vs deployment spaces, reliability (tenacity retries, IAM token refresh, fallback), token-based cost, observability wired to watsonx.governance monitors, and securing IAM credentials.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
Taking an Amazon Bedrock Go service from a working prototype to something you can run on-call — least-privilege IAM, credentials without static keys, tuning the SDK's built-in retryer, tracking token cost, and wiring up logging and metrics with aws-sdk-go-v2.
Taking a Bedrock Go service to production: least-privilege IAM and role-based credentials, tuning the SDK's built-in retryer for throttling, token-based cost tracking, and observability via model-invocation logging, structured metrics, and request IDs.
The real skill isn't cutting a v2 — it's knowing which changes are safe to ship silently, which ones break clients you'll never meet, and how to retire an old version responsibly instead of forever.
Changing an API without breaking consumers: what counts as breaking, versioning strategies and their trade-offs (URI vs header vs additive evolution), the robustness principle and its limits, and responsible deprecation with Sunset headers and usage telemetry.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
A production RPC is more than a request and a response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and it should pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms are how a gRPC system becomes observable, secure, and resilient — and they're the pieces beginners most often skip.
A production RPC is more than a request and response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms make a gRPC system observable, secure, and resilient.
Part three of the AI Security Engineering series: protecting the data that flows through an LLM system — how sensitive information leaks out of prompts, logs, and retrieval, and the engineering controls (redaction, data minimization, per-user retrieval authz, residency choices) that actually stop it.
Protecting data in LLM systems: sensitive-information disclosure, PII in prompts and logs (your observability can be the leak), redaction with Presidio, data residency, and per-user access control on the retrieval layer so RAG doesn't leak across tenants.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
An agent that burns its turn budget without finishing is neither a success nor an error. Give it a distinct outcome and a typed event stream.
The scariest agent failure is the runaway loop that never finishes. Cap every run at max_turns and make EXHAUSTED a first-class outcome, neither a failure nor a silent success, with a typed event stream you can route on.
Instrument auth rate, settlement lag, and decline-reason breakdowns instead of raw uptime.
Instrument the payment SLOs that actually matter: auth rate, settlement lag, and decline-reason breakdowns.
Wrap every run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap model providers behind one agent.Agent.
Wrap every Microsoft Agent Framework run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap Anthropic, OpenAI, Gemini, Copilot, and Azure behind one agent.
How OpenTelemetry traces, structured logs, and token metrics turn an agent's event stream into something you can debug in production.
Seeing inside a running agent: OpenTelemetry tracing with spans for agent, model, and tool steps, structured logging, and exporting traces to debug latency and tool-call trajectories.
How WithTelemetry instruments a whole workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so it behaves like one agent.
WithTelemetry instruments a French-to-English workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so RunText drives the whole pipeline.
Turn agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
Turn Microsoft Agent Framework agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
Agents return a clean 200 OK and still be wrong, unsafe, or expensive. Why agentic AI needs a new observability layer — LLM-as-judge, safety metrics, and the four lifecycle stages.
OpenTelemetry through the Microsoft Agent Framework's configure_otel_providers, custom workflow spans, custom metrics for runs/duration/agent selection, Jaeger + Prometheus + Grafana wiring, and the set-once latch gotcha.
How to instrument agents for observability, error handling, and audit logging.
Migrate ADK callbacks to Microsoft Agent Framework composable middleware: decorators for audit logging, retry with backoff, token budget enforcement, and OpenTelemetry tracing.
How one middleware wraps every agent run in an OpenTelemetry span tagged with the gen_ai semantic attributes — the same one-line hook you use for logging.
otelprovider.NewMiddleware wraps every run in an OpenTelemetry span tagged with gen_ai attributes — the same one-line middleware hook you use for logging.
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Notes from integrating OpenTelemetry into airshipit, an open-source bare-metal Kubernetes lifecycle project with contributions from Ericsson, AT&T, Microsoft, and others. The hard part wasn't OTel; it was making distributed traces useful across foreign code.
How to turn every agent run into an OpenTelemetry span with the SDK's otelprovider middleware.
Wire otelprovider.NewMiddleware into agent.Config.Middlewares to open a gen_ai span around every run, exported to whatever TracerProvider you register globally.
Test coverage and observability are the boring infrastructure that makes the interesting changes safe. Notes on how the Picnic team built both, and the on-call experience they enabled.
Microsoft's Multi-Agent Reference Architecture in Go. Protocol, registry, bus, governance, orchestration, observability, evaluation — and how the seven hold each other up.