A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — and the smartest move platform engineering makes is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default instead of each team reinventing them. Reliability becomes a paved road too.
A platform that ships fast but falls over isn't a good platform. Reliability is a first-class platform capability — the smartest move is to build observability and SRE practices into the platform, so every service gets monitoring, SLOs, and reliability by default.
A model-driven agent decides its own path, which means you cannot know what it did without watching — so observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use, Strands treats seeing inside the agent as first-class, because a loop you can't see is a loop you can't trust.
A model-driven agent decides its own path, so you cannot know what it did without watching — observability isn't a nice-to-have in Strands, it's a requirement. Built on OpenTelemetry and shaped by AWS's own production use.
The gap between a LangChain demo and a LangChain production system is the same gap as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline the rest of your engineering already has. This closing post covers operating LangChain applications and the honest verdict on when to use the framework.
The gap between a LangChain demo and a production system is the same as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline your engineering already has.
A framework earns its place not in the demo but in production — under real traffic, real failures, and the need to observe, control cost, and evolve. Pydantic AI's typed, testable design carries into production well, and paired with observability and the model-agnosticism it's had all along, it makes agents you can actually operate. This closing post covers taking a Pydantic AI agent live.
A framework earns its place in production — under real traffic, failures, and the need to observe, control cost, and evolve. Pydantic AI's typed, testable design carries into production well, paired with observability and model-agnosticism.
Every lever in this playbook shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload. Measurement, and the right unit of measurement, is what turns a list of tactics into an actual reduction.
Every lever shares one precondition: you have to measure. The vendor percentages are directional signposts, not promises — the only number that governs your bill is the one from your own workload, measured as cost per unit of value.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you have to manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage. This closing post turns the concepts into a way of working.
Knowing the pillars is not the same as having an observable system. In practice, observability is built incrementally, costs real money you must manage, and only pays off if the whole team treats telemetry as part of building software — not something added after the outage.
In a distributed system, failure is not an exception to handle — it's the steady state. Nodes are always crashing, recovering, slowing down, and being partitioned somewhere in your cluster. Resilience is not preventing failure; it's designing so that the failures happening right now don't become the outage your users see.
In a distributed system, failure is not an exception to handle — it's the steady state. Resilience is designing so the failures happening right now don't become the outage your users see: detection, safe retries, isolation, and graceful degradation.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe — the work that starts after the demo impresses everyone.
A four-line RAG demo and a production RAG system share almost no operational concerns. Getting to production means treating ingestion as a pipeline, retrieval quality as something you measure, and the whole system as something you observe.
Kafka's defaults will run; whether they'll survive a broker failure, a traffic spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor — that are far cheaper to make now than to retrofit later.
Kafka's defaults will run; whether they survive a broker failure, a spike, or a year of growth depends on a handful of decisions — replication, durability, partitioning, and what you monitor.
An alert that fires when nothing is actually wrong trains people to ignore alerts — and an ignored alert is worse than no alert, because it creates false confidence while the real incident scrolls past unnoticed. Good alerting is ruthlessly selective: page a human only for things that are both urgent and real, alert on what users feel, and treat every noisy alert as a bug to fix.
An alert that fires when nothing is wrong trains people to ignore alerts — and an ignored alert is worse than none. Good alerting is ruthlessly selective: page a human only for things both urgent and real, alert on what users feel, and treat every noisy alert as a bug.
"Is the system reliable?" is a useless question until you define reliability as a number. SLOs turn reliability from a vague aspiration into a measurable target, and the error budget — the small amount of unreliability you're allowed — turns it into a tool that settles the eternal fight between shipping features and keeping things stable. This is where telemetry becomes a way to run engineering, not just watch it.
'Is the system reliable?' is useless until you define reliability as a number. SLOs turn reliability into a measurable target, and the error budget — the unreliability you're allowed — settles the eternal fight between shipping features and keeping things stable.
You cannot manage what you cannot see, and the difference between a team that controls its AI spend and one that is surprised by it is almost always whether they measure cost per feature, per user, and per call.
You cannot manage what you cannot see. The difference between a team that controls its AI spend and one surprised by it is whether they measure cost per feature, per user, and per call.
Before OpenTelemetry, instrumenting a system meant picking a vendor and wiring their proprietary agent into all your code — and switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard for producing metrics, logs, and traces, so you instrument once and send the data anywhere. It's become the default way to make systems observable.
Before OpenTelemetry, instrumenting meant picking a vendor and wiring their agent into all your code — switching vendors meant re-instrumenting everything. OpenTelemetry ended that: one open, vendor-neutral standard, so you instrument once and send the data anywhere.
You cannot operate what you cannot see, and AI systems fail in ways ordinary monitoring misses — quality silently degrades, cost silently climbs, and inputs silently drift — so observability has to watch the things that don't throw exceptions.
AI fails in ways ordinary monitoring misses — quality degrades, cost climbs, inputs drift, all silently. Phase 8: observability that watches the signals that don't throw exceptions, plus runbooks and a kill-switch.
When a request touches ten services and comes back slow, metrics tell you it's slow and logs tell you what each service did — but neither shows you the one thing you need: where, along that journey, the time actually went. Distributed tracing is the pillar built for exactly this, following a single request across every service it touches and showing you the whole path at once.
When a request touches ten services and comes back slow, metrics say it's slow and logs say what each service did — but neither shows where the time went. Distributed tracing follows a single request across every service and shows the whole path at once.
Logs are the oldest and most detailed telemetry — the granular record of what actually happened. But the log line you write for a human to read with grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m. The shift from text logs to structured logs is the single biggest upgrade most teams can make.
Logs are the most detailed telemetry — the record of what actually happened. But the log line you write for a human to grep is nearly useless at scale; the one you write as structured data for a machine to query is the one that saves you at 3 a.m.
Metrics are the cheapest, most efficient telemetry you have — a handful of numbers that summarize millions of events and tell you, at a glance, whether your system is healthy. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies to you.
Metrics are the cheapest, most efficient telemetry — a handful of numbers summarizing millions of events. Their power is aggregation; their trap is cardinality; and knowing which numbers to watch (and how to read percentiles) is the difference between a dashboard that warns you and one that lies.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside. In a world of distributed services where failures are novel and emergent, that difference — between watching known dashboards and investigating unknown problems — is the difference between guessing and knowing.
Monitoring tells you whether the things you thought to check are okay. Observability lets you ask questions you never anticipated about a system you can't see inside — the difference between watching known dashboards and investigating unknown problems.
The DevSecOps series finale — shifting right to runtime, turning compliance into code, closing the incident feedback loop, measuring what matters, and the culture that makes secure the default path.
The capstone: shifting right to continuous security — runtime detection and vulnerability management as a loop, security observability and tamper-evident audit, compliance-as-code with evidence from the pipeline, metrics that matter, and the culture (paved roads, champions).
How APIs are designed, shipped, and kept consistent at scale — the design review, the enforceable style guide, spec linting as policy-as-code, contract tests, an API catalog, and the org model that makes ten teams' APIs feel like one.
The capstone: designing and shipping consistent APIs at scale — the lifecycle, design review and style guides, linting the spec as policy-as-code in CI, contract testing so changes don't break consumers, an API catalog with ownership, and production observability.
Taking a watsonx.ai system from a notebook to production in Python — deployment spaces, reliability with retries and fallbacks, cost and throughput control, observability wired to watsonx.governance, and a hardening checklist.
Run a watsonx system in production from Python: IBM Cloud vs Cloud Pak for Data, project_id vs deployment spaces, reliability (tenacity retries, IAM token refresh, fallback), token-based cost, observability wired to watsonx.governance monitors, and securing IAM credentials.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
Taking an Amazon Bedrock Go service from a working prototype to something you can run on-call — least-privilege IAM, credentials without static keys, tuning the SDK's built-in retryer, tracking token cost, and wiring up logging and metrics with aws-sdk-go-v2.
Taking a Bedrock Go service to production: least-privilege IAM and role-based credentials, tuning the SDK's built-in retryer for throttling, token-based cost tracking, and observability via model-invocation logging, structured metrics, and request IDs.
The real skill isn't cutting a v2 — it's knowing which changes are safe to ship silently, which ones break clients you'll never meet, and how to retire an old version responsibly instead of forever.
Changing an API without breaking consumers: what counts as breaking, versioning strategies and their trade-offs (URI vs header vs additive evolution), the robustness principle and its limits, and responsible deprecation with Sunset headers and usage telemetry.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
A production RPC is more than a request and a response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and it should pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms are how a gRPC system becomes observable, secure, and resilient — and they're the pieces beginners most often skip.
A production RPC is more than a request and response. Every call should carry a deadline so it can't hang forever, metadata for cross-cutting concerns like auth and tracing, and pass through interceptors that apply logging, authentication, and metrics uniformly. These three mechanisms make a gRPC system observable, secure, and resilient.
Part three of the AI Security Engineering series: protecting the data that flows through an LLM system — how sensitive information leaks out of prompts, logs, and retrieval, and the engineering controls (redaction, data minimization, per-user retrieval authz, residency choices) that actually stop it.
Protecting data in LLM systems: sensitive-information disclosure, PII in prompts and logs (your observability can be the leak), redaction with Presidio, data residency, and per-user access control on the retrieval layer so RAG doesn't leak across tenants.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
Where good eval cases actually come from — seeding by hand, harvesting from production telemetry, and curating a golden dataset in Go that doesn't rot the moment your prompt changes.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
How to pause a workflow for a human decision, package a whole workflow as an agent, and see exactly what a run did — through OpenTelemetry spans and a rendered graph — in Microsoft Agent Framework.
An agent that burns its turn budget without finishing is neither a success nor an error. Give it a distinct outcome and a typed event stream.
The scariest agent failure is the runaway loop that never finishes. Cap every run at max_turns and make EXHAUSTED a first-class outcome, neither a failure nor a silent success, with a typed event stream you can route on.
Instrument auth rate, settlement lag, and decline-reason breakdowns instead of raw uptime.
Instrument the payment SLOs that actually matter: auth rate, settlement lag, and decline-reason breakdowns.
Wrap every run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap model providers behind one agent.Agent.
Wrap every Microsoft Agent Framework run in an OpenTelemetry span, gate risky tool actions behind a permission handler, and swap Anthropic, OpenAI, Gemini, Copilot, and Azure behind one agent.
How OpenTelemetry traces, structured logs, and token metrics turn an agent's event stream into something you can debug in production.
Seeing inside a running agent: OpenTelemetry tracing with spans for agent, model, and tool steps, structured logging, and exporting traces to debug latency and tool-call trajectories.
How WithTelemetry instruments a whole workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so it behaves like one agent.
WithTelemetry instruments a French-to-English workflow with OpenTelemetry spans, then agentworkflow.NewAgent wraps the graph so RunText drives the whole pipeline.
Turn agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
Turn Microsoft Agent Framework agent runs into OpenTelemetry spans, block prompt injection with information-flow control, and swap model providers behind one Agent API.
Agents return a clean 200 OK and still be wrong, unsafe, or expensive. Why agentic AI needs a new observability layer — LLM-as-judge, safety metrics, and the four lifecycle stages.
OpenTelemetry through the Microsoft Agent Framework's configure_otel_providers, custom workflow spans, custom metrics for runs/duration/agent selection, Jaeger + Prometheus + Grafana wiring, and the set-once latch gotcha.
How to instrument agents for observability, error handling, and audit logging.
Migrate ADK callbacks to Microsoft Agent Framework composable middleware: decorators for audit logging, retry with backoff, token budget enforcement, and OpenTelemetry tracing.
How one middleware wraps every agent run in an OpenTelemetry span tagged with the gen_ai semantic attributes — the same one-line hook you use for logging.
otelprovider.NewMiddleware wraps every run in an OpenTelemetry span tagged with gen_ai attributes — the same one-line middleware hook you use for logging.
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Notes from integrating OpenTelemetry into airshipit, an open-source bare-metal Kubernetes lifecycle project with contributions from Ericsson, AT&T, Microsoft, and others. The hard part wasn't OTel; it was making distributed traces useful across foreign code.
How to turn every agent run into an OpenTelemetry span with the SDK's otelprovider middleware.
Wire otelprovider.NewMiddleware into agent.Config.Middlewares to open a gen_ai span around every run, exported to whatever TracerProvider you register globally.
Test coverage and observability are the boring infrastructure that makes the interesting changes safe. Notes on how the Picnic team built both, and the on-call experience they enabled.
Microsoft's Multi-Agent Reference Architecture in Go. Protocol, registry, bus, governance, orchestration, observability, evaluation — and how the seven hold each other up.