A financial AI prototype that works in a demo is a long way from a system an institution trusts with capital. Productionizing financial AI means meeting the reliability, latency, security, and operational bars of a regulated financial environment — where a silent failure loses money, an outage stops trading, and a breach is catastrophic. This post is about the engineering that turns a promising model into a system people actually rely on.
A prototype that works in a demo is far from a system an institution trusts with capital. Productionizing financial AI means meeting the reliability, latency, security, and operational bars of a regulated environment where silent failures lose money. The posture: trust is the actual product. (Educational, not financial advice.)
The final discipline of LLMOps is running the thing as a product that costs real money, has to be fast enough to use, breaks in new ways, and is expected to get better over time. Cost and latency aren't afterthoughts you optimize later — in an LLM app they're first-class constraints that shape the architecture. And incidents aren't failures of planning; they're a certainty you prepare for. This closing post ties the series together into the operating posture that keeps a living system healthy.
Running an LLM app means treating cost and latency as first-class constraints (model routing, token hygiene, caching, streaming), preparing for incidents as certainties (provider failures, instant rollback, blameless postmortems), and the closing idea: operating is continuous improvement — a tightening loop, because an LLM app is never done.
When an LLM application gives a bad answer, "the model messed up" is almost never a useful diagnosis. Was it the retrieval that pulled the wrong document? The prompt template? A tool that returned garbage? The model's generation itself? Without observability built for LLM systems, you can't tell — and you're left re-running prompts hoping to reproduce a non-deterministic failure. This post is about instrumenting an LLM app so that when something goes wrong, you can actually see why.
'The model messed up' is never a useful diagnosis. Why ordinary monitoring isn't enough, tracing as the spine (spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/cost), and turning aggregated traces into metrics, quality trends, and drift detection.
An LLM can say anything, which is exactly the problem. The same open-endedness that makes it useful means it can hallucinate, leak data, follow a malicious instruction hidden in its input, or produce something harmful — and in production, in front of real users, those aren't edge cases, they're inevitabilities. Guardrails are the controls that sit around the model to contain what it might do. This post covers the new failure modes LLMs introduce and the layered defenses that keep a production system safe.
An LLM can say anything, which is the problem. The new failure modes — hallucination, prompt injection, data leakage, harmful output — and layered, fail-closed guardrails (input, output, action) with humans in the loop, treated as part of the lifecycle and red-teamed.
Offline evaluation tells you whether a change is safe to ship; online evaluation tells you whether it actually worked. No matter how good your eval set is, production always surprises you — real users send inputs you never imagined, and the gap between your offline scores and real-world quality is where LLM apps quietly fail. This post is about measuring quality on live traffic and turning user behavior into the feedback that makes the system improve.
Offline evals say a change is safe to ship; online evals say whether it worked. The intrinsic offline–online gap, deploying to measure (canary, A/B, shadow), capturing explicit and implicit feedback, and the pipeline that turns production behavior into continuous improvement.
The single practice that separates teams who improve their LLM apps from teams who just change them is evaluation. Without a trustworthy way to measure quality, every prompt tweak is a gamble — you fix one case, break two others, and never know. Eval-driven development makes the eval set the center of gravity: you build it before you optimize, you gate every change on it, and you treat it as the most valuable asset the project owns. This post is about building evaluation you can actually trust.
The practice that separates improving an LLM app from merely changing it. The curated eval set as the core asset, how to score open-ended output (deterministic checks, reference metrics, LLM-as-judge, human review), and how to gate every change on evals and compare B vs A.
In an LLM application the prompt is the source code — it's where behavior is defined and where most changes and most bugs happen. Yet teams routinely paste prompts inline, edit them in production, and have no idea which version produced last week's output. Prompt and version management is the unglamorous discipline of treating prompts as the first-class, versioned, tested artifacts they actually are. It's the foundation the rest of the lifecycle stands on.
In an LLM app the prompt is the source code — where behavior is defined and most bugs happen. Treat prompts like code: version the whole configuration (template + model + parameters + context strategy), decouple them from app code, and route every change through review, eval, rollout, and rollback.
Post 1 argued an LLM application is never "done." This post gives that idea a shape: the LLMOps lifecycle, the repeating loop of develop → evaluate → deploy → observe → improve that a production LLM system lives inside. Seeing the whole loop at once is what turns a pile of practices — prompt management, evals, guardrails, monitoring — into a coherent operating model, and it's the map for the rest of the series.
The LLMOps lifecycle — develop, evaluate, deploy, observe, improve — as a loop an LLM app lives inside, not a line it runs once. Why evaluation is a first-class stage appearing twice, why production feedback flows backward, and why the loop turns whether you act or not.
Getting an LLM feature to work in a demo is easy; keeping it working, safe, and affordable in production for a year is a different discipline entirely. That discipline is LLMOps — the operational practices for running LLM-powered applications in production. It inherits a lot from MLOps, but the differences are sharp enough that treating an LLM app like a traditional ML model is one of the most common and expensive mistakes teams make. This series is about the operational layer that the demo never shows you.
Getting an LLM to work in a demo is easy; keeping it working, safe, and affordable for a year is a discipline. What LLMOps is, how it differs from MLOps, and the five properties of LLMs — opaque external models, prompt-defined behavior, fuzzy correctness, non-determinism, new failures/costs — that force a different playbook.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — and then handing it over so the customer operates it without you. The forward deployed engineer's goal, in the end, is to make themselves unnecessary: a deployment that only works while you're standing next to it hasn't actually been delivered.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — then handing it over so the customer operates it without you. The FDE's goal, in the end, is to make themselves unnecessary. Cost, latency, drift, observability, and a handover that transfers the eval discipline, not just the code.
Building a good model is maybe half the work; running a recommender in production is the other half. Real systems must serve in milliseconds, stay fresh as the catalog and tastes change, handle cold start gracefully, resist the filter bubbles their own optimization creates, and be monitored like any critical service. This closing post assembles everything into what it takes to run a recommender for real — and how to build one.
Building a good model is half the work; running a recommender in production is the other half. Real systems must serve in milliseconds, stay fresh as the catalog and tastes change, handle cold start gracefully, resist the filter bubbles their own optimization creates, and be monitored like any critical service. This closing post assembles everything — and gives a practical path to building one.
A voice demo that works in a quiet room with a good headset is a long way from a voice agent that survives a noisy phone call from a real customer. Production voice AI has to handle bad audio, unpredictable humans, failures at every stage, and the peculiar demands of telephony — and it has to be evaluated in ways text systems never require. This closing post is about making a voice agent real.
A voice demo that works in a quiet room with a good headset is a long way from a voice agent that survives a noisy phone call from a real customer. Production voice AI has to handle bad audio, unpredictable humans, failures at every stage, and the peculiar demands of telephony — and be evaluated in ways text systems never require. Making a voice agent real.
Governance that stops at deployment governs the wrong moment. An AI system's real behavior — and its real risks — emerge in production, over time, as the world drifts and users do things no test anticipated. Post-deployment governance is the discipline of watching systems in the wild, catching when they go wrong, and maintaining a clear line of accountability for what they do. This post is about governance as an ongoing operational practice, not a launch checklist.
Governance that stops at deployment governs the wrong moment — risk emerges in production as the world drifts and users find the edges. Monitoring the governance-relevant signals (drift, continuous bias, safety) with thresholds tied to action, plus accountability for a system's whole life: durable owners, AI incident response, traceability, and redress.
Knowing the algorithms is maybe 20% of doing machine learning. The other 80% is the workflow around them: framing the problem, preparing the data, iterating honestly, and getting a model into production where it keeps working. This closing post assembles the series into the end-to-end process real ML follows — and names the places where projects actually fail, which are almost never the algorithm.
Algorithms are ~20% of ML; the other 80% is the workflow — framing the problem, preparing data, iterating honestly, and keeping a model working in production where it decays (concept drift). Why classical ML isn't obsolete, and why these foundations are universal to deep learning and LLMs.
Offline evaluation tells you whether a change is promising; production tells you whether it actually works. Once your system is serving real users, evaluation becomes continuous: online experiments, guardrail metrics, drift monitoring, and gating deploys on eval scores. This closing post moves evaluation from the lab into the running system and ties the whole series into a working loop.
Offline evaluation tells you whether a change is promising; production tells you whether it works. Once you're serving real users, evaluation becomes continuous — online experiments, guardrail metrics, drift monitoring, CI gating. This closing post moves evaluation from the lab into the running system and ties the series into one loop.
Taking a watsonx.ai system from a notebook to production in Python — deployment spaces, reliability with retries and fallbacks, cost and throughput control, observability wired to watsonx.governance, and a hardening checklist.
Run a watsonx system in production from Python: IBM Cloud vs Cloud Pak for Data, project_id vs deployment spaces, reliability (tenacity retries, IAM token refresh, fallback), token-based cost, observability wired to watsonx.governance monitors, and securing IAM credentials.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
A gRPC service that works on localhost is a long way from one that runs reliably at scale. Production raises questions localhost never does: how do calls get load-balanced when connections are long-lived? How do you secure them, expose them to browsers, observe them, and evolve the contract without breaking anyone? This closing post covers what it takes to run gRPC for real.
A gRPC service that works on localhost is far from one that runs reliably at scale. Production raises questions localhost never does: how do calls get load-balanced when connections are long-lived? How do you secure them, expose them to browsers, observe them, and evolve the contract without breaking anyone? This closing post covers running gRPC for real.
Taking an Amazon Bedrock Go service from a working prototype to something you can run on-call — least-privilege IAM, credentials without static keys, tuning the SDK's built-in retryer, tracking token cost, and wiring up logging and metrics with aws-sdk-go-v2.
Taking a Bedrock Go service to production: least-privilege IAM and role-based credentials, tuning the SDK's built-in retryer for throttling, token-based cost tracking, and observability via model-invocation logging, structured metrics, and request IDs.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
Every LLM-backed agent in this platform has a deterministic rule-based fallback. The case always finalises. The fallback isn't a workaround — it's the contract.
Multi-agent stacks have state: vector indexes, chat histories, agent memory. GKE for AI workloads needs StatefulSets, PVCs, gateway controllers, and the patterns that work in 2026.