LLMOps in practice — the hands-on operational discipline for running LLM-powered applications in production, the operational companion to The AI Production Roadmap. What LLMOps is (MLOps adapted for foundation models you didn't train, where behavior comes from prompting/retrieval/orchestration; the five properties that force a different playbook — opaque external models that change under you, prompt-defined behavior, fuzzy correctness, non-determinism, and new failures/token-costs), the LLMOps lifecycle (develop → evaluate → deploy → observe → improve as a loop the app lives inside; evaluation as a first-class stage appearing twice; production feedback flowing backward so the loop has memory; the loop turning whether you act or not), prompt and version management (the prompt is the source code; version the whole configuration — template + model + parameters + context strategy; decouple prompts from code; route every change through review, eval, rollout, rollback), eval-driven development (the curated eval set as the core asset; scoring open-ended output with deterministic checks, reference metrics, LLM-as-judge, and human review; gating every change and comparing B vs A; avoiding overfitting to the set), online evaluation and feedback (the offline–online gap; deploying to measure via canary, A/B, and shadow testing; explicit vs implicit feedback; the feedback pipeline that turns production behavior into improvement), guardrails and safety in production (the new failure modes — hallucination, prompt injection, data leakage, harmful output; layered fail-closed guardrails at input, output, and action; humans in the loop; red-teaming; guardrails as part of the lifecycle), observability and tracing (why ordinary monitoring isn't enough; tracing as the spine — spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/latency/cost; aggregated into metrics, quality trends, drift detection, and cost attribution), and operating in production (cost and latency as first-class constraints — model routing, token hygiene, caching, batching, streaming, the cost–latency–quality triangle; preparing for incidents as certainties — provider failures, instant rollback, blameless postmortems; and the closing idea that operating is continuous improvement — a tightening loop, because an LLM app is never done). Grounded in Wikipedia (MLOps, large language model, CI/CD, version control, configuration management, evaluation of machine translation, test-driven development, A/B testing, RLHF, prompt injection, AI safety, observability, concept drift, site reliability engineering, incident management).
Getting an LLM feature to work in a demo is easy; keeping it working, safe, and affordable in production for a year is a different discipline entirely. That discipline is LLMOps — the operational practices for running LLM-powered applications in production. It inherits a lot from MLOps, but the differences are sharp enough that treating an LLM app like a traditional ML model is one of the most common and expensive mistakes teams make. This series is about the operational layer that the demo never shows you.
Getting an LLM to work in a demo is easy; keeping it working, safe, and affordable for a year is a discipline. What LLMOps is, how it differs from MLOps, and the five properties of LLMs — opaque external models, prompt-defined behavior, fuzzy correctness, non-determinism, new failures/costs — that force a different playbook.
Post 1 argued an LLM application is never "done." This post gives that idea a shape: the LLMOps lifecycle, the repeating loop of develop → evaluate → deploy → observe → improve that a production LLM system lives inside. Seeing the whole loop at once is what turns a pile of practices — prompt management, evals, guardrails, monitoring — into a coherent operating model, and it's the map for the rest of the series.
The LLMOps lifecycle — develop, evaluate, deploy, observe, improve — as a loop an LLM app lives inside, not a line it runs once. Why evaluation is a first-class stage appearing twice, why production feedback flows backward, and why the loop turns whether you act or not.
In an LLM application the prompt is the source code — it's where behavior is defined and where most changes and most bugs happen. Yet teams routinely paste prompts inline, edit them in production, and have no idea which version produced last week's output. Prompt and version management is the unglamorous discipline of treating prompts as the first-class, versioned, tested artifacts they actually are. It's the foundation the rest of the lifecycle stands on.
In an LLM app the prompt is the source code — where behavior is defined and most bugs happen. Treat prompts like code: version the whole configuration (template + model + parameters + context strategy), decouple them from app code, and route every change through review, eval, rollout, and rollback.
The single practice that separates teams who improve their LLM apps from teams who just change them is evaluation. Without a trustworthy way to measure quality, every prompt tweak is a gamble — you fix one case, break two others, and never know. Eval-driven development makes the eval set the center of gravity: you build it before you optimize, you gate every change on it, and you treat it as the most valuable asset the project owns. This post is about building evaluation you can actually trust.
The practice that separates improving an LLM app from merely changing it. The curated eval set as the core asset, how to score open-ended output (deterministic checks, reference metrics, LLM-as-judge, human review), and how to gate every change on evals and compare B vs A.
Offline evaluation tells you whether a change is safe to ship; online evaluation tells you whether it actually worked. No matter how good your eval set is, production always surprises you — real users send inputs you never imagined, and the gap between your offline scores and real-world quality is where LLM apps quietly fail. This post is about measuring quality on live traffic and turning user behavior into the feedback that makes the system improve.
Offline evals say a change is safe to ship; online evals say whether it worked. The intrinsic offline–online gap, deploying to measure (canary, A/B, shadow), capturing explicit and implicit feedback, and the pipeline that turns production behavior into continuous improvement.
An LLM can say anything, which is exactly the problem. The same open-endedness that makes it useful means it can hallucinate, leak data, follow a malicious instruction hidden in its input, or produce something harmful — and in production, in front of real users, those aren't edge cases, they're inevitabilities. Guardrails are the controls that sit around the model to contain what it might do. This post covers the new failure modes LLMs introduce and the layered defenses that keep a production system safe.
An LLM can say anything, which is the problem. The new failure modes — hallucination, prompt injection, data leakage, harmful output — and layered, fail-closed guardrails (input, output, action) with humans in the loop, treated as part of the lifecycle and red-teamed.
When an LLM application gives a bad answer, "the model messed up" is almost never a useful diagnosis. Was it the retrieval that pulled the wrong document? The prompt template? A tool that returned garbage? The model's generation itself? Without observability built for LLM systems, you can't tell — and you're left re-running prompts hoping to reproduce a non-deterministic failure. This post is about instrumenting an LLM app so that when something goes wrong, you can actually see why.
'The model messed up' is never a useful diagnosis. Why ordinary monitoring isn't enough, tracing as the spine (spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/cost), and turning aggregated traces into metrics, quality trends, and drift detection.
The final discipline of LLMOps is running the thing as a product that costs real money, has to be fast enough to use, breaks in new ways, and is expected to get better over time. Cost and latency aren't afterthoughts you optimize later — in an LLM app they're first-class constraints that shape the architecture. And incidents aren't failures of planning; they're a certainty you prepare for. This closing post ties the series together into the operating posture that keeps a living system healthy.
Running an LLM app means treating cost and latency as first-class constraints (model routing, token hygiene, caching, streaming), preparing for incidents as certainties (provider failures, instant rollback, blameless postmortems), and the closing idea: operating is continuous improvement — a tightening loop, because an LLM app is never done.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.