Offline evaluation tells you whether a change is promising; production tells you whether it actually works. Once your system is serving real users, evaluation becomes continuous: online experiments, guardrail metrics, drift monitoring, and gating deploys on eval scores. This closing post moves evaluation from the lab into the running system and ties the whole series into a working loop.
Offline evaluation tells you whether a change is promising; production tells you whether it works. Once you're serving real users, evaluation becomes continuous — online experiments, guardrail metrics, drift monitoring, CI gating. This closing post moves evaluation from the lab into the running system and ties the series into one loop.
A system can be reliable, safe, and responsible and still fail — by being uneconomical — and the final move is turning one production system into a repeatable capability many teams can build safely.
A system can be reliable, safe, and responsible and still fail by being uneconomical. Phases 10-11: known unit economics and spend guardrails, then a paved road that lets many teams ship governed AI safely.
A system that changes itself can improve itself right off a cliff, so the evaluation and guardrails are not an afterthought to self-evolving agents — they are the thing that makes them safe to run at all.
A system that changes itself can improve right off a cliff. Measuring evolution honestly, reward hacking, drift and collapse, and the guardrails that keep it safe.
A model that scores well in evaluation still has to serve real traffic within a latency budget, isolate tenants, plan for capacity, and enforce safety in the request path — and the guardrails have to be inline, not a filter someone can route around.
A model that scores well still has to serve traffic within a latency budget, isolate tenants, plan capacity, and enforce safety in the request path — inline, not as a filter someone can route around. Phase 5.
Beyond the interactive terminal, Claude Code can run headless in scripts and CI — which unlocks automation, and raises the stakes on permissions, review, and trust.
The capstone: running Claude Code headless in scripts and CI (PR review, batch ops, scheduled jobs) — and the guardrails it demands: least privilege, sandboxing, gating the produced artifact with human review, and the series' layered recap.
Why your Terraform is a security control point, how misconfiguration scanners catch public buckets and open ingress before apply, and how to encode org guardrails as executable policy with OPA/Rego, Conftest, and Sentinel instead of a wiki page nobody reads.
Securing infrastructure definitions and enforcing guardrails automatically: IaC scanning (Checkov/tfsec/Trivy) for misconfig, drift detection, and policy-as-code with OPA/Rego + Conftest so org rules block bad infra before apply — not in a wiki.
The defensive layer that screens what goes into a model and what comes out — input rails, output rails, topical rails, and groundedness checks — plus the real tooling ecosystem and a vendor-neutral Python pipeline that wraps a model call and knows how to refuse.
The defensive layer that screens inputs and outputs: input/output/topical/groundedness rails, the real ecosystem (Llama Guard, Granite Guardian, NeMo Guardrails, Guardrails AI, Presidio, hosted moderation), and a vendor-neutral guardrail pipeline — with honest false-positive/negative trade-offs.
Adding safety to a watsonx app in Python with two complementary layers — the built-in HAP and PII moderations that watsonx.ai applies to generation, and IBM's dedicated Granite Guardian risk-detection models run as classifiers around your main model to catch harm, jailbreaks, and RAG hallucination before a response reaches a user.
Add safety to a watsonx app in Python with two layers: built-in HAP/PII moderations on generation, and Granite Guardian — IBM's dedicated risk models — run as a classifier to screen input and check RAG groundedness, wrapped in a guarded-generate flow.
Adding a safety layer to an NVIDIA-stack Python app with NeMo Guardrails — and why running it in-process, with no HTTP boundary, is the quiet advantage Python gives you over a separate guardrails server.
Add safety with NeMo Guardrails, in-process in Python: the rail types (input/output/dialog/retrieval), the config.yml + Colang model, loading with RailsConfig/LLMRails backed by a NIM model, and detecting when a rail trips instead of treating a refusal as a normal answer.
How to put Amazon Bedrock Guardrails in front of a model from Go — attaching one to a Converse call, screening raw text with ApplyGuardrail, and reading whether the guardrail actually intervened.
Guardrails for Amazon Bedrock in Go: content filters, denied topics, PII/sensitive-information filters, and contextual grounding — attaching a guardrail to a Converse call and screening arbitrary text with ApplyGuardrail, checking for intervention.
Before you attack an AI system you need a map of it: the components an adversary can influence, the trust boundaries between them, and a taxonomy that sorts attacks by goal and stage so your red-teaming is systematic instead of a grab-bag of the attacks that happen to trend that week.
Mapping the AI attack surface so red-teaming is systematic: threat-modeling the components (model, prompts, training/RAG data, tools, guardrails) and an attack taxonomy by goal (integrity/availability/privacy/abuse) and stage (training vs inference), aligned to NIST AI 100-2 and ATLAS.
The opening post of a hands-on series for builders: what it means to systematically stress-test an AI system — model, prompts, retrieval, tools, and guardrails — to surface its failures before adversaries or ordinary users do, how that differs from pentesting and robustness testing, and the frameworks and ethics that keep the work rigorous.
The opener to a defensive AI red-teaming series: stress-testing AI systems (models + prompts + RAG + tools + guardrails) to find failures before adversaries do — how it differs from pentesting, the frameworks (OWASP GenAI, MITRE ATLAS, NIST AI 100-2), and ethical rules of engagement.
A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.
The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.
Predicting who will leave is the easy half. The hard half is acting on it: treating the persuadable, respecting a budget, and proving the intervention actually kept anyone.
Predicting attrition and acting on it: churn features and labels, uplift vs propensity (treat the persuadable), and an agentic retention workflow that recommends and executes interventions with guardrails and holdout mea…
Prompt injection can hijack what a model says, but not what it's allowed to do — as long as policy lives in a middleware pipeline the model never sees.
A system prompt saying never delete records is a suggestion the model can be talked out of. A gateway that returns FORBID for the delete tool cannot. Put enforcement where the model can't reach it and prompt injection stops mattering for access control.
Large language models earn their place in regulated finance as fast, well-supervised assistants — not as autonomous agents with a hand on the money.
Where LLMs fit in regulated finance: document analysis/extraction, retrieval-grounded Q&A, and agentic decision support — plus the guardrails (grounding, human-in-the-loop, output validation, audit trails) that keep them…
An agent that burns its turn budget without finishing is neither a success nor an error. Give it a distinct outcome and a typed event stream.
The scariest agent failure is the runaway loop that never finishes. Cap every run at max_turns and make EXHAUSTED a first-class outcome, neither a failure nor a silent success, with a typed event stream you can route on.
If your gateway will block a tool call, don't make the agent discover that by trying. Hand it a capabilities brief and stop paying for turns it can't complete.
If your gateway will block a tool call, don't make the agent find out by trying, because each blocked attempt is a wasted turn. A capabilities brief aligns what the model thinks it can do with what enforcement actually allows.
An autonomous agent injects its own plan-and-execute tools at runtime. If your gateway is fail-closed, you have to find and allowlist them — deliberately.
An autonomous harness injects its own tools at runtime, tools that don't exist at build time. Fail-closed governance is only complete when it turns that blind spot into a visible block, then allowlists the known-safe internals deliberately.
The single most important pattern in LangGraph — a branch plus a back-edge, and the `recursion_limit` that keeps it from running forever.
Branching plus a back-edge is a cycle, and that cycle IS the agent loop: model proposes tool calls, tools run, control returns to the model, repeat until done. Plus recursion_limit, the guardrail that stops a runaway loop.
Stack the guardrails — callbacks, model filters, restricted tools, and clean-room sandboxing — so that if one layer misses, the next one catches
Layered defense-in-depth for agents: input/output guardrails via callbacks, Gemini safety settings, restricting tools, and sandboxing untrusted actions.
A guardrail that blocks a run before the model, chained ahead of a logger — and the upstream nil-update panic I fixed along the way.
A guardrail middleware that blocks an Microsoft Agent Framework Go run before the model, chained ahead of a logger — plus the tool-approval nil-update panic I fixed upstream in PR #472.
before/after the agent, model, and tool steps — and the single short-circuit rule that turns them into guardrails
Callbacks are lifecycle hooks around the agent, model, and tool steps — before/after each — used for guardrails (short-circuit by returning a response), logging, and mutating requests and responses.
Wrapping an agent run with async seams that log, time, guard, and short-circuit — without touching the agent's logic.
Wrapping an Microsoft Agent Framework agent run in Python with async middleware seams — timing, logging, and a guardrail that short-circuits a tool call before it runs.
This lesson teaches a middleware chain where a guardrail can block a run before the model is ever called, chained ahead of a logger.
Chain a guardrail middleware that yields its own refusal and skips the model ahead of a logger — short-circuiting a harmful request offline before Foundry is called.
All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.