Every AI forward deployed engineer builds one-offs — a bespoke deployment for one customer's data, workflow, and trust. The ones who create lasting value turn those one-offs into product: the patterns that repeat become a platform, the platform makes the next deployment faster, and the field learnings flow back to shape what gets built. This closing post is about the flywheel that turns bespoke AI work into a compounding asset, and the career arc of the engineer who runs it.
Every AI FDE builds one-offs — a bespoke deployment for one customer's data, workflow, and trust. The ones who create lasting value turn those one-offs into product: the patterns that repeat become a platform, the platform makes the next deployment faster, and field learnings flow back to shape what gets built. The flywheel that turns bespoke AI work into a compounding asset — and the AI FDE career arc.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — and then handing it over so the customer operates it without you. The forward deployed engineer's goal, in the end, is to make themselves unnecessary: a deployment that only works while you're standing next to it hasn't actually been delivered.
A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — then handing it over so the customer operates it without you. The FDE's goal, in the end, is to make themselves unnecessary. Cost, latency, drift, observability, and a handover that transfers the eval discipline, not just the code.
A technically excellent AI system that nobody uses has delivered zero value. The last mile of an AI deployment is not the model — it's fitting the system into how real people actually do their jobs, designing an interface that handles uncertainty honestly, and managing the human change of introducing AI into someone's work. This is where deployments succeed or quietly fail, and where the forward deployed engineer's non-technical skills matter most.
A technically excellent AI system that nobody uses has delivered zero value. The last mile of an AI deployment is not the model — it's fitting the system into how real people actually do their jobs, designing an interface that handles uncertainty honestly, and managing the human change of introducing AI into someone's work. Human-in-the-loop, calibrated reliance, autonomy levels, and change management.
You cannot ship AI you cannot measure, and no enterprise grants a probabilistic system authority over real work on faith. Both problems have the same answer: evaluation. Building the customer's own evaluation set — real examples, their definition of correct — is how the AI forward deployed engineer turns "it seemed good in the demo" into a reliability number, and that number is how trust gets earned.
You cannot ship AI you cannot measure, and no enterprise grants a probabilistic system authority over real work on faith. Both problems have the same answer: evaluation. Building the customer's own eval set — real examples, their definition of correct — turns 'it seemed good in the demo' into a reliability number, and that number is how trust gets earned. Offline/online eval, safe failure, and trust.
A frontier model knows the public internet and nothing about the customer. All the value of an AI deployment comes from the opposite: making the model reason over the customer's own documents, records, and knowledge. Grounding is the technical heart of the AI forward deployed engineer's job — connecting a general model to a specific company's messy, permissioned, incomplete data so its answers are about their reality, not the model's imagination.
A frontier model knows the public internet and nothing about the customer. All the value of an AI deployment comes from the opposite: making the model reason over the customer's own documents, records, and knowledge. Grounding — retrieval-augmented generation over messy, permissioned, incomplete data — is the technical heart of the AI FDE's job. With an interactive reference-architecture diagram.
An AI demo is the easiest impressive thing to build and the most misleading. It runs on hand-picked inputs, in a clean environment, with the failures edited out — and it convinces everyone the problem is nearly solved when the real work has barely begun. The AI forward deployed engineer's job in this phase is to use the demo to win belief, then walk the customer honestly across the chasm to a pilot that survives real data.
An AI demo is the easiest impressive thing to build and the most misleading: it runs on cherry-picked inputs, in a clean environment, with the failures edited out — and convinces everyone the problem is nearly solved when the real work has barely begun. Use the demo to win belief, then walk the customer honestly across the chasm to a pilot that survives real data.
The most important decision an AI forward deployed engineer makes happens before any code: which problem to point the model at. Choose a problem AI is genuinely suited for, with real value and a clear way to measure it, and the engagement can succeed. Choose AI theater — impressive-sounding but ill-fit — and no amount of engineering saves it. Scoping is where AI deployments are won or lost.
The most important decision an AI FDE makes happens before any code: which problem to point the model at. Choose a problem AI is genuinely suited for, with real value and a clear way to measure it, and the engagement can succeed. Choose AI theater — impressive-sounding but ill-fit — and no engineering saves it. Fit vs value, resisting AI theater, augmentation over automation, and picking the wedge.
The forward deployed engineer was born at Palantir to bridge powerful software and messy customer reality. In the AI era the role has exploded, because frontier models have made that gap wider than ever: a model that dazzles in a demo is a long way from a system that works inside one company's data, workflows, and trust constraints. This series is the technical playbook for the engineer who closes that gap.
The forward deployed engineer was born at Palantir to bridge powerful software and messy customer reality. In the AI era the role exploded, because frontier models widened that gap: a model that dazzles in a demo is a long way from a system that works inside one company's data, workflows, and trust constraints. This series is the technical playbook for the engineer who closes that gap.
The cascaded pipeline — ASR, then LLM, then TTS — has powered voice agents for years, but it has an inherent ceiling: every stage adds latency and every conversion loses information. A newer architecture collapses the three models into one that goes from speech directly to speech, promising lower latency and richer understanding. This post compares the cascade with the emerging end-to-end approach, and where each fits.
The cascaded pipeline — ASR, then LLM, then TTS — has powered voice agents for years, but it has an inherent ceiling: every stage adds latency and every conversion loses information. A newer architecture collapses the three models into one that goes from speech directly to speech, promising lower latency and richer understanding. Comparing the cascade with the emerging end-to-end approach.
You've seen what an AI gateway does; the last question is how to get one — build it, adopt an open-source proxy, or use a managed service — and how to run it once you have it. This closing post assembles the full architecture, weighs build-versus-buy honestly, and covers operating the gateway as the critical piece of infrastructure it becomes.
You've seen what an AI gateway does; the last question is how to get one — build it, adopt an open-source proxy, or use a managed service — and how to run it once you have it. This closing post assembles the full architecture, weighs build-versus-buy honestly, and covers operating the gateway as the critical infrastructure it becomes.
You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it: what was called, what it cost, how long it took, and whether it was allowed. This post is about turning the gateway into your AI system's source of truth and its governance point.
You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it and govern it: what was called, what it cost, how long it took, whether it was allowed. Turning the gateway into your AI system's source of truth.
Nothing concentrates the mind like a surprise five-figure AI bill from one runaway loop, or one team's traffic spike exhausting the rate limit everyone shares. Because every model call flows through the gateway, it's the one place you can enforce limits and budgets that actually hold — protecting your spend, your providers' rate limits, and fairness across teams. This post is about spending control as a first-class gateway capability.
Nothing concentrates the mind like a surprise five-figure AI bill from one runaway loop, or one team's spike exhausting the shared rate limit. Because every model call flows through the gateway, it's the one place you can enforce limits and budgets that actually hold — protecting your spend, your providers' rate limits, and fairness across teams. Spending control as a first-class capability.
The last stage of the pipeline is where the agent finally speaks, and it's where an interaction either sounds human or sounds like a robot reading a menu. Modern text-to-speech is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and it must pronounce the messy real world correctly. This post covers TTS for real-time voice.
The last stage of the pipeline is where the agent finally speaks — and where an interaction either sounds human or sounds like a robot reading a menu. Modern TTS is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and pronounce the messy real world correctly. TTS for real-time voice.
Model calls are slow and expensive, and a surprising fraction of them are repeats or near-repeats. Caching at the gateway turns those into instant, free responses — and because the gateway sees all traffic, it's the one place a cache benefits every application at once. Beyond exact-match caching, semantic caching catches queries that mean the same thing in different words, which is where the real savings live.
Model calls are slow and expensive, and a surprising fraction are repeats or near-repeats. Caching at the gateway turns those into instant, free responses — and because the gateway sees all traffic, it's the one place a cache benefits every app at once. Beyond exact-match caching, semantic caching catches queries that mean the same thing in different words, where the real savings live.
The language model is where a voice agent stops being a transcription toy and becomes something you can actually talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: it must respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming its answer so the user isn't left waiting. This post is about the LLM stage, adapted for voice.
The language model is where a voice agent stops being a transcription toy and becomes something you can talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming so the user isn't left waiting. The LLM stage, adapted for voice.
Model providers go down, rate-limit you, and time out — regularly. If your application calls one provider directly, its reliability is capped at that provider's. An AI gateway breaks that ceiling: because it can route across providers, a failure on one becomes a transparent retry on another. This post covers the reliability patterns that turn provider outages into non-events.
Model providers go down, rate-limit you, and time out — regularly. If your application calls one provider directly, its reliability is capped at that provider's. An AI gateway breaks that ceiling: because it can route across providers, a failure on one becomes a transparent retry on another. Retries, fallback, and circuit breakers — with an interactive request-flow sequence diagram.
Once every model call flows through one place, that place can make an intelligent decision on every request: which model should serve this, and through which of your capacity? Routing picks the right model for the task; load balancing spreads traffic across providers and keys so no single limit or outage bottlenecks you. Together they turn the gateway from a passthrough into a control plane.
Once every model call flows through one place, that place can make an intelligent decision on every request: which model should serve this, and through which of your capacity? Routing picks the right model for the task; load balancing spreads traffic across providers and keys so no single limit or outage bottlenecks you. Together they turn the gateway into a control plane.
Put the pieces together and a modern frontier language model comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, and which is engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.
Put the pieces together and a modern frontier LLM comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.
The first thing an AI gateway gives you is one interface to every model. Instead of your applications learning each provider's SDK, request format, and quirks, they speak a single API and the gateway translates. That translation layer is what decouples your code from any one vendor — and it's what makes model-swapping a config change instead of a rewrite.
The first thing an AI gateway gives you is one interface to every model. Instead of your applications learning each provider's SDK, request format, and quirks, they speak a single API and the gateway translates. That translation layer is what decouples your code from any one vendor — and makes model-swapping a config change instead of a rewrite.
The two costs of long context — quadratic attention compute and linear KV-cache memory — each have a family of solutions, and together they're why modern models can handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. This post covers the techniques that made long context practical.
The two costs of long context each have a family of solutions, and together they're why modern models handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. The techniques that made long context practical, and which cost each attacks.
The moment your application talks to more than one model — or one model but seriously — you accumulate a pile of cross-cutting concerns: provider APIs that differ, outages you must survive, costs you must control, calls you must log. An AI gateway is the single control point that handles all of it, sitting between your applications and every model provider. This series builds one from first principles.
The moment your application talks to more than one model — or one model but seriously — you accumulate cross-cutting concerns: differing provider APIs, outages, costs, logging. An AI gateway is the single control point that handles all of it, sitting between your applications and every model provider. This series builds one from first principles, with interactive architecture diagrams.
MoE scales a model's parameters cheaply. But there's a second scaling axis that matters just as much for modern LLMs: context length — how much text the model can attend to at once. Attention's cost grows with the square of the sequence, and the memory to run it grows linearly and relentlessly, which is why long context was hard and why so much architectural ingenuity has gone into it.
MoE scales a model's parameters cheaply, but there's a second axis that matters just as much: context length. Attention's compute grows with the square of the sequence, and the KV-cache memory grows linearly and relentlessly — two distinct costs, often confused, that make long context hard. Understanding both is the setup for the efficiency techniques that solved them.
A defense you haven't tested is a hope, not a control. The final discipline of LLM security is treating your guardrails as a system to be measured, attacked, and monitored continuously — because the threat evolves, your application changes, and a defense that worked last quarter can silently rot. This closing post covers red-teaming your own system, operating it in production, and the honest state of the art.
A defense you haven't tested is a hope, not a control. The final discipline of LLM security is treating your guardrails as a system to be measured, attacked, and monitored continuously — because the threat evolves, your application changes, and a defense that worked last quarter can silently rot. Red-teaming, production monitoring, and the honest state of the art.
Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. This post covers the trade MoE actually makes and the parallelism it forces.
Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. The trade MoE makes and the expert parallelism it forces.
"Guardrails" is the umbrella term for the runtime checks that sit around a model and screen what goes in and comes out — content classifiers, moderation models, topic and format validators, PII detectors. They're a real and useful layer, distinct from the architectural defenses, and they come with their own design rules: layer them, fail safe, and never mistake them for a wall.
"Guardrails" is the umbrella term for the runtime checks around a model — content classifiers, moderation models, topic and format validators, PII detectors. They're a real and useful layer, distinct from architectural defenses, with their own design rules: layer them, fail safe, and never mistake them for a wall. This post covers building the layered defense in practice.
The router is where Mixture of Experts succeeds or fails. Left to its own devices, it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE, and the techniques for it are what separate a working sparse model from a broken one.
The router is where Mixture of Experts succeeds or fails. Left alone it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE: load-balancing losses, expert capacity, and token dropping.
Injection defense usually focuses on what goes into the model. But an equally dangerous class of bug lives on the way out: whatever the model produces gets passed to another system — a browser, a shell, a database, another service — that trusts it. If the model can be made to emit a malicious payload, and your code renders or executes it, the injection escapes the model and lands in your infrastructure.
Injection defense usually focuses on what goes into the model, but an equally dangerous class of bug lives on the way out: whatever the model produces gets passed to another system that trusts it. If the model can be made to emit a malicious payload and your code renders or executes it, the injection escapes the model and lands in your infrastructure — XSS, SQLi, SSRF, exfiltration.
Now we open up the mechanism at the heart of modern LLMs. A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel "experts" and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling the first post promised, made concrete.
A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel experts and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling of capacity from per-token compute, made concrete. Experts, routing, and how outputs combine.
Everything before this post raised the probability barrier against injection. This post lowers the impact — and impact is what actually protects you. The load-bearing defense against prompt injection isn't a prompt or a filter; it's an architecture where a fully-hijacked model still can't do anything catastrophic, because it was never granted the power to.
Everything before this raised the probability barrier against injection; this post lowers the impact — and impact is what actually protects you. The load-bearing defense isn't a prompt or a filter but an architecture where a fully-hijacked model still can't do anything catastrophic: least privilege, the confused-deputy trap, human-in-the-loop, and the dual-LLM pattern.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter for the rest of the series: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.
Between filtering the input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack. None of them close the hole — because they are all still text in the one channel the attacker also writes to — but used well they meaningfully shift the odds.
Between filtering input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack — but none close the hole, because they're all still text in the one channel the attacker also writes to. The value is knowing exactly how much they buy.
The frontier language models of the last few years share a structural secret that isn't obvious from the outside: most of them aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while only using a fraction of them on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are actually built.
Frontier language models share a structural secret that isn't obvious from the outside: most aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while using only a fraction on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are built.
The first instinct when facing prompt injection is to inspect the input and block the bad stuff. It's a reasonable layer — but a treacherous one, because it creates a feeling of safety far larger than the protection it provides. This post covers the input-side defenses that are genuinely worth having, and draws a hard line around what they can and cannot do.
The first instinct against injection is to inspect the input and block the bad stuff. It's a reasonable layer but a treacherous one — it creates a feeling of safety far larger than the protection it provides. This post covers the input defenses genuinely worth having, and draws a hard line around what they can't do (starting with: indirect injection bypasses them entirely).
You can't defend against what you can't categorize. Prompt-based attacks come in a few structurally distinct shapes — direct injection, indirect injection through content the model reads, and jailbreaks that target the model's safety training — and each demands a different defense. This post maps the attack surface so the rest of the series can defend it systematically.
You can't defend what you can't categorize. Prompt-based attacks come in structurally distinct shapes — direct injection, indirect injection through content the model reads, and jailbreaks targeting the model's safety training — and each demands a different defense. This post maps the attack surface so the rest of the series can defend it systematically.
Prompt injection is the defining security problem of LLM applications, and — unlike SQL injection, which it superficially resembles — it has no clean fix. The reason is structural: a language model reads instructions and data through the same channel, and cannot reliably tell which is which. This opening post explains why that makes injection fundamentally hard, and reframes the goal from "prevent it" to "contain the blast radius."
Prompt injection is the defining security problem of LLM applications, and unlike SQL injection it has no clean fix — because a model reads instructions and data through the same channel and can't reliably tell them apart. This opening post explains why that makes injection fundamentally hard, and reframes the goal from prevent to contain the blast radius.
The gap between a LangChain demo and a LangChain production system is the same gap as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline the rest of your engineering already has. This closing post covers operating LangChain applications and the honest verdict on when to use the framework.
The gap between a LangChain demo and a production system is the same as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline your engineering already has.
A code agent is only as good as the model writing the code — and smolagents, true to its Hugging Face roots and minimalist ethos, lets you run almost any model behind it: open models from the Hub, hosted APIs, local models, anything reachable through its integrations. Model choice matters especially here, because the model's code-writing ability is the agent's ability.
A code agent is only as good as the model writing the code — and smolagents lets you run almost any model: open models from the Hub, hosted APIs, local models. Model choice matters especially here, because the model's code-writing ability is the agent's ability.
Most agent code is tested by running it and eyeballing the output — because testing agents properly is genuinely hard. Pydantic AI's whole design has been quietly building toward making it easy: type safety, dependency injection, and test models combine so you can unit-test agent logic deterministically, offline, without ever calling a real LLM. This is arguably the framework's biggest practical advantage.
Most agent code is tested by running it and eyeballing the output, because testing agents properly is hard. Pydantic AI's design builds toward making it easy: type safety, dependency injection, and test models let you unit-test agent logic deterministically, offline, with no real LLM.
Training a good fine-tune is the easy half. Running it in production — deciding it's even worth it, serving adapters efficiently, keeping it from going stale as base models leap ahead, and re-tuning as your needs shift — is where fine-tuning becomes an ongoing commitment rather than a one-time project. This is the reality check that closes the series.
Training a good fine-tune is the easy half. Running it in production — deciding it's worth it, serving adapters efficiently, keeping it from going stale as base models leap ahead, and re-tuning — is where fine-tuning becomes an ongoing commitment.
There is no single "make it fast" for LLM serving — latency and throughput pull against each other, and both trade against cost. The job isn't to maximize one number; it's to hit your latency targets at the lowest cost per token, which means knowing exactly which knob moves which metric and in which direction.
There is no single 'make it fast' for LLM serving — latency and throughput pull against each other, and both trade against cost. The job is to hit your latency targets at the lowest cost per token, knowing which knob moves which metric.
The vector-storage decision has a boringly practical answer that cuts against the hype: for most systems, the database you already run with a vector extension beats adding a new specialized system — until scale or specific features force the upgrade.
A boringly practical answer that cuts against the hype: for most systems the database you already run with a vector extension beats adding a specialized system — until scale or specific features force the upgrade.
Offline evaluation tells you whether a change is promising; production tells you whether it actually works. Once your system is serving real users, evaluation becomes continuous: online experiments, guardrail metrics, drift monitoring, and gating deploys on eval scores. This closing post moves evaluation from the lab into the running system and ties the whole series into a working loop.
Offline evaluation tells you whether a change is promising; production tells you whether it works. Once you're serving real users, evaluation becomes continuous — online experiments, guardrail metrics, drift monitoring, CI gating. This closing post moves evaluation from the lab into the running system and ties the series into one loop.
The model drives a Strands agent, so which model you use is the single biggest determinant of how well it works — and Strands keeps that a swappable choice across providers rather than locking you to one. Model-agnosticism isn't a convenience here; in a model-driven framework it's foundational.
The model drives a Strands agent, so which model you use is the biggest determinant of how well it works — and Strands keeps that a swappable choice across providers. Model-agnosticism isn't a convenience here; it's foundational.
A fine-tune that looks great on a handful of hand-picked prompts can be quietly broken — overfit to your training data, worse than the base model you started from, or catastrophically forgetful of skills it used to have. Without real evaluation you can't tell, and shipping a fine-tune you haven't measured is shipping a guess.
A fine-tune that looks great on a handful of prompts can be quietly broken — overfit, worse than the base model, or catastrophically forgetful. Without real evaluation you can't tell, and shipping a fine-tune you haven't measured is shipping a guess.
At some point a model doesn't fit on one GPU, or the traffic doesn't, and you have to spread inference across many. The choices — which kind of parallelism, how to place replicas, when to autoscale — are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication). Get the memory math right and most scaling decisions follow.
At some point a model doesn't fit on one GPU, or the traffic doesn't. The scaling choices are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication) — get the memory math right and most decisions follow.
This is the classic fixed-versus-marginal decision, and it has a clean answer: managed APIs win until your volume is high and steady enough to keep expensive GPUs busy — which is a much higher bar than most teams assume.
The classic fixed-vs-marginal decision with a clean answer: managed APIs win until your volume is high and steady enough to keep expensive GPUs busy — a much higher bar than most teams assume.
Automated metrics and LLM judges are scalable proxies for the thing that actually matters: whether a human finds the output good. Human evaluation is the ground truth those proxies are calibrated against — and getting it right is its own discipline, full of subtle traps around agreement, bias, and how you ask the question. This post is about doing human eval well, and about how human preference became the signal that trains models themselves.
Automated metrics and LLM judges are scalable proxies for the thing that matters: whether a human finds the output good. Human evaluation is the ground truth those proxies are calibrated against — a discipline full of traps around agreement, bias, and how you ask — and human preference is now the signal that trains models themselves.
The power of code agents comes with a sharp edge: you are executing code written by an LLM, and an LLM can be wrong, or manipulated into writing something harmful. Running that code unsandboxed is one of the most dangerous things you can do in an application, so sandboxing isn't optional for code agents — it's the price of admission.
The power of code agents comes with a sharp edge: you are executing code written by an LLM, which can be wrong or manipulated. Running it unsandboxed is one of the most dangerous things you can do — so sandboxing is the price of admission.
Two more levers of control: how hard the model thinks, and how you bound what it spends. Both have measured sweet spots — and both have a trap that quietly wastes money. The sharpest is a cap that most people reach for first and that saves nothing at all: max_tokens.
Two levers of control: how hard the model thinks, and how you bound what it spends. Both have measured sweet spots — and both have a trap that quietly wastes money. The sharpest is a cap most people reach for first that saves nothing: max_tokens.
Answering questions over your own data is the most common LLM application, and LangChain gives you the whole pipeline as composable, swappable components — loaders, splitters, embeddings, vector stores, retrievers — behind standard interfaces. The retriever, in particular, is just another Runnable, so RAG becomes a chain like any other.
Answering questions over your own data is the most common LLM application, and LangChain gives you the whole pipeline as composable, swappable components — loaders, splitters, embeddings, vector stores, retrievers. The retriever is just another Runnable.
Supervised fine-tuning teaches a model to produce a correct answer; alignment teaches it to produce the answer people actually prefer. That shift — from "right" to "better" — is what turned raw language models into helpful assistants, and the move from RLHF's complex machinery to DPO's direct approach made it something ordinary teams can do.
Supervised fine-tuning teaches a model to produce a correct answer; alignment teaches it to produce the answer people actually prefer. That shift turned raw models into helpful assistants, and the move from RLHF to DPO made it something ordinary teams can do.
You don't assemble the KV cache, continuous batching, quantization, and speculative decoding by hand — you use a serving engine that has already solved the hard parts. And the idea that ties them together, PagedAttention, is a borrowed operating-systems trick: manage the KV cache like virtual memory, in pages, and the waste that throttled everything disappears.
You don't assemble the KV cache, batching, and quantization by hand — you use a serving engine. And PagedAttention, the idea that ties them together, is a borrowed OS trick: manage the KV cache like virtual memory, in pages.
The most common question about the two big agent protocols is which one to use — and the answer is almost always "both," because they solve different problems: MCP connects an agent to its tools, A2A connects an agent to other agents.
The most common question about the two big agent protocols is which to use — and the answer is almost always both, because MCP connects an agent to its tools and A2A connects an agent to other agents.
You cannot manage what you cannot see, and the difference between a team that controls its AI spend and one that is surprised by it is almost always whether they measure cost per feature, per user, and per call.
You cannot manage what you cannot see. The difference between a team that controls its AI spend and one surprised by it is whether they measure cost per feature, per user, and per call.
Everything in this series — the budget, the system prompt, retrieval, memory, tools, and compaction — comes together as a pipeline that assembles the right window on every single turn, deliberately rather than by accident.
The budget, system prompt, retrieval, memory, tools, and compaction come together as a pipeline that assembles the right window on every turn — deliberately rather than by accident.
A benchmark score is trustworthy only if the model hasn't seen the answers and no one has optimized directly for the test. Both assumptions fail constantly. Training data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target and destroys its meaning. This post is about the ways evaluation gets corrupted — and how to defend against them.
A benchmark score is trustworthy only if the model hasn't seen the answers and no one optimized directly for the test — and both assumptions fail constantly. Training-data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target. Here's how evaluation gets corrupted, and how to defend it.
The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This post examines the evidence and the mechanism, so you understand not just that code actions win but why.
The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This examines the evidence and the mechanism.
Model selection is the most commonly botched cost decision, because the intuitive answer — pick the cheaper model — is measured by the wrong number. The right number is cost per completed task, and by that measure the more capable model often wins. Paired with it is the least-known lever of all: auditing prompts written for an older model against your current one.
Model selection is the most commonly botched cost decision, because the intuitive answer — pick the cheaper model — is measured by the wrong number. The right number is cost per completed task, priced on the tail not the median.
You can pick the perfect technique, tune every hyperparameter, and rent the best GPU — and still get a worse model than you started with, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones. This is the post that actually determines whether your fine-tune works.
You can pick the perfect technique and rent the best GPU, and still get a worse model, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones.
This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI. The plugin hides the runtime; you write an app.
This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI.
Decode is slow because it's sequential — one token at a time, each waiting for the last. Speculative decoding cheats that limit with a beautiful trick: let a small, fast model guess several tokens ahead, then let the big model verify them all in a single pass. When the guesses are good, you get several tokens for the price of one — with mathematically identical output.
Decode is slow because it's sequential. Speculative decoding cheats that with a beautiful trick: a small fast model guesses several tokens ahead, the big model verifies them in one pass — several tokens for the price of one, with identical output.
The most common architecture mistake in applied AI is reaching for fine-tuning to fix a knowledge problem — so the single most useful rule here is that RAG is for knowledge and fine-tuning is for behavior, and long-context is a convenience, not a strategy.
The most common architecture mistake is reaching for fine-tuning to fix a knowledge problem — so the key rule: RAG is for knowledge, fine-tuning is for behavior, and long-context is a convenience, not a strategy.
Some of the biggest cost decisions are architectural — whether to feed knowledge through retrieval or bake it into a model, whether to prompt a big model or fine-tune a small one, and whether to rent tokens or run your own — and each trade turns on your volume and your task.
Some of the biggest cost decisions are architectural — retrieval versus long context, prompting a big model versus fine-tuning a small one, and renting tokens versus self-hosting — each turning on your volume.
When context threatens to overflow, you compress it; and when you have a huge window to spend, you still should not fill it — because a long context is not used as well as a short, focused one.
When context threatens to overflow you compress it; and when you have a huge window you still should not fill it — because a long context is not used as well as a short, focused one.
The scores that dominate model announcements — MMLU, GSM8K, HumanEval, and the rest — are benchmarks: standardized public tests that let the whole field compare models on common ground. They've driven enormous progress, but a benchmark is only as good as its design, and a number without understanding of what it measures is easy to misread. Knowing how benchmarks are built, and what makes a good one, is how you read a leaderboard critically instead of credulously.
The scores in model announcements — MMLU, GSM8K, HumanEval — are benchmarks: standardized public tests for comparing models on common ground. They've driven huge progress, but a benchmark is only as good as its design, and a number without understanding is easy to misread. Here's how they're built and what makes a good one.
The second-largest cost lever is almost embarrassingly simple: if no one is waiting on a response, run it in a batch for half off — and the discount is identical across all three major clouds, which makes it a safe architectural assumption. Then comes token hygiene, where the wins are real but one popular technique quietly costs more than it saves.
The second-largest cost lever is almost embarrassingly simple: if no one is waiting on a response, run it in a batch for half off — identical across all three major clouds. Then comes token hygiene, where one popular technique quietly costs more than it saves.
This is the feature Pydantic AI is named for and built around: you declare a Pydantic model as your agent's output type, and you get back a validated instance of it — not a string to parse, not JSON to hope about, but a real typed object. It turns the single most brittle part of LLM applications into the most reliable.
This is the feature Pydantic AI is named for: you declare a Pydantic model as your agent's output type and get back a validated instance — not a string to parse. It turns the most brittle part of LLM applications into the most reliable.
LoRA shrank the trainable parameters, but you still had to hold the full base model in memory to train against it — and for a large model that alone needs serious hardware. QLoRA closes the gap: quantize the frozen base model to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss. It's what truly democratized fine-tuning.
LoRA shrank the trainable parameters, but you still had to hold the full base model in memory. QLoRA closes the gap: quantize the frozen base to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss.
Quantization shrinks a model by storing its numbers in fewer bits — and because LLM decode is bottlenecked on moving those numbers from memory, making them smaller makes inference both cheaper to host and faster to run. It's the rare optimization that improves memory, cost, and speed at once, if you respect its limits on quality.
Quantization stores a model's numbers in fewer bits — and because decode is bottlenecked on moving those numbers from memory, making them smaller makes inference cheaper to host and faster to run, if you respect the limits on quality.
The model platform decision is usually decided before you compare models at all — by which cloud you're already on, what governance you need, and whether you're renting inference or running it — and getting that framing right matters more than any benchmark.
The model-platform decision is usually settled before you compare models — by which cloud you're on, what governance you need, and whether you're renting inference or running it.
The technique that produced superhuman game-playing — a system improving by competing against copies of itself — has an LLM analogue: models that generate their own training signal and bootstrap their way up without new human labels.
The technique that produced superhuman game-playing has an LLM analogue: models that generate their own training signal and bootstrap up without new human labels — and the reward-source problem at its center.
Not all AI work needs an answer this second, and for the work that can wait, batching and asynchronous processing buy meaningfully cheaper tokens in exchange for latency you were not using anyway.
Not all AI work needs an answer this second, and for the work that can wait, batching and asynchronous processing buy meaningfully cheaper tokens in exchange for latency you were not using anyway.
Tool definitions and structured data quietly consume a large share of the context budget, and how you select, describe, and format them shapes both what fits and how well the model uses it.
Tool definitions and structured data quietly consume a large share of the context budget, and how you select, describe, and format them shapes both what fits and how well the model uses it.
A server is only half the story; the client is what connects to it, discovers its capabilities, and turns a model's intent into real tool calls.
Build an MCP client that launches a server, discovers its tools, and drives them from a language model — the core of what every MCP host does internally.
Metrics and judges are ingredients; a harness is the kitchen. An evaluation harness is the system that takes a dataset of test cases, runs your LLM system over them, scores the outputs, and reports the results — reproducibly, every time. Building one well is what turns evaluation from a one-off spreadsheet into an engineering asset you run on every change, like a test suite.
Metrics and judges are ingredients; a harness is the kitchen. An eval harness takes a dataset of test cases, runs your system over them, scores the outputs, and reports results — reproducibly, every time. Building one well turns evaluation from a one-off spreadsheet into an engineering asset you run on every change.
Of every AI cost lever, one dominates the rest, and all the major vendors agree on it: prompt caching. On long prompts and agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.
Of every AI cost lever, one dominates and all the major vendors agree on it: prompt caching. On agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.
Every LangChain application, no matter how elaborate, is built from three humble pieces: a model you call, a prompt you send it, and a parser that turns its reply into something usable. Master these three and the rest of LangChain is just composing them — which is exactly what the framework is designed to let you do.
Every LangChain application, no matter how elaborate, is built from three humble pieces: a model you call, a prompt you send it, and a parser that turns its reply into something usable. Master these three and the rest is just composing them.
Full fine-tuning updates every weight in a model — billions of numbers — which needs enormous memory and produces a full-size copy per task. LoRA sidesteps all of it with one insight: the change a model needs for a task is "low-rank," so you can train a tiny pair of matrices instead of the whole model. It's the technique that put fine-tuning within reach of anyone with a single GPU.
Full fine-tuning updates every weight — billions of numbers — needing enormous memory. LoRA sidesteps it with one insight: the change a model needs for a task is low-rank, so you train a tiny pair of matrices instead of the whole model.
On the cloud, quantization is an optimization you reach for to cut costs. On a phone, it's the difference between the model running and not running at all. Nearly every on-device LLM you'll ever ship is quantized, because full-precision weights simply don't fit — so understanding the bit-width trade-off is non-negotiable for edge AI.
On the cloud, quantization cuts costs. On a phone, it's the difference between the model running and not running at all — nearly every on-device LLM you'll ship is quantized, so the bit-width trade-off is non-negotiable for edge AI.
A single request leaves an expensive GPU almost entirely idle. Batching is how you fill it — and the leap from naive static batching to continuous batching is the single biggest throughput improvement in modern LLM serving, often several times more requests from the exact same hardware.
A single request leaves an expensive GPU almost entirely idle. Batching fills it — and the leap from static to continuous batching is the single biggest throughput improvement in modern LLM serving.
Four popular agent frameworks, four genuinely different philosophies — and the right choice is decided less by features than by how much control you want, how your team thinks, and what you're actually building.
Four popular agent frameworks, four genuinely different philosophies — the right choice is decided less by features than by how much control you want, how your team thinks, and what you're building.
A large share of what an AI system processes is repeated — the same system prompt, the same documents, the same questions — and caching is how you stop paying full price for work you have already done.
A large share of what an AI system processes is repeated — the same system prompt, the same documents, the same questions. Caching is how you stop paying full price for work you have already done.
A conversation that never forgets eventually overflows, so managing what history an agent carries forward — and how it remembers across sessions — is one of the defining problems of context engineering.
A conversation that never forgets eventually overflows, so managing what history an agent carries forward — and how it remembers across sessions — is a defining problem of context engineering.
The most ambitious form of self-evolution stops tweaking one agent and starts searching a space of many, letting a meta-process discover agent designs no human wrote.
The most ambitious self-evolution searches a population of agent designs. Automated Design of Agentic Systems, evolutionary prompt search, debate, and self-play.
When there's no reference answer and no rule to check, you can ask a strong model to grade the output — LLM-as-a-judge. It's the technique that made open-ended evaluation practical at scale, and it's also a minefield: judges have systematic biases, can be gamed, and agree with humans only when you design the grading carefully. Used well it's indispensable; used naively it produces confident numbers that mean nothing.
When there's no reference and no rule to check, you can ask a strong model to grade the output — LLM-as-a-judge. It made open-ended evaluation practical at scale, and it's a minefield: judges have systematic biases (position, verbosity, self-preference), can be gamed, and agree with humans only when you design the grading carefully.
LangChain is the framework everyone starts with and everyone has opinions about — a vast toolkit for building LLM applications whose real value isn't any one feature but the standardization it brings: one interface across every model, vector store, and tool, so you write your application once and swap the pieces underneath. Understanding what it is (and its relationship to LangGraph) cuts through the confusion.
LangChain is the framework everyone starts with and everyone has opinions about — a vast toolkit whose real value is the standardization it brings: one interface across every model, vector store, and tool, so you write your application once and swap the pieces underneath.
Most agent frameworks treat the LLM's output as text you hope to parse. Pydantic AI treats it as typed, validated data — bringing the discipline that made Pydantic the backbone of Python data validation to the messy world of LLM agents. If you've ever wished your agent's output was a real typed object instead of a string you cross your fingers over, this framework was built for you.
Most agent frameworks treat the LLM's output as text you hope to parse. Pydantic AI treats it as typed, validated data — bringing the discipline that made Pydantic the backbone of Python data validation to the messy world of LLM agents.
"Fine-tuning" is not one thing — it's a family of techniques that range from cheaply nudging a model's output format to expensively rebuilding its knowledge base. Confusing them leads to using a sledgehammer for a thumbtack. Knowing where your task sits on the spectrum tells you which technique, how much data, and how much compute you actually need.
'Fine-tuning' is not one thing — it's a family from cheaply nudging output format to expensively rebuilding a model's knowledge. Knowing where your task sits on the spectrum tells you which technique, how much data, and how much compute you actually need.
The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in LLM serving, from how many users you can batch to why long contexts cost so much, traces back to this one data structure.
The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in serving traces back to this one data structure.
Most AI architecture debates are settled by hype, familiarity, or whoever spoke last — this series settles them by requirements and trade-offs, starting with the meta-framework that every specific decision reduces to.
Most AI architecture debates are settled by hype or familiarity; this series settles them by requirements and trade-offs, starting with the meta-framework every specific decision reduces to.
The user's question is written to be asked, not to be searched, so the first thing an agentic RAG system should do is turn that question into queries that actually retrieve well.
The user's question is written to be asked, not searched, so the first thing an agentic RAG system should do is turn that question into queries that actually retrieve well.
Since you pay for every input token, the fastest way to cut the cost of a call without changing the model is to send fewer tokens — and most production prompts are carrying far more than they need.
Since you pay for every input token, the fastest way to cut the cost of a call without changing the model is to send fewer tokens — and most production prompts carry far more than they need.
Retrieval is how you put external, current knowledge into a model's window, and doing it well is a context-engineering problem — the goal is not the most documents, but the right ones.
Retrieval is how you put external, current knowledge into a model's window, and doing it well is a context-engineering problem — the goal is not the most documents, but the right ones.
Tools let a model act, but resources and prompts are how a Model Context Protocol server feeds it the right context and gives users repeatable ways to invoke it.
Beyond tools, MCP servers expose resources (read-only context by URI) and prompts (reusable templates). What each is for, how they appear on the wire, and how to choose.
Before you can score an LLM, you have to decide what "good" even means for your task — and that choice determines everything downstream. Metrics fall into a few families, from exact string matching to reference overlap to semantic similarity to task-specific checks, each measuring something different and each with blind spots. Picking the wrong metric is worse than no metric: it gives you a confident number that points the wrong way.
Before you can score an LLM you must decide what "good" means — and that choice determines everything. Metrics fall into families (exact match, reference overlap, semantic similarity, task-specific), each measuring something different with different blind spots. Picking the wrong metric is worse than none: it points confidently the wrong way.
Fine-tuning is the most misunderstood tool in the LLM toolkit. Reach for it to teach a model new facts and you'll waste weeks and get worse results than a day of RAG. Reach for it to change how a model behaves — its format, tone, or a narrow skill — and nothing else comes close. The whole discipline starts with knowing which problem you actually have.
Fine-tuning is the most misunderstood tool in the LLM toolkit. Reach for it to teach a model facts and you'll waste weeks; reach for it to change how a model behaves — format, tone, a narrow skill — and nothing else comes close. The discipline starts with knowing which problem you have.
Running an LLM is not one computation — it's two very different ones stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization that follows makes sense only once you see that inference has these two phases with opposite bottlenecks.
Running an LLM is two very different computations stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization makes sense only once you see these two phases.
LlamaIndex began as the fastest way to build RAG and has grown into a full data framework for LLM applications — connect your data, index it, retrieve it, and reason over it, with agents and workflows on top.
LlamaIndex began as the fastest way to build RAG and has grown into a full data framework for LLM applications — connect your data, index it, retrieve it, and reason over it, with agents and workflows on top.
The single biggest cost lever in most AI systems is not clever prompting — it is not using an expensive model for work a cheap one would do just as well.
The single biggest cost lever in most AI systems is not clever prompting — it is not using an expensive model for work a cheap one would do just as well. Right-sizing and routing models to tasks.
The system prompt is the one piece of context present on every single call, so how you structure its role, rules, and format is the highest-leverage writing in the whole system.
The system prompt is the one piece of context present on every call, so how you structure its role, rules, and format is the highest-leverage writing in the whole system.
The series finale — where LLM reviewers genuinely help on every pull request, where they quietly fail, and how to build a human-plus-AI workflow that speeds review up without letting judgment or accountability leak away.
The capstone: using AI/LLM reviewers well and where humans stay essential — AI as a tireless first pass that clears the noise, its limits (design judgment, contextual security, accountability, hallucinations), and a healthy human+AI division of labor.
Building something with an LLM is easy for a weekend and hard for a year. The wall almost everyone hits is not the model, the prompt, or the framework — it is knowing whether a change made things better or worse. Without a way to measure quality, every improvement is a guess and every deploy is a gamble. Evaluation is the discipline that turns "it seems better" into "it is better, by this much," and it is the real bottleneck in shipping AI systems.
Building with an LLM is easy for a weekend and hard for a year, and the wall almost everyone hits is knowing whether a change made things better or worse. Without measurement every improvement is a guess — evaluation is the discipline that turns "it seems better" into "it is better, by this much," and it's the real bottleneck in shipping AI.
The retrieve-then-generate pipeline that launched a thousand demos hits a wall on real questions, and understanding exactly where it breaks is the case for making retrieval agentic.
The retrieve-then-generate pipeline that launched a thousand demos hits a wall on real questions. Understanding exactly where naive RAG breaks is the case for making retrieval agentic.
You cannot optimize what you cannot price, and pricing an AI system starts with understanding the token — what counts as one, why input and output cost differently, and how to compute the true cost of a request.
You cannot optimize what you cannot price. Pricing an AI system starts with the token — what counts as one, why input and output cost differently, and how to compute the true cost of a request.
Every token in the window costs money, adds latency, and competes for the model's attention, so the first skill of context engineering is treating context as a scarce budget to be spent deliberately.
Every token in the window costs money, adds latency, and competes for attention. The first skill of context engineering is treating context as a scarce budget to spend deliberately.
Turning red-team attacks into metrics you can act on and track over time — attack success rate, coverage, severity, and trend — plus the honest limits of what any of those numbers can tell you.
Turning attacks into metrics: attack success rate and why it's subtle, scoring success (rule/classifier/LLM-judge with its biases), coverage across the taxonomy, severity weighting, tracking trends per model/prompt version, and honest reporting of residual risk.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
An LLM demo costs almost nothing, so teams ship without a cost model — and then production traffic turns a rounding error into the biggest line item on the bill.
An LLM demo costs almost nothing, so teams ship without a cost model — then production traffic turns a rounding error into the biggest line item on the bill. Why AI costs explode, and how to see it coming.
Prompt engineering was about wording a single instruction well; context engineering is the broader discipline of deciding everything a model sees at inference time — and for agents, it is the discipline that matters most.
Prompt engineering was about wording one instruction well; context engineering is the broader discipline of deciding everything a model sees at inference — and for agents it is the skill that matters most.
Scaling red-teaming beyond manual probing — the building blocks of an automated harness (seed library, mutation, orchestrator, scorer), LLM-driven adaptive attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and wiring it all into CI as a repeatable gate.
Scaling red-teaming: the harness building blocks (attack seeds, mutation, orchestrator, scorer), adaptive LLM-driven attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and integrating an automated red-team gate into CI.
When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.
Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.
A model is only as useful as the context and tools it can reach, and MCP is the open standard that lets any AI app plug into any tool through one interface instead of a hundred bespoke integrations.
MCP turns the M×N mess of wiring every AI app to every tool into M+N: wrap each system as a server once, make each app a client once, and any app can use any system.
Most agents are frozen the moment they ship, repeating the same mistakes forever, and self-evolving agents are the attempt to break that ceiling by letting the system improve itself as it runs.
Most agents are frozen at deployment and repeat their mistakes forever. Self-evolving agents route their own experience back into their own behavior — here are the axes of change and the loop underneath them.
Governance doesn't stop at deploy. This is the NIST RMF MANAGE function in practice: what to monitor for an LLM system, how to detect the drift — including the silent kind where a provider swaps the model under you — and why the audit trail you log is the regulatory deliverable, not a debugging convenience.
Governance doesn't stop at deploy — the Manage function in production: what to monitor (operational, quality, safety), detecting drift including silent provider-model drift behind moving aliases, and the audit trail (versions + retrieved context, PII-redacted) that becomes regulatory evidence.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
Reasoning models are a genuine advance — and it's precisely because they're so impressive that their limits are worth stating plainly. A model that writes out careful, step-by-step reasoning invites you to trust the reasoning, and that trust is not always warranted. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve. Knowing the edges is part of using the capability wisely.
Reasoning models are a genuine advance — and precisely because they're so impressive, their limits are worth stating plainly. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve.
The three trustworthy-AI properties regulators and users press on hardest — where bias enters a system, why the fairness definitions contradict each other so you must choose one deliberately, and why an explanation you can read is not the same as an explanation you can trust.
The trustworthy-AI properties regulators care about: where bias enters, why fairness notions conflict (you must choose one), disaggregated evaluation, mitigation with Fairlearn/AIF360, and explainability (SHAP/LIME) — with the honest caveat that LLM rationales are not faithful explanations.
Two tightly-linked OWASP LLM risks that turn a clever prompt injection into real-world damage — and the Python patterns that shrink the blast radius: treat model output as untrusted input, and give agents the least agency they can get away with.
Two OWASP risks that turn an injection into damage: insecure output handling (model output is untrusted input — never eval/shell/SQL it unescaped) and excessive agency (least-privilege tools, allow-lists, human approval for irreversible actions, audit logs).
Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.
Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.
The habits that made you good at prompting standard models can actively hurt you with reasoning models. "Let's think step by step" is redundant — even counterproductive — when the model already thinks natively. Few-shot examples can constrain reasoning that's better left free. The shift from standard to reasoning models isn't just picking a different model; it's unlearning some prompting reflexes and learning where deep thinking actually earns its cost.
The habits that made you good at prompting standard models can actively hurt you with reasoning models. 'Let's think step by step' is redundant when the model already thinks natively. The shift isn't just picking a different model — it's unlearning some prompting reflexes and learning where deep thinking earns its cost.
Governance is only as real as your ability to measure it. This is the MEASURE function of an AI risk program made concrete — a versioned eval set, the metric families that matter for an LLM system, and a CI gate that fails the build when quality regresses instead of just logging a warning.
Governance enforced through evaluation — the Measure function made real: build a versioned eval set, pick the metric families (quality, faithfulness, safety, bias, PII, cost/latency), and turn eval into a CI quality gate that fails the build on regression.
Part three of the AI Security Engineering series: protecting the data that flows through an LLM system — how sensitive information leaks out of prompts, logs, and retrieval, and the engineering controls (redaction, data minimization, per-user retrieval authz, residency choices) that actually stop it.
Protecting data in LLM systems: sensitive-information disclosure, PII in prompts and logs (your observability can be the leak), redaction with Presidio, data residency, and per-user access control on the retrieval layer so RAG doesn't leak across tenants.
How to know whether an LLM system actually works — building an eval dataset, the four metric families (deterministic checks, text overlap, embedding similarity, LLM-as-judge) in Go, task-specific eval for RAG and classification, and wiring a scored regression gate into CI so you measure instead of vibe.
How to know whether an LLM system works when outputs are non-deterministic: build an eval dataset, score with deterministic checks, embedding similarity, and LLM-as-judge (with its biases), evaluate RAG and classification, and gate regressions in CI.
Test-time compute reframes a question engineers rarely had to ask before: how much is a correct answer worth? Because thinking now costs money and time in direct proportion to how much of it you do, reasoning is no longer free — it's a purchase. A reasoning model can generate many times more tokens working through a problem than a standard model uses to answer it, and you pay for every one. Deciding when that's worth it is the core practical skill of the reasoning era.
Test-time compute reframes a question engineers rarely had to ask: how much is a correct answer worth? Thinking now costs money and time in direct proportion to how much of it you do. A reasoning model can generate many times more tokens than a standard model — and you pay for every one.
Why the #1 risk on the OWASP LLM Top 10 has no clean fix — the model can't tell your instructions apart from the text it reads — and the defense-in-depth pattern that actually shrinks the blast radius.
The #1 LLM risk: direct vs indirect prompt injection (the latter is the real threat for RAG and agents), jailbreak families, and honestly-rated defense-in-depth — delimiting untrusted data, least-privilege tools, human-in-the-loop, and detection's limits.
A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model, reasoning or not, to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps, pruning the bad branches. These techniques share one shape — do more work at inference, then choose well — and understanding them turns test-time compute from a model property into a toolkit you control.
A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps. These techniques share one shape: do more work, then choose well.
The opening post of a hands-on series for builders: what it means to systematically stress-test an AI system — model, prompts, retrieval, tools, and guardrails — to surface its failures before adversaries or ordinary users do, how that differs from pentesting and robustness testing, and the frameworks and ethics that keep the work rigorous.
The opener to a defensive AI red-teaming series: stress-testing AI systems (models + prompts + RAG + tools + guardrails) to find failures before adversaries do — how it differs from pentesting, the frameworks (OWASP GenAI, MITRE ATLAS, NIST AI 100-2), and ethical rules of engagement.
Turning the four functions of the NIST AI Risk Management Framework — GOVERN, MAP, MEASURE, MANAGE — into something an engineering team can actually run: a risk taxonomy for LLM apps, a versioned risk register in code, and the eval hooks that keep MEASURE honest.
Make the NIST AI Risk Management Framework actionable: the Govern/Map/Measure/Manage functions in your own words, a risk taxonomy for LLM apps, and a versioned risk register (YAML/Python) capturing per-use-case risks, likelihood/impact, mitigations, and owners.
Why LLM and agent applications open a genuinely new attack surface, the mental models to reason about it (OWASP Top 10 for LLM Applications, MITRE ATLAS, NIST AI RMF), and how to threat-model an AI system before you write a line of defensive code.
The opener to a series on securing LLM and agent systems: why AI apps have a new attack surface (instructions and data share one channel, tools grant real power), the OWASP Top 10 for LLM Applications, MITRE ATLAS, and how to threat-model an AI system.
Your first real inference calls with the ibm-watsonx-ai SDK — credentials, ModelInference, the generate and chat APIs, streaming, token usage, error handling, and the langchain-ibm path.
Make your first watsonx.ai calls in Python: credentials (IAM key + project_id + regional url), ModelInference with generate_text vs the modern chat API, decoding methods and parameters, streaming, token usage, and the langchain-ibm ChatWatsonx path.
Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.
Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.
How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.
Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating Agents in Go.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating...
You can't teach deep reasoning by showing a model more examples of good reasoning — because the best reasoning for a hard problem often isn't in any dataset, and imitation caps a model at the quality of what it imitates. The breakthrough behind modern reasoning models was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers. That shift — from imitation to reinforcement on verifiable outcomes — is why reasoning models can think in ways no one wrote down.
You can't teach deep reasoning by showing more examples — imitation caps a model at what it imitates. The breakthrough was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers.
A working definition of AI governance for engineers — what it means, how it differs from security and compliance, why you already own a slice of it, and the frameworks and lifecycle map that anchor the rest of this series.
The opener to a governance series for engineers: what AI governance is and how it differs from security and compliance, why engineers produce the evidence (docs, evals, monitors, audit trails), and the frameworks used throughout — NIST AI RMF, the EU AI Act, and ISO/IEC 42001.
A Python engineer's map of IBM watsonx — what watsonx.ai, watsonx.governance, watsonx.data and watsonx Orchestrate actually are, why enterprises pick them, and the smallest amount of `ibm-watsonx-ai` code that gets a foundation model answering you.
The opener to a series on building LLM and agent applications on IBM watsonx from Python: how watsonx.ai (Granite + third-party models), watsonx.governance, watsonx.data and Orchestrate fit together, and why the ibm-watsonx-ai SDK and langchain-ibm make it Python-native.
A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.
The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.
Your first real inference call in Go against Amazon Bedrock — using the unified, model-agnostic Converse API and the AWS SDK for Go v2, from client construction to reading tokens back off the response.
Your first real inference call on Bedrock in Go via the unified Converse API: building the client, the ConverseInput message/content-block union, extracting the assistant text, and reading stop reason and token usage — with the content-block union explained.
The dominant story of AI progress for years was training-time scale: bigger models, more data, more training compute. Test-time compute is a second, independent axis — spend more computation when you run the model, not when you train it, and get better answers on hard problems. It reframes a trained model not as a fixed-capability artifact but as one whose performance you can dial up per query by letting it think more.
The dominant story of AI progress was training-time scale: bigger models, more data. Test-time compute is a second, independent axis — spend more computation when you run the model, not when you train it, and get better answers on hard problems. It reframes a model as one whose performance you can dial up per query.
The opener for a Go series on building LLM and agent applications with Amazon Bedrock — what the service actually is, why it sits between your Go code and a dozen foundation models, and which aws-sdk-go-v2 packages you will lean on for the rest of the way.
The opener to a series on building LLM and agent applications on Amazon Bedrock in Go: what Bedrock actually is, what it adds over calling a provider API directly (one API across models, IAM auth, data residency), and the aws-sdk-go-v2 packages you'll use.
The observation that launched reasoning models was almost embarrassingly simple: if you ask a language model to "think step by step" before answering, it gets dramatically better at hard problems — with no change to the model at all. That a few words of prompting could unlock reasoning already latent in the model was a clue about something deep: the ability to reason was there, waiting to be elicited, and giving the model room to work was the key.
The observation that launched reasoning models was almost embarrassingly simple: ask a model to 'think step by step' and it gets dramatically better at hard problems, with no change to the model. That a few words could unlock latent reasoning was a clue about something deep.
For years, the way to make a language model better was to make it bigger — more parameters, more training data. Reasoning models represent a different bet: instead of only scaling what the model knows, scale how much it thinks before answering. A reasoning model spends extra computation at inference time working through a problem step by step before committing to an answer — and on hard problems, that changes what's possible.
For years, making a language model better meant making it bigger. Reasoning models represent a different bet: scale how much the model thinks before answering, not just what it knows. A reasoning model spends extra computation working through a problem step by step — and on hard problems, that changes what's possible.
Treating the prompt as a real engineering artifact — grounded in how a next-token predictor actually works — with roles, specificity, few-shot examples, decomposition, chain-of-thought, grounding, temperature, injection defense, and versioned Go templates you can test.
Prompt engineering as a real engineering discipline: roles, specificity, few-shot, decomposition, chain-of-thought and its cost, grounding, and prompt injection — plus building prompts as versioned, testable Go text/template templates you treat like code.
From-scratch Go for the two mechanisms that turn an LLM from a text generator into a component you can wire into real software — schema-constrained JSON and function calling — both spoken over the same OpenAI-compatible chat JSON.
Getting reliable machine-readable output from an LLM in Go: structured output (json-schema mode, decode into a typed struct, validate with a bounded retry) and tool/function calling (the full round-trip loop, decoding tool arguments, returning results tied to the call id).
A hybrid allocator where a quantitative core owns the money, a language model only whispers tilts, and hard constraints plus a human bound everything before a single order goes out.
A hybrid allocator: quantitative signals set the baseline (mean-variance / risk parity) while an LLM proposes tilts from qualitative context, constrained by risk limits and human review.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.
Turning a firehose of unstructured headlines into a disciplined, point-in-time trading signal.
Turning unstructured news into a tradable signal with LLMs: entity/event extraction, sentiment scoring, aggregation to a per-asset signal, latency and point-in-time constraints, and combining qualitative signals with a q…
The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.
The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.
The working mental model an AI engineer needs — next-token prediction, attention, training, and sampling — without the transformer math, and with every fact tied back to a decision you make in code.
The working mental model an AI engineer needs — next-token prediction, attention at an intuition level, pretraining vs post-training, and sampling — with every fact tied back to a concrete decision you make in code.
The opener to a from-scratch series on building applications on top of foundation models in Go — what AI engineering actually is, how it differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
The opener to a from-scratch AI-engineering-in-Go series: what AI engineering actually is, how building on foundation models differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
Large language models earn their place in regulated finance as fast, well-supervised assistants — not as autonomous agents with a hand on the money.
Where LLMs fit in regulated finance: document analysis/extraction, retrieval-grounded Q&A, and agentic decision support — plus the guardrails (grounding, human-in-the-loop, output validation, audit trails) that keep them…
How one agent routes work to specialists — and why the description field is the most important string you write.
Agent hierarchies and LLM-driven delegation: sub_agents, how the description field drives auto-transfer, and coordinator/dispatcher patterns — contrasted with deterministic workflow agents.
description, instruction, generation params, and structured output — the dials on almost every agent you'll build
The four knobs on almost every ADK agent: description (for delegation), instruction with {state} templating, generation params, and structured output — Python Pydantic model vs Go genai.Schema.
The multimodal capability people actually experience — uploading a photo and asking a chatbot what's in it, or having it read a screenshot, explain a diagram, or extract data from a chart — comes from vision-language models: LLMs that can see. The clever part is how it's done. Rather than build a seeing-and-reasoning model from scratch, you take a language model that already reasons brilliantly and give it eyes, by connecting a vision encoder to it. Understanding how that connection works explains the multimodal AI most people use.
The multimodal capability people actually experience — uploading a photo and asking a chatbot what's in it — comes from vision-language models: LLMs that can see. The clever part is how it's done: take a language model that already reasons brilliantly and give it eyes by connecting a vision encoder to it.
Agents return a clean 200 OK and still be wrong, unsafe, or expensive. Why agentic AI needs a new observability layer — LLM-as-judge, safety metrics, and the four lifecycle stages.
Single-turn evals check one decision. Multi-turn evals check the whole trajectory. A Python harness with three evaluators, an offline test suite, and the judge prompt that actually works.
The demo is easy; the production agent is hard. An agent that works impressively in a demo can fail unpredictably in the real world — looping, hallucinating, taking wrong actions, or racking up huge costs — because the same flexibility that makes agents powerful makes them unreliable. Building agents that actually work in production is a discipline of managing that unreliability: adding guardrails, evaluating rigorously, and, most importantly, knowing when not to use an agent at all. This closing post is about that discipline.
The demo is easy; the production agent is hard. The same flexibility that makes agents powerful makes them unreliable — looping, hallucinating, taking wrong actions, racking up costs. Building agents that actually work is a discipline of managing that unreliability, and knowing when NOT to use an agent at all.
The instinct, once single agents work, is to build teams of them — a researcher agent, a writer agent, a critic agent, all collaborating like a little organization. It's an appealing vision, and sometimes exactly right. But multi-agent systems are also where a lot of complexity and cost hides, and the honest guidance is more restrained than the hype: use multiple agents when the problem genuinely calls for it, and prefer a single well-designed agent when it doesn't. Understanding the multi-agent patterns — and their real tradeoffs — is what lets you make that call well.
The instinct, once single agents work, is to build teams of them — a researcher, a writer, a critic, collaborating like an organization. Sometimes that's right. But multi-agent systems are also where a lot of complexity and cost hides, and the honest guidance is restrained: use multiple agents when the problem genuinely calls for it, and prefer a single well-designed agent when it doesn't.
The first output is rarely the best output — a truth as old as writing, and one that applies to agents too. An agent that acts once and moves on repeats its mistakes; an agent that looks back at what it did, judges whether it worked, and tries again can dramatically improve. Reflection — the agent evaluating and correcting its own work — is what turns a one-shot attempt into an iterative process that gets better, and it's one of the most powerful patterns for making agents reliable on hard tasks.
The first output is rarely the best output. An agent that acts once and moves on repeats its mistakes; an agent that looks back at what it did, judges whether it worked, and tries again can dramatically improve. Reflection turns a one-shot attempt into an iterative process that gets better.
An LLM is, fundamentally, stateless — it remembers nothing between calls except what you put in its context window. For an agent that takes many steps or works across many sessions, that's a serious problem: without memory, every step starts from scratch, and nothing is ever learned. Memory is how agents overcome statelessness — holding the context of the current task, and carrying knowledge across tasks and time. Understanding the kinds of agent memory, and their limits, is essential to building agents that can handle real, extended work.
An LLM is fundamentally stateless — it remembers nothing between calls except what you put in its context. For an agent that takes many steps or works across sessions, that's a serious problem. Memory is how agents overcome statelessness — holding the current task's context, and carrying knowledge across tasks and time.
Ask an agent to "research this market and write a report" and it faces the same problem a person would: the task is too big to do in one leap. The answer, for agents as for people, is to break it down — decompose the goal into steps, and work through them. Planning is how agents handle complexity that the basic reason-act loop alone would fumble, and the patterns for doing it — from planning upfront to decomposing on the fly — are among the most important in agent design.
Ask an agent to 'research this market and write a report' and it faces the same problem a person would: the task is too big to do in one leap. The answer, for agents as for people, is to break it down. Planning is how agents handle complexity that the basic reason-act loop alone would fumble.
PROVIDER=ollama, granite4.1:3b, zero API keys, no Azure account. How to make a multi-agent project that demonstrates enterprise patterns without requiring enterprise infrastructure.
An LLM on its own can only do one thing: generate text. It can't search the web, run code, query a database, check the current time, or send a message — it can only produce words. Tools are what break that confinement, turning a model that can only talk into an agent that can act. Tool use is arguably the single most important capability that makes agents possible, and understanding how it works — and how to design tools well — is central to building effective agents.
An LLM on its own can only do one thing: generate text. Tools are what break that confinement, turning a model that can only talk into an agent that can act. Tool use is arguably the single most important capability that makes agents possible.
Strip away the frameworks, the tooling, and the jargon, and every LLM agent reduces to one simple loop: think about what to do, do it, look at what happened, repeat. This reason-act-observe cycle — crystallized by the ReAct pattern — is the beating heart of every agent, and understanding it deeply is understanding agents themselves. Once you see the loop clearly, agent frameworks stop being mysterious: they're all just implementations of this same fundamental cycle.
Strip away the frameworks and every LLM agent reduces to one simple loop: think about what to do, do it, look at what happened, repeat. This reason-act-observe cycle — crystallized by the ReAct pattern — is the beating heart of every agent, and understanding it deeply is understanding agents themselves.
"Agent" has become one of the most overused and least precise words in AI — applied to everything from a chatbot with a system prompt to a fully autonomous system that writes and ships code. Cutting through the hype requires a clear definition: an agent is a system where an LLM decides its own actions in a loop, using tools, until a goal is met. That one distinction — the model choosing what to do next, rather than following a fixed script — is what separates a genuine agent from a workflow, and it's where both the power and the difficulty come from.
'Agent' has become one of the most overused words in AI. Cutting through the hype requires a clear definition: an agent is a system where an LLM decides its own actions in a loop, using tools, until a goal is met. That one distinction — the model choosing what to do next, rather than following a fixed script — is where both the power and the difficulty come from.
How to port ADK's model hard-codes to Microsoft Agent Framework's provider factory pattern.
Zero-code LLM provider swaps across environments: Ollama for dev, OpenAI for staging, Azure Foundry for prod. Same agents, different models.
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Every LLM-backed agent in this platform has a deterministic rule-based fallback. The case always finalises. The fallback isn't a workaround — it's the contract.
PAN check-digit validation, Aadhaar offline KYC, DigiLocker, PEP/sanctions — all in Go code, not in a prompt. The LLM's job is to translate the verdict into something a human can read.
Not every query needs the production agent. A cost-aware dispatcher decides whether to route to the cheap-and-fast agent or the expensive-and-thorough one. Same UX, dramatically lower bill.