#LLMs

Articles about LLMs — exploring patterns, best practices, and real-world implementations in production systems.

197 posts tagged with llms. ← Blog home

#A/B Testing (3)#A2A (14)#ADK (8)#AG-UI (6)#AI (9)#AI Adoption (9)#AI Agents (355)#AI Architecture (22)#AI Cost (10)#AI Cost Optimization (8)#AI Engineering (316)#AI Evaluation (9)#AI Financial Engineering (14)#AI Gateway (8)#AI Governance (43)#AI Red Teaming (9)#AI Research (17)#AI Safety (12)#AI Security (35)#AI Strategy (8)#AI in Production (12)#AML (3)#API Design (17)#API Security (9)#APIs (69)#AWS (17)#Accountability (8)#Accounting (9)#Agent Memory (8)#Agent Skills (11)#Agentic AI (25)#Agentic Commerce (12)#Agentic Engineering (12)#Agentic RAG (8)#Agents (4)#Amazon Bedrock (8)#Analytics (3)#Architecture (45)#Attention (3)#Audit (4)#Authentication (11)#Authorization (3)#Automation (13)#Azure (11)#Azure AI Foundry (9)#Backend Engineering (322)#Benchmarks (3)#Best Practices (5)#BigQuery (6)#Billing (8)#Build Systems (4)#Business Finance (8)#Business Strategy (55)#C (8)#CI/CD (23)#Caching (11)#Capital Markets (14)#Card Payments (12)#Cards (13)#Career (30)#Causal Inference (8)#Cellular Automata (8)#Change Management (3)#Checkpointing (4)#Claude Code (8)#Cloud (5)#Cloud Architecture (3)#Cloud Native (10)#Code Review (9)#Coding Agents (20)#Collaboration (6)#Communication (9)#Complexity (8)#Compliance (54)#Computer Networking (9)#Computer Science (40)#Computer Vision (15)#Computer-Use Agents (8)#Concurrency (42)#Consulting (26)#Containers (10)#Context Engineering (19)#Conversational AI (8)#Cost Optimisation (5)#Cost Optimization (3)#Credit (16)#Credit Risk (16)#CrewAI (8)#Crypto (12)#Cryptocurrency (12)#Cryptography (9)#Custody (9)#DSPy (8)#Data (14)#Data Engineering (25)#Data Governance (3)#Data Science (8)#Data Structures (10)#Databases (48)#Deep Learning (20)#Deployment (5)#Design (3)#Design Patterns (10)#DevOps (24)#DevSecOps (22)#Developer Experience (7)#Developer Productivity (15)#Developer Tools (5)#Differential Privacy (3)#Diffusion Models (10)#Discovery (5)#Distributed Systems (104)#Documentation (3)#Edge AI (8)#Embeddings (21)#Emotional Intelligence (8)#Energy (8)#Engineering (14)#Engineering Culture (3)#Engineering Judgment (7)#Engineering Practices (21)#Enterprise AI (8)#Error Handling (4)#Evaluation (79)#Event-Driven (9)#Event-Driven Architecture (8)#Explainability (3)#Explainable AI (8)#FREE-AI (8)#FX (5)#Feature Engineering (3)#Feature Flags (3)#Feedback (4)#FinOps (23)#FinTech (6)#Financial AI (14)#Financial Engineering (8)#Financial Systems (133)#Fine-Tuning (12)#Fintech (134)#Flutter (8)#Forecasting (10)#Foreign Exchange (5)#Forward Deployed Engineer (43)#Forward Deployed Engineering (8)#Fraud (10)#Fraud Detection (8)#Function Tools (5)#Functional Programming (3)#Fundamentals (23)#Fundraising (8)#GCP (5)#GPU (8)#GUI Automation (8)#Gemma (4)#Generative AI (13)#Git (8)#Go (232)#Go-to-Market (31)#Google ADK (36)#Governance (61)#Granite (6)#Graph Neural Networks (8)#GraphQL (3)#Growth (3)#Guardrails (35)#HIPAA (3)#HTTP (3)#Harness Engineering (8)#Hiring (9)#Hugging Face (8)#Human-in-the-Loop (10)#IBM watsonx (8)#Idempotency (3)#Identity (11)#Image Generation (10)#Inference (8)#Information Retrieval (8)#Integration (3)#Integrations (8)#Intellectual Property (8)#Interfaces (3)#Interpretability (8)#JavaScript (8)#KYC (11)#KYC and AML (12)#Kafka (11)#Knowledge Graphs (11)#Kubernetes (17)#LLM (5)#LLM Inference (8)#LLM Infrastructure (8)#LLM-as-Judge (3)#LLMOps (9)#LLMs (197)#LangChain (8)#LangGraph (11)#Leadership (33)#Ledger (12)#Legal (8)#Lending (14)#Linux (9)#LlamaIndex (8)#Load Balancing (3)#MCP (25)#MLOps (39)#Machine Learning (119)#Marketing (16)#Markets (4)#Mathematical Finance (8)#Memory (16)#Memory Management (5)#Metrics (7)#Microservices (3)#Microsoft Agent Framework (150)#Middleware (6)#Migration (9)#Migrations (8)#Mixture of Experts (5)#Model Risk (3)#Monetization (9)#Monitoring (5)#Monorepo (3)#Multi-Agent (10)#Multi-Agent AI (14)#Multi-Agent Systems (76)#Multimodal (3)#Multimodal AI (11)#NIM (5)#NVIDIA (8)#Networking (3)#OAuth (3)#OWASP (7)#Observability (53)#On-Device AI (8)#Ontology (8)#Open Source (7)#OpenAPI (3)#OpenTelemetry (5)#Operating Systems (9)#Operations (14)#Opinion (6)#Orchestration (10)#Organizational Design (8)#Payment Rails (16)#Payments (55)#People (8)#Performance (56)#Personalization (9)#Platform Engineering (17)#Policy as Code (3)#PreSales (8)#Pricing (4)#Privacy (14)#Privacy Engineering (3)#Process (4)#Product (30)#Product Management (9)#Production (22)#Programming (10)#Programming Languages (48)#Prompt Engineering (86)#Prompt Injection (16)#Proof of Concept (3)#Protocol Buffers (3)#Protocols (9)#Providers (4)#Pydantic AI (8)#Python (142)#Quality (3)#Quantitative Finance (22)#RAG (72)#RBI (3)#REST (5)#Rails (16)#Real-Time (8)#Reasoning Models (8)#Recommender Systems (8)#Reconciliation (5)#RegTech (8)#Regulation (10)#Reinforcement Learning (9)#Reliability (68)#Resilience (4)#Responsible AI (5)#Retrieval (3)#Risk (22)#Risk Management (11)#Rust (32)#SLSA (3)#SRE (23)#SaaS (9)#Sales (9)#Sales Engineering (8)#Scalability (6)#Search (10)#Security (103)#Security Engineering (8)#Self-Evolving Agents (24)#Semantic Web (8)#Sessions (3)#Settlement (9)#Soft Skills (8)#Software (3)#Software Architecture (36)#Software Delivery (9)#Software Engineering (152)#Spanner (4)#Specification Engineering (3)#Speech (8)#Startups (30)#Statistics (8)#Strands (8)#Strategy (6)#Stream Processing (9)#Streaming (38)#Structured Output (4)#Supply Chain Security (10)#Sustainability (9)#System Design (32)#Systems Programming (56)#Technical Sales (6)#Testing (79)#Threat Modeling (3)#Time Series (9)#Tool Use (23)#Tooling (11)#Tools (4)#Trading (8)#Transformers (3)#Treasury (6)#Trust (4)#Type System (3)#Type Systems (12)#TypeScript (8)#Validation (3)#Valuation (3)#Vector Databases (23)#Vector Search (12)#Venture Capital (8)#Version Control (9)#Versioning (7)#Voice AI (9)#Web Development (6)#Webhooks (9)#Workflow (4)#Workflows (14)#eBPF (8)#gRPC (13)#smolagents (8)
Pratik Dhanave · ·4 min read

Model Evolution I: Learning Without Gold Answers

The deepest locus of self-evolution is the weights — and the interesting frontier there is not supervised fine-tuning on human labels but the family of methods where the model improves from signals it generates or uncovers itself: pseudo-labels from its own consensus, rewards from self-play, curricula it proposes to itself. This post organizes that family by one question — where does the training signal come from when no gold answer exists — and traces the shared failure mode every branch inherits: a model learning from itself can amplify its knowledge, and can just as efficiently amplify its errors.

Label-free weight evolution sorted by signal source: consensus as pseudo-truth (sharpens knowledge, launders bias), self-play as curriculum (risks private equilibria), verifier reward (internalizes verifier exploits), environment signal (honest, slow, governance-heavy). One shared circularity — training on model-shaped data — and the anchor discipline that contains it.

Pratik Dhanave · ·4 min read

Where Artifact Evolution Works: The Verifier Spectrum

Artifact evolution's track record is strangely lumpy: superhuman results on matrix-multiply schedules and scheduling heuristics, respectable results on general code optimization, and little to show on "write me a better product strategy." The lumpiness isn't mysterious — it traces exactly one variable: how good a verifier the domain affords. This post organizes the application domains along that spectrum, from exact checkers to vibes, and reads off where the loop is trustworthy, where it needs guardrails, and where it quietly becomes fiction.

Artifact evolution's lumpy track record traces one variable: verifier strength. The spectrum — exact checkers, measured execution, test suites, proxy scores, LLM judges — with each tier's signature failure, the pressure-budget rule, and the highest-leverage move: reshaping problems toward checkability.

Pratik Dhanave · ·4 min read

Artifact Evolution: The Generate–Evaluate–Refine Loop

The least philosophical form of self-evolving AI is also the one with the hardest results to argue with: hold the model fixed, hold the harness fixed, and evolve the artifact — the program, the schedule, the mathematical construction — through thousands of generate-evaluate-refine rounds. This post dissects that loop: why it's evolutionary computation with the mutation operator replaced by a language model, why the evaluator is the entire ballgame, and what changes when the LLM is both the operator and, increasingly, the optimizer.

Artifact evolution is evolutionary search with an LLM as the mutation operator — population, generator, evaluator, selector — where the LLM contributes semantically valid, evidence-driven edits and the evaluator contributes everything else. Offline loop, mechanical feedback, externally checkable results: the honesty standard for the rest of the map.

Pratik Dhanave · ·5 min read

From U-Net to Diffusion Transformer — The Architecture Underneath the Denoiser

For the first years of diffusion, "the model" meant a U-Net — a convolutional hourglass borrowed from medical image segmentation. Then the same migration that had already run through NLP and vision ran through generation: replace the handcrafted spatial architecture with a plain transformer over patch tokens, and let scale do the work. The Diffusion Transformer (DiT) is now the backbone of the frontier image and video systems — and the reasons why are a case study in what architectures are actually for.

The denoiser was a U-Net — convolutional hourglass, skip connections, interleaved cross-attention — until the Diffusion Transformer re-answered the same requirements generically: patchify the latent, run a standard transformer, condition via adaptive layer norm. The real win is economic: LLM-style scaling curves, inherited infrastructure, and spacetime patches that set up video.

Pratik Dhanave · ·5 min read

Why Agents Forget — The Context Window Is Not Memory

Every production agent has the same embarrassing failure mode: ask it to continue yesterday's work and it has no idea what you're talking about. The model that reasoned brilliantly for two hours retains nothing of those two hours. This is not a bug to patch but an architecture to build — memory is the system around the model, and designing it well is becoming the distinguishing skill of agent engineering. This series builds agent memory from first principles.

LLMs are stateless, and the context window is working memory — bounded, ephemeral, and unevenly attended — not memory. Forgetting costs re-work, broken user relationships, repeated mistakes, and drift, and no model upgrade fixes it: memory is the system around the model. Opens an 8-post series on building it.

Pratik Dhanave · ·6 min read

Operating in Production — Cost, Latency, Incidents, and the Tightening Loop

The final discipline of LLMOps is running the thing as a product that costs real money, has to be fast enough to use, breaks in new ways, and is expected to get better over time. Cost and latency aren't afterthoughts you optimize later — in an LLM app they're first-class constraints that shape the architecture. And incidents aren't failures of planning; they're a certainty you prepare for. This closing post ties the series together into the operating posture that keeps a living system healthy.

Running an LLM app means treating cost and latency as first-class constraints (model routing, token hygiene, caching, streaming), preparing for incidents as certainties (provider failures, instant rollback, blameless postmortems), and the closing idea: operating is continuous improvement — a tightening loop, because an LLM app is never done.

Pratik Dhanave · ·6 min read

Observability and Tracing — Seeing Inside an LLM System

When an LLM application gives a bad answer, "the model messed up" is almost never a useful diagnosis. Was it the retrieval that pulled the wrong document? The prompt template? A tool that returned garbage? The model's generation itself? Without observability built for LLM systems, you can't tell — and you're left re-running prompts hoping to reproduce a non-deterministic failure. This post is about instrumenting an LLM app so that when something goes wrong, you can actually see why.

'The model messed up' is never a useful diagnosis. Why ordinary monitoring isn't enough, tracing as the spine (spans capturing the rendered prompt, retrieved context, tool calls, raw output, tokens/cost), and turning aggregated traces into metrics, quality trends, and drift detection.

Pratik Dhanave · ·6 min read

Guardrails and Safety in Production — Containing Open-Ended Output

An LLM can say anything, which is exactly the problem. The same open-endedness that makes it useful means it can hallucinate, leak data, follow a malicious instruction hidden in its input, or produce something harmful — and in production, in front of real users, those aren't edge cases, they're inevitabilities. Guardrails are the controls that sit around the model to contain what it might do. This post covers the new failure modes LLMs introduce and the layered defenses that keep a production system safe.

An LLM can say anything, which is the problem. The new failure modes — hallucination, prompt injection, data leakage, harmful output — and layered, fail-closed guardrails (input, output, action) with humans in the loop, treated as part of the lifecycle and red-teamed.

Pratik Dhanave · ·6 min read

Eval-Driven Development — You Can't Ship What You Can't Measure

The single practice that separates teams who improve their LLM apps from teams who just change them is evaluation. Without a trustworthy way to measure quality, every prompt tweak is a gamble — you fix one case, break two others, and never know. Eval-driven development makes the eval set the center of gravity: you build it before you optimize, you gate every change on it, and you treat it as the most valuable asset the project owns. This post is about building evaluation you can actually trust.

The practice that separates improving an LLM app from merely changing it. The curated eval set as the core asset, how to score open-ended output (deterministic checks, reference metrics, LLM-as-judge, human review), and how to gate every change on evals and compare B vs A.

Pratik Dhanave · ·7 min read

NLP and LLMs on Financial Documents — Reading at Scale

Finance runs on documents — filings, offering memos, indentures, contracts, credit agreements, research — and until recently machines couldn't read them in any useful way. Large language models changed that, and it's arguably the most consequential development in AI financial engineering: the ability to read, extract, and structure the vast unstructured corpus that finance has always been drowning in. This post is about turning financial documents into analyzable information, reliably and auditably.

Finance runs on documents — filings, memos, indentures, contracts — and LLMs can finally read them at scale. The most consequential capability in AI financial engineering, and the discipline it requires: grounding, provenance on every output, hallucination defense, and human verification. (Educational, not financial advice.)

Pratik Dhanave · ·5 min read

Prompt and Version Management — Treat Prompts Like Code

In an LLM application the prompt is the source code — it's where behavior is defined and where most changes and most bugs happen. Yet teams routinely paste prompts inline, edit them in production, and have no idea which version produced last week's output. Prompt and version management is the unglamorous discipline of treating prompts as the first-class, versioned, tested artifacts they actually are. It's the foundation the rest of the lifecycle stands on.

In an LLM app the prompt is the source code — where behavior is defined and most bugs happen. Treat prompts like code: version the whole configuration (template + model + parameters + context strategy), decouple them from app code, and route every change through review, eval, rollout, and rollback.

Pratik Dhanave · ·5 min read

The LLMOps Lifecycle — The Loop an LLM App Lives In

Post 1 argued an LLM application is never "done." This post gives that idea a shape: the LLMOps lifecycle, the repeating loop of develop → evaluate → deploy → observe → improve that a production LLM system lives inside. Seeing the whole loop at once is what turns a pile of practices — prompt management, evals, guardrails, monitoring — into a coherent operating model, and it's the map for the rest of the series.

The LLMOps lifecycle — develop, evaluate, deploy, observe, improve — as a loop an LLM app lives inside, not a line it runs once. Why evaluation is a first-class stage appearing twice, why production feedback flows backward, and why the loop turns whether you act or not.

Pratik Dhanave · ·5 min read

What LLMOps Is — and Why It's Not Just MLOps

Getting an LLM feature to work in a demo is easy; keeping it working, safe, and affordable in production for a year is a different discipline entirely. That discipline is LLMOps — the operational practices for running LLM-powered applications in production. It inherits a lot from MLOps, but the differences are sharp enough that treating an LLM app like a traditional ML model is one of the most common and expensive mistakes teams make. This series is about the operational layer that the demo never shows you.

Getting an LLM to work in a demo is easy; keeping it working, safe, and affordable for a year is a discipline. What LLMOps is, how it differs from MLOps, and the five properties of LLMs — opaque external models, prompt-defined behavior, fuzzy correctness, non-determinism, new failures/costs — that force a different playbook.

Pratik Dhanave · ·5 min read

Memory Management for LLM Inference — The KV Cache Problem

The single largest consumer of GPU memory during inference often isn't the model — it's the KV cache, the growing store of attention state that lets a model generate text without recomputing the past. Managing it well is the difference between serving a handful of users and serving thousands on the same hardware. This post explains what the KV cache is, why it's such a brutal memory problem, and how PagedAttention borrowed a 1960s operating-systems idea to solve it.

The KV cache often exceeds the model weights in HBM, and naive contiguous allocation wrecks concurrency with fragmentation. PagedAttention applies operating-system paging to fix it — a core inference bottleneck solved by borrowing a classic systems idea.

Pratik Dhanave · ·5 min read

Numeric Precision — Why Lower Precision Makes Inference Faster

The fastest way to speed up LLM inference is often not a better algorithm but fewer bits per number. Precision — how many bits represent each weight and activation — is a direct lever on both of the limits this series has built up: it halves or quarters the data you move (bandwidth, post 2) and unlocks the GPU's fastest arithmetic units (compute, post 3). Understanding floating-point formats is understanding why fp16, bf16, and int8 are everywhere in modern inference.

FP32, FP16, BF16, and INT8/INT4 — how a float splits bits between range and precision, why bf16 won, and why fewer bits is a double win: less data to stream from HBM and faster tensor-core math. Quantization and why LLMs tolerate it.

Pratik Dhanave · ·7 min read

From Bespoke AI to Product

Every AI forward deployed engineer builds one-offs — a bespoke deployment for one customer's data, workflow, and trust. The ones who create lasting value turn those one-offs into product: the patterns that repeat become a platform, the platform makes the next deployment faster, and the field learnings flow back to shape what gets built. This closing post is about the flywheel that turns bespoke AI work into a compounding asset, and the career arc of the engineer who runs it.

Every AI FDE builds one-offs — a bespoke deployment for one customer's data, workflow, and trust. The ones who create lasting value turn those one-offs into product: the patterns that repeat become a platform, the platform makes the next deployment faster, and field learnings flow back to shape what gets built. The flywheel that turns bespoke AI work into a compounding asset — and the AI FDE career arc.

Pratik Dhanave · ·5 min read

The Roofline Model — Are You Compute-Bound or Memory-Bound?

Posts 1 and 2 kept drawing a line between two worlds: workloads limited by how fast the GPU can compute, and workloads limited by how fast it can move data. The roofline model is the single tool that tells you which world you're in for any given operation — and therefore which optimizations will help and which are a waste of time. It is the most useful mental model in performance engineering, and it fits on one chart.

The roofline model answers the one question that governs every optimization — compute-bound or memory-bound? — from arithmetic intensity (FLOPs per byte). Why LLM prefill is compute-bound and decode is memory-bound, and why batching is the master throughput lever.

Pratik Dhanave · ·7 min read

Productionizing and Handing Over AI

A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — and then handing it over so the customer operates it without you. The forward deployed engineer's goal, in the end, is to make themselves unnecessary: a deployment that only works while you're standing next to it hasn't actually been delivered.

A pilot that works is not a system the customer can run. Productionizing AI means making it reliable, affordable, fast, and observable enough to be real infrastructure — then handing it over so the customer operates it without you. The FDE's goal, in the end, is to make themselves unnecessary. Cost, latency, drift, observability, and a handover that transfers the eval discipline, not just the code.

Pratik Dhanave · ·5 min read

The Memory Hierarchy — Why Bandwidth Rules Inference

Post 1 ended on a reversal: GPUs are fast at arithmetic, but arithmetic is rarely the bottleneck — moving data is. This post makes that concrete by walking the GPU's memory hierarchy, from the tiny ultra-fast registers next to the compute units down to the big slow memory that holds the model's weights. Once you see the orders-of-magnitude gaps between these levels, every performance decision in LLM inference starts to make sense.

The GPU memory hierarchy from registers to HBM, the difference between capacity and bandwidth limits, and the defining fact of LLM generation: it's memory-bandwidth-bound because every token reads the whole model from HBM — which is why batching and quantization work.

Pratik Dhanave · ·7 min read

Integrating AI into Real Workflows

A technically excellent AI system that nobody uses has delivered zero value. The last mile of an AI deployment is not the model — it's fitting the system into how real people actually do their jobs, designing an interface that handles uncertainty honestly, and managing the human change of introducing AI into someone's work. This is where deployments succeed or quietly fail, and where the forward deployed engineer's non-technical skills matter most.

A technically excellent AI system that nobody uses has delivered zero value. The last mile of an AI deployment is not the model — it's fitting the system into how real people actually do their jobs, designing an interface that handles uncertainty honestly, and managing the human change of introducing AI into someone's work. Human-in-the-loop, calibrated reliance, autonomy levels, and change management.

Pratik Dhanave · ·5 min read

Why GPUs Power AI — The Hardware Underneath Inference

Every large language model runs on a GPU, but most engineers treat the GPU as a black box that "does the math fast." Understanding why it's fast — and, more usefully, why it's often not as fast as its spec sheet promises — is the foundation of inference engineering. This series works below the serving layer: not how to run an inference server, but how the silicon actually executes a model and where the real bottlenecks live.

Every LLM runs on a GPU, but the spec-sheet FLOPS number is a near-fiction in practice. Why neural nets map onto GPUs (matrix multiply, SMs, tensor cores), and the governing intuition of the series: compute is cheap, data movement is expensive.

Pratik Dhanave · ·7 min read

Evaluation and Trust for Deployed AI

You cannot ship AI you cannot measure, and no enterprise grants a probabilistic system authority over real work on faith. Both problems have the same answer: evaluation. Building the customer's own evaluation set — real examples, their definition of correct — is how the AI forward deployed engineer turns "it seemed good in the demo" into a reliability number, and that number is how trust gets earned.

You cannot ship AI you cannot measure, and no enterprise grants a probabilistic system authority over real work on faith. Both problems have the same answer: evaluation. Building the customer's own eval set — real examples, their definition of correct — turns 'it seemed good in the demo' into a reliability number, and that number is how trust gets earned. Offline/online eval, safe failure, and trust.

Pratik Dhanave · ·7 min read

Grounding AI in the Customer's Data

A frontier model knows the public internet and nothing about the customer. All the value of an AI deployment comes from the opposite: making the model reason over the customer's own documents, records, and knowledge. Grounding is the technical heart of the AI forward deployed engineer's job — connecting a general model to a specific company's messy, permissioned, incomplete data so its answers are about their reality, not the model's imagination.

A frontier model knows the public internet and nothing about the customer. All the value of an AI deployment comes from the opposite: making the model reason over the customer's own documents, records, and knowledge. Grounding — retrieval-augmented generation over messy, permissioned, incomplete data — is the technical heart of the AI FDE's job. With an interactive reference-architecture diagram.

Pratik Dhanave · ·7 min read

Designing for the Context Window

The single constraint that shapes every aspect of agent skill design is the context window — the finite amount of information an agent can hold and attend to at once. A skill that ignores this constraint bloats the agent's context, degrades its focus, and defeats its own purpose. Designing for the context window — loading only what is relevant, only when relevant, as compactly as possible — is the craft at the heart of good skills. This post is about that constraint and how it drives skill design.

The context window is the constraint that shapes every aspect of skill design. Why context is finite AND precious (more context degrades focus), the master principle — load only what's relevant, only when relevant, as compactly as possible — and why this makes skills beat big prompts.

Pratik Dhanave · ·7 min read

From Demo to Pilot

An AI demo is the easiest impressive thing to build and the most misleading. It runs on hand-picked inputs, in a clean environment, with the failures edited out — and it convinces everyone the problem is nearly solved when the real work has barely begun. The AI forward deployed engineer's job in this phase is to use the demo to win belief, then walk the customer honestly across the chasm to a pilot that survives real data.

An AI demo is the easiest impressive thing to build and the most misleading: it runs on cherry-picked inputs, in a clean environment, with the failures edited out — and convinces everyone the problem is nearly solved when the real work has barely begun. Use the demo to win belief, then walk the customer honestly across the chasm to a pilot that survives real data.

Pratik Dhanave · ·7 min read

GraphRAG — Grounding an LLM in a Knowledge Graph

This is the post the whole series was building toward. Retrieval-augmented generation gives an LLM relevant text to ground its answers, but plain RAG retrieves disconnected chunks and misses the relationships between facts. GraphRAG grounds the model in a knowledge graph instead — so it can follow relationships, reason across multiple hops, and answer questions that no single chunk contains. It's where ontologies stop being theory and become the thing that makes an AI system trustworthy on your domain.

The post the series built toward. Plain vector RAG retrieves disconnected chunks and misses relationships; GraphRAG retrieves a subgraph by traversing the knowledge graph, so multi-hop answers come from walked edges, not guessed connections — typed, verifiable, citable. When to use it, and why hybrid often wins.

Pratik Dhanave · ·7 min read

Scoping an AI Use Case

The most important decision an AI forward deployed engineer makes happens before any code: which problem to point the model at. Choose a problem AI is genuinely suited for, with real value and a clear way to measure it, and the engagement can succeed. Choose AI theater — impressive-sounding but ill-fit — and no amount of engineering saves it. Scoping is where AI deployments are won or lost.

The most important decision an AI FDE makes happens before any code: which problem to point the model at. Choose a problem AI is genuinely suited for, with real value and a clear way to measure it, and the engagement can succeed. Choose AI theater — impressive-sounding but ill-fit — and no engineering saves it. Fit vs value, resisting AI theater, augmentation over automation, and picking the wedge.

Pratik Dhanave · ·6 min read

The Rise of the AI Forward Deployed Engineer

The forward deployed engineer was born at Palantir to bridge powerful software and messy customer reality. In the AI era the role has exploded, because frontier models have made that gap wider than ever: a model that dazzles in a demo is a long way from a system that works inside one company's data, workflows, and trust constraints. This series is the technical playbook for the engineer who closes that gap.

The forward deployed engineer was born at Palantir to bridge powerful software and messy customer reality. In the AI era the role exploded, because frontier models widened that gap: a model that dazzles in a demo is a long way from a system that works inside one company's data, workflows, and trust constraints. This series is the technical playbook for the engineer who closes that gap.

Pratik Dhanave · ·6 min read

Speech-to-Speech and the New Realtime Architectures

The cascaded pipeline — ASR, then LLM, then TTS — has powered voice agents for years, but it has an inherent ceiling: every stage adds latency and every conversion loses information. A newer architecture collapses the three models into one that goes from speech directly to speech, promising lower latency and richer understanding. This post compares the cascade with the emerging end-to-end approach, and where each fits.

The cascaded pipeline — ASR, then LLM, then TTS — has powered voice agents for years, but it has an inherent ceiling: every stage adds latency and every conversion loses information. A newer architecture collapses the three models into one that goes from speech directly to speech, promising lower latency and richer understanding. Comparing the cascade with the emerging end-to-end approach.

Pratik Dhanave · ·6 min read

Building and Operating an AI Gateway

You've seen what an AI gateway does; the last question is how to get one — build it, adopt an open-source proxy, or use a managed service — and how to run it once you have it. This closing post assembles the full architecture, weighs build-versus-buy honestly, and covers operating the gateway as the critical piece of infrastructure it becomes.

You've seen what an AI gateway does; the last question is how to get one — build it, adopt an open-source proxy, or use a managed service — and how to run it once you have it. This closing post assembles the full architecture, weighs build-versus-buy honestly, and covers operating the gateway as the critical infrastructure it becomes.

Pratik Dhanave · ·5 min read

Observability and Governance

You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it: what was called, what it cost, how long it took, and whether it was allowed. This post is about turning the gateway into your AI system's source of truth and its governance point.

You can't manage what you can't see, and AI systems are unusually hard to see into — non-deterministic outputs, per-token costs, quality that's a matter of degree. Because every model call flows through the gateway, it's the one place you can observe all of it and govern it: what was called, what it cost, how long it took, whether it was allowed. Turning the gateway into your AI system's source of truth.

Pratik Dhanave · ·6 min read

Rate Limiting, Quotas, and Budgets

Nothing concentrates the mind like a surprise five-figure AI bill from one runaway loop, or one team's traffic spike exhausting the rate limit everyone shares. Because every model call flows through the gateway, it's the one place you can enforce limits and budgets that actually hold — protecting your spend, your providers' rate limits, and fairness across teams. This post is about spending control as a first-class gateway capability.

Nothing concentrates the mind like a surprise five-figure AI bill from one runaway loop, or one team's spike exhausting the shared rate limit. Because every model call flows through the gateway, it's the one place you can enforce limits and budgets that actually hold — protecting your spend, your providers' rate limits, and fairness across teams. Spending control as a first-class capability.

Pratik Dhanave · ·5 min read

What an Ontology Is — Meaning You Can Compute With

"Ontology" sounds like philosophy, and historically it was — the study of what exists. In computing it means something more concrete and more useful: a formal, explicit description of the things in a domain and how they relate, written so a machine can reason about it. As AI systems increasingly need to ground their answers in structured truth rather than guess from text, ontologies and the knowledge graphs built on them have become quietly essential. This series builds both from first principles.

'Ontology' sounds like philosophy; in computing it's a formal, machine-readable description of a domain's things and relationships. The ladder from schema to taxonomy to ontology, the building blocks (classes, individuals, properties, axioms), and why ontologies give LLMs the verifiable source of truth they lack.

Pratik Dhanave · ·5 min read

Text-to-Speech: Giving the Agent a Voice

The last stage of the pipeline is where the agent finally speaks, and it's where an interaction either sounds human or sounds like a robot reading a menu. Modern text-to-speech is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and it must pronounce the messy real world correctly. This post covers TTS for real-time voice.

The last stage of the pipeline is where the agent finally speaks — and where an interaction either sounds human or sounds like a robot reading a menu. Modern TTS is remarkably natural, but for a voice agent it must also be fast and streaming — synthesizing speech as the LLM's words arrive, not after — and pronounce the messy real world correctly. TTS for real-time voice.

Pratik Dhanave · ·5 min read

Caching at the Gateway

Model calls are slow and expensive, and a surprising fraction of them are repeats or near-repeats. Caching at the gateway turns those into instant, free responses — and because the gateway sees all traffic, it's the one place a cache benefits every application at once. Beyond exact-match caching, semantic caching catches queries that mean the same thing in different words, which is where the real savings live.

Model calls are slow and expensive, and a surprising fraction are repeats or near-repeats. Caching at the gateway turns those into instant, free responses — and because the gateway sees all traffic, it's the one place a cache benefits every app at once. Beyond exact-match caching, semantic caching catches queries that mean the same thing in different words, where the real savings live.

Pratik Dhanave · ·5 min read

The LLM Turn: The Brain in the Loop

The language model is where a voice agent stops being a transcription toy and becomes something you can actually talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: it must respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming its answer so the user isn't left waiting. This post is about the LLM stage, adapted for voice.

The language model is where a voice agent stops being a transcription toy and becomes something you can talk to. But dropping an LLM into a real-time voice loop imposes constraints a chatbot never faces: respond in a tight latency budget, produce speech-friendly text, and carry conversation state — all while streaming so the user isn't left waiting. The LLM stage, adapted for voice.

Pratik Dhanave · ·6 min read

Reliability: Fallback, Retries, and Circuit Breakers

Model providers go down, rate-limit you, and time out — regularly. If your application calls one provider directly, its reliability is capped at that provider's. An AI gateway breaks that ceiling: because it can route across providers, a failure on one becomes a transparent retry on another. This post covers the reliability patterns that turn provider outages into non-events.

Model providers go down, rate-limit you, and time out — regularly. If your application calls one provider directly, its reliability is capped at that provider's. An AI gateway breaks that ceiling: because it can route across providers, a failure on one becomes a transparent retry on another. Retries, fallback, and circuit breakers — with an interactive request-flow sequence diagram.

Pratik Dhanave · ·5 min read

Routing and Load Balancing

Once every model call flows through one place, that place can make an intelligent decision on every request: which model should serve this, and through which of your capacity? Routing picks the right model for the task; load balancing spreads traffic across providers and keys so no single limit or outage bottlenecks you. Together they turn the gateway from a passthrough into a control plane.

Once every model call flows through one place, that place can make an intelligent decision on every request: which model should serve this, and through which of your capacity? Routing picks the right model for the task; load balancing spreads traffic across providers and keys so no single limit or outage bottlenecks you. Together they turn the gateway into a control plane.

Pratik Dhanave · ·6 min read

The Anatomy of a Modern Frontier LLM

Put the pieces together and a modern frontier language model comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, and which is engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.

Put the pieces together and a modern frontier LLM comes into focus: a deep transformer whose feed-forward blocks are Mixtures of Experts, whose attention is made efficient with grouped-query and FlashAttention, engineered end to end around one goal — maximum capability per unit of compute and memory. This closing post assembles the anatomy and looks at where architecture is heading.

Pratik Dhanave · ·5 min read

The Unified API

The first thing an AI gateway gives you is one interface to every model. Instead of your applications learning each provider's SDK, request format, and quirks, they speak a single API and the gateway translates. That translation layer is what decouples your code from any one vendor — and it's what makes model-swapping a config change instead of a rewrite.

The first thing an AI gateway gives you is one interface to every model. Instead of your applications learning each provider's SDK, request format, and quirks, they speak a single API and the gateway translates. That translation layer is what decouples your code from any one vendor — and makes model-swapping a config change instead of a rewrite.

Pratik Dhanave · ·5 min read

Efficient Attention

The two costs of long context — quadratic attention compute and linear KV-cache memory — each have a family of solutions, and together they're why modern models can handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. This post covers the techniques that made long context practical.

The two costs of long context each have a family of solutions, and together they're why modern models handle context lengths that were impossible a few years ago. Grouped-query attention shrinks the KV cache; FlashAttention computes exact attention far faster; sliding-window and sparse patterns break the quadratic. The techniques that made long context practical, and which cost each attacks.

Pratik Dhanave · ·5 min read

Why You Need an AI Gateway

The moment your application talks to more than one model — or one model but seriously — you accumulate a pile of cross-cutting concerns: provider APIs that differ, outages you must survive, costs you must control, calls you must log. An AI gateway is the single control point that handles all of it, sitting between your applications and every model provider. This series builds one from first principles.

The moment your application talks to more than one model — or one model but seriously — you accumulate cross-cutting concerns: differing provider APIs, outages, costs, logging. An AI gateway is the single control point that handles all of it, sitting between your applications and every model provider. This series builds one from first principles, with interactive architecture diagrams.

Pratik Dhanave · ·5 min read

The Long-Context Problem

MoE scales a model's parameters cheaply. But there's a second scaling axis that matters just as much for modern LLMs: context length — how much text the model can attend to at once. Attention's cost grows with the square of the sequence, and the memory to run it grows linearly and relentlessly, which is why long context was hard and why so much architectural ingenuity has gone into it.

MoE scales a model's parameters cheaply, but there's a second axis that matters just as much: context length. Attention's compute grows with the square of the sequence, and the KV-cache memory grows linearly and relentlessly — two distinct costs, often confused, that make long context hard. Understanding both is the setup for the efficiency techniques that solved them.

Pratik Dhanave · ·6 min read

Evaluating and Operating Guardrails

A defense you haven't tested is a hope, not a control. The final discipline of LLM security is treating your guardrails as a system to be measured, attacked, and monitored continuously — because the threat evolves, your application changes, and a defense that worked last quarter can silently rot. This closing post covers red-teaming your own system, operating it in production, and the honest state of the art.

A defense you haven't tested is a hope, not a control. The final discipline of LLM security is treating your guardrails as a system to be measured, attacked, and monitored continuously — because the threat evolves, your application changes, and a defense that worked last quarter can silently rot. Red-teaming, production monitoring, and the honest state of the art.

Pratik Dhanave · ·6 min read

Training and Serving MoE Models

Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. This post covers the trade MoE actually makes and the parallelism it forces.

Mixture of Experts saves compute but not memory — and that single fact reshapes everything about how these models are trained and served. All the experts must exist in memory even though only a few run per token, which turns MoE into a distributed-systems problem as much as a machine-learning one. The trade MoE makes and the expert parallelism it forces.

Pratik Dhanave · ·6 min read

Guardrails in Practice

"Guardrails" is the umbrella term for the runtime checks that sit around a model and screen what goes in and comes out — content classifiers, moderation models, topic and format validators, PII detectors. They're a real and useful layer, distinct from the architectural defenses, and they come with their own design rules: layer them, fail safe, and never mistake them for a wall.

"Guardrails" is the umbrella term for the runtime checks around a model — content classifiers, moderation models, topic and format validators, PII detectors. They're a real and useful layer, distinct from architectural defenses, with their own design rules: layer them, fail safe, and never mistake them for a wall. This post covers building the layered defense in practice.

Pratik Dhanave · ·6 min read

Routing and the Load-Balancing Problem

The router is where Mixture of Experts succeeds or fails. Left to its own devices, it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE, and the techniques for it are what separate a working sparse model from a broken one.

The router is where Mixture of Experts succeeds or fails. Left alone it tends to collapse — sending most tokens to a handful of favorite experts while the rest starve, wasting the model's capacity. Making routing spread load evenly, without hurting quality, is the central engineering challenge of MoE: load-balancing losses, expert capacity, and token dropping.

Pratik Dhanave · ·6 min read

Treating Model Output as Untrusted

Injection defense usually focuses on what goes into the model. But an equally dangerous class of bug lives on the way out: whatever the model produces gets passed to another system — a browser, a shell, a database, another service — that trusts it. If the model can be made to emit a malicious payload, and your code renders or executes it, the injection escapes the model and lands in your infrastructure.

Injection defense usually focuses on what goes into the model, but an equally dangerous class of bug lives on the way out: whatever the model produces gets passed to another system that trusts it. If the model can be made to emit a malicious payload and your code renders or executes it, the injection escapes the model and lands in your infrastructure — XSS, SQLi, SSRF, exfiltration.

Pratik Dhanave · ·5 min read

Mixture of Experts: The Core Idea

Now we open up the mechanism at the heart of modern LLMs. A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel "experts" and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling the first post promised, made concrete.

A Mixture-of-Experts layer replaces the transformer's single feed-forward block with many parallel experts and a router that sends each token to just a few of them. The result is a model with enormous total capacity that spends only a little compute on any given token — the decoupling of capacity from per-token compute, made concrete. Experts, routing, and how outputs combine.

Pratik Dhanave · ·6 min read

Architecture and Least Privilege: The Real Defense

Everything before this post raised the probability barrier against injection. This post lowers the impact — and impact is what actually protects you. The load-bearing defense against prompt injection isn't a prompt or a filter; it's an architecture where a fully-hijacked model still can't do anything catastrophic, because it was never granted the power to.

Everything before this raised the probability barrier against injection; this post lowers the impact — and impact is what actually protects you. The load-bearing defense isn't a prompt or a filter but an architecture where a fully-hijacked model still can't do anything catastrophic: least privilege, the confused-deputy trap, human-in-the-loop, and the dual-LLM pattern.

Pratik Dhanave · ·5 min read

The Transformer Backbone, Recapped

Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter for the rest of the series: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.

Mixture of Experts is a modification to the transformer, so you can't understand modern LLM architecture without a clear picture of the transformer itself. This post is a focused recap of the parts that matter: attention as the mechanism that mixes information across tokens, the feed-forward block that MoE replaces, and how the pieces stack into the model everyone is now modifying.

Pratik Dhanave · ·5 min read

Prompt Hardening and Its Limits

Between filtering the input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack. None of them close the hole — because they are all still text in the one channel the attacker also writes to — but used well they meaningfully shift the odds.

Between filtering input and re-architecting the system sits a tempting middle ground: make the prompt itself more resistant. Delimiters, spotlighting, instruction placement, and defensive system prompts all raise the cost of an attack — but none close the hole, because they're all still text in the one channel the attacker also writes to. The value is knowing exactly how much they buy.

Pratik Dhanave · ·5 min read

From Dense to Sparse: Why Modern LLMs Changed Shape

The frontier language models of the last few years share a structural secret that isn't obvious from the outside: most of them aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while only using a fraction of them on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are actually built.

Frontier language models share a structural secret that isn't obvious from the outside: most aren't dense. They're sparse — built from Mixture-of-Experts layers that let a model have hundreds of billions of parameters while using only a fraction on any given token. Understanding why models moved from dense to sparse is the key to understanding how modern LLMs are built.

Pratik Dhanave · ·5 min read

Input Defenses and Their Limits

The first instinct when facing prompt injection is to inspect the input and block the bad stuff. It's a reasonable layer — but a treacherous one, because it creates a feeling of safety far larger than the protection it provides. This post covers the input-side defenses that are genuinely worth having, and draws a hard line around what they can and cannot do.

The first instinct against injection is to inspect the input and block the bad stuff. It's a reasonable layer but a treacherous one — it creates a feeling of safety far larger than the protection it provides. This post covers the input defenses genuinely worth having, and draws a hard line around what they can't do (starting with: indirect injection bypasses them entirely).

Pratik Dhanave · ·6 min read

A Taxonomy of Injection and Jailbreak Attacks

You can't defend against what you can't categorize. Prompt-based attacks come in a few structurally distinct shapes — direct injection, indirect injection through content the model reads, and jailbreaks that target the model's safety training — and each demands a different defense. This post maps the attack surface so the rest of the series can defend it systematically.

You can't defend what you can't categorize. Prompt-based attacks come in structurally distinct shapes — direct injection, indirect injection through content the model reads, and jailbreaks targeting the model's safety training — and each demands a different defense. This post maps the attack surface so the rest of the series can defend it systematically.

Pratik Dhanave · ·6 min read

Why Prompt Injection Is Unsolved

Prompt injection is the defining security problem of LLM applications, and — unlike SQL injection, which it superficially resembles — it has no clean fix. The reason is structural: a language model reads instructions and data through the same channel, and cannot reliably tell which is which. This opening post explains why that makes injection fundamentally hard, and reframes the goal from "prevent it" to "contain the blast radius."

Prompt injection is the defining security problem of LLM applications, and unlike SQL injection it has no clean fix — because a model reads instructions and data through the same channel and can't reliably tell them apart. This opening post explains why that makes injection fundamentally hard, and reframes the goal from prevent to contain the blast radius.

Pratik Dhanave · ·6 min read

RLHF — How Reinforcement Learning Aligns Language Models

The single most consequential application of reinforcement learning today isn't a game or a robot — it's the technique that turned raw, unruly language models into helpful assistants. Reinforcement learning from human feedback solved the problem that you can't write a reward function for "be helpful, honest, and harmless," so you learn one from human judgments instead. This post connects everything in the series to the method that made modern AI assistants possible.

The most consequential use of RL today: you can't write a reward for 'be helpful, honest, harmless', so RLHF learns one from human preference comparisons and optimizes with PPO. The three-stage pipeline, the KL constraint against reward hacking, and the series' capstone — plus its limits (sycophancy, bias).

Pratik Dhanave · ·5 min read

Policy Gradients and Actor-Critic — Learning the Policy Directly

Value-based methods like Q-learning learn how good each action is, then act greedily. There's a different and often better approach: skip the value estimates and learn the policy itself, directly adjusting the probability of actions to get more reward. Policy gradient methods are how RL handles continuous control and stochastic strategies — and the actor-critic architecture that combines them with value learning is the foundation of the algorithms used to fine-tune today's language models.

Instead of learning values then acting greedily, learn the policy directly: adjust action probabilities to earn more reward. Why this handles continuous and stochastic policies, the high-variance problem, and actor-critic (and PPO) — the algorithm family behind RLHF and LLM alignment.

Pratik Dhanave · ·6 min read

The Machine Learning Workflow — From Problem to Production

Knowing the algorithms is maybe 20% of doing machine learning. The other 80% is the workflow around them: framing the problem, preparing the data, iterating honestly, and getting a model into production where it keeps working. This closing post assembles the series into the end-to-end process real ML follows — and names the places where projects actually fail, which are almost never the algorithm.

Algorithms are ~20% of ML; the other 80% is the workflow — framing the problem, preparing data, iterating honestly, and keeping a model working in production where it decays (concept drift). Why classical ML isn't obsolete, and why these foundations are universal to deep learning and LLMs.

Pratik Dhanave · ·6 min read

LangChain in Production

The gap between a LangChain demo and a LangChain production system is the same gap as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline the rest of your engineering already has. This closing post covers operating LangChain applications and the honest verdict on when to use the framework.

The gap between a LangChain demo and a production system is the same as any LLM application — observability, evaluation, cost, and reliability — and LangChain's answer is LangSmith plus the discipline your engineering already has.

Pratik Dhanave · ·5 min read

Models

A code agent is only as good as the model writing the code — and smolagents, true to its Hugging Face roots and minimalist ethos, lets you run almost any model behind it: open models from the Hub, hosted APIs, local models, anything reachable through its integrations. Model choice matters especially here, because the model's code-writing ability is the agent's ability.

A code agent is only as good as the model writing the code — and smolagents lets you run almost any model: open models from the Hub, hosted APIs, local models. Model choice matters especially here, because the model's code-writing ability is the agent's ability.

Pratik Dhanave · ·6 min read

Testing and Evals

Most agent code is tested by running it and eyeballing the output — because testing agents properly is genuinely hard. Pydantic AI's whole design has been quietly building toward making it easy: type safety, dependency injection, and test models combine so you can unit-test agent logic deterministically, offline, without ever calling a real LLM. This is arguably the framework's biggest practical advantage.

Most agent code is tested by running it and eyeballing the output, because testing agents properly is hard. Pydantic AI's design builds toward making it easy: type safety, dependency injection, and test models let you unit-test agent logic deterministically, offline, with no real LLM.

Pratik Dhanave · ·6 min read

Fine-Tuning in Production

Training a good fine-tune is the easy half. Running it in production — deciding it's even worth it, serving adapters efficiently, keeping it from going stale as base models leap ahead, and re-tuning as your needs shift — is where fine-tuning becomes an ongoing commitment rather than a one-time project. This is the reality check that closes the series.

Training a good fine-tune is the easy half. Running it in production — deciding it's worth it, serving adapters efficiently, keeping it from going stale as base models leap ahead, and re-tuning — is where fine-tuning becomes an ongoing commitment.

Pratik Dhanave · ·6 min read

Latency, Throughput, and Cost

There is no single "make it fast" for LLM serving — latency and throughput pull against each other, and both trade against cost. The job isn't to maximize one number; it's to hit your latency targets at the lowest cost per token, which means knowing exactly which knob moves which metric and in which direction.

There is no single 'make it fast' for LLM serving — latency and throughput pull against each other, and both trade against cost. The job is to hit your latency targets at the lowest cost per token, knowing which knob moves which metric.

Pratik Dhanave · ·5 min read

Postgres/pgvector vs a Dedicated Vector Database

The vector-storage decision has a boringly practical answer that cuts against the hype: for most systems, the database you already run with a vector extension beats adding a new specialized system — until scale or specific features force the upgrade.

A boringly practical answer that cuts against the hype: for most systems the database you already run with a vector extension beats adding a specialized system — until scale or specific features force the upgrade.

Pratik Dhanave · ·5 min read

Regression and Linear Models — The Workhorses You Should Understand First

Before neural networks, the two models that still quietly run more of the world's predictions than anything else are linear and logistic regression. They are simple enough to understand completely, fast enough to train on anything, and interpretable enough to explain to a regulator — and understanding them teaches the mechanics (fitting, loss, gradient descent) that every more complex model shares. This post builds both from scratch and shows why "just use a linear model" is often the right answer.

Linear and logistic regression still run more of the world's predictions than anything else — and they teach the universal loop (model, loss, gradient descent) that scales all the way to training an LLM. Fitting a line, the learning rate, and the sigmoid that becomes the softmax.

Pratik Dhanave · ·6 min read

Evaluation in Production

Offline evaluation tells you whether a change is promising; production tells you whether it actually works. Once your system is serving real users, evaluation becomes continuous: online experiments, guardrail metrics, drift monitoring, and gating deploys on eval scores. This closing post moves evaluation from the lab into the running system and ties the whole series into a working loop.

Offline evaluation tells you whether a change is promising; production tells you whether it works. Once you're serving real users, evaluation becomes continuous — online experiments, guardrail metrics, drift monitoring, CI gating. This closing post moves evaluation from the lab into the running system and ties the series into one loop.

Pratik Dhanave · ·5 min read

Model Providers

The model drives a Strands agent, so which model you use is the single biggest determinant of how well it works — and Strands keeps that a swappable choice across providers rather than locking you to one. Model-agnosticism isn't a convenience here; in a model-driven framework it's foundational.

The model drives a Strands agent, so which model you use is the biggest determinant of how well it works — and Strands keeps that a swappable choice across providers. Model-agnosticism isn't a convenience here; it's foundational.

Pratik Dhanave · ·6 min read

Evaluating a Fine-Tuned Model

A fine-tune that looks great on a handful of hand-picked prompts can be quietly broken — overfit to your training data, worse than the base model you started from, or catastrophically forgetful of skills it used to have. Without real evaluation you can't tell, and shipping a fine-tune you haven't measured is shipping a guess.

A fine-tune that looks great on a handful of prompts can be quietly broken — overfit, worse than the base model, or catastrophically forgetful. Without real evaluation you can't tell, and shipping a fine-tune you haven't measured is shipping a guess.

Pratik Dhanave · ·7 min read

Scaling Across GPUs

At some point a model doesn't fit on one GPU, or the traffic doesn't, and you have to spread inference across many. The choices — which kind of parallelism, how to place replicas, when to autoscale — are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication). Get the memory math right and most scaling decisions follow.

At some point a model doesn't fit on one GPU, or the traffic doesn't. The scaling choices are governed by one unforgiving resource (GPU memory) and one expensive one (inter-GPU communication) — get the memory math right and most decisions follow.

Pratik Dhanave · ·5 min read

Managed API vs Self-Hosting Open Models

This is the classic fixed-versus-marginal decision, and it has a clean answer: managed APIs win until your volume is high and steady enough to keep expensive GPUs busy — which is a much higher bar than most teams assume.

The classic fixed-vs-marginal decision with a clean answer: managed APIs win until your volume is high and steady enough to keep expensive GPUs busy — a much higher bar than most teams assume.

Pratik Dhanave · ·6 min read

Human Evaluation and Preference

Automated metrics and LLM judges are scalable proxies for the thing that actually matters: whether a human finds the output good. Human evaluation is the ground truth those proxies are calibrated against — and getting it right is its own discipline, full of subtle traps around agreement, bias, and how you ask the question. This post is about doing human eval well, and about how human preference became the signal that trains models themselves.

Automated metrics and LLM judges are scalable proxies for the thing that matters: whether a human finds the output good. Human evaluation is the ground truth those proxies are calibrated against — a discipline full of traps around agreement, bias, and how you ask — and human preference is now the signal that trains models themselves.

Pratik Dhanave · ·6 min read

Secure Code Execution

The power of code agents comes with a sharp edge: you are executing code written by an LLM, and an LLM can be wrong, or manipulated into writing something harmful. Running that code unsandboxed is one of the most dangerous things you can do in an application, so sandboxing isn't optional for code agents — it's the price of admission.

The power of code agents comes with a sharp edge: you are executing code written by an LLM, which can be wrong or manipulated. Running it unsandboxed is one of the most dangerous things you can do — so sandboxing is the price of admission.

Pratik Dhanave · ·7 min read

Effort Tuning, Budgets, and the max_tokens Trap

Two more levers of control: how hard the model thinks, and how you bound what it spends. Both have measured sweet spots — and both have a trap that quietly wastes money. The sharpest is a cap that most people reach for first and that saves nothing at all: max_tokens.

Two levers of control: how hard the model thinks, and how you bound what it spends. Both have measured sweet spots — and both have a trap that quietly wastes money. The sharpest is a cap most people reach for first that saves nothing: max_tokens.

Pratik Dhanave · ·6 min read

Retrieval and RAG

Answering questions over your own data is the most common LLM application, and LangChain gives you the whole pipeline as composable, swappable components — loaders, splitters, embeddings, vector stores, retrievers — behind standard interfaces. The retriever, in particular, is just another Runnable, so RAG becomes a chain like any other.

Answering questions over your own data is the most common LLM application, and LangChain gives you the whole pipeline as composable, swappable components — loaders, splitters, embeddings, vector stores, retrievers. The retriever is just another Runnable.

Pratik Dhanave · ·6 min read

Alignment: RLHF and DPO

Supervised fine-tuning teaches a model to produce a correct answer; alignment teaches it to produce the answer people actually prefer. That shift — from "right" to "better" — is what turned raw language models into helpful assistants, and the move from RLHF's complex machinery to DPO's direct approach made it something ordinary teams can do.

Supervised fine-tuning teaches a model to produce a correct answer; alignment teaches it to produce the answer people actually prefer. That shift turned raw models into helpful assistants, and the move from RLHF to DPO made it something ordinary teams can do.

Pratik Dhanave · ·6 min read

Serving Engines and PagedAttention

You don't assemble the KV cache, continuous batching, quantization, and speculative decoding by hand — you use a serving engine that has already solved the hard parts. And the idea that ties them together, PagedAttention, is a borrowed operating-systems trick: manage the KV cache like virtual memory, in pages, and the waste that throttled everything disappears.

You don't assemble the KV cache, batching, and quantization by hand — you use a serving engine. And PagedAttention, the idea that ties them together, is a borrowed OS trick: manage the KV cache like virtual memory, in pages.

Pratik Dhanave · ·5 min read

MCP vs A2A: Tools vs Agents

The most common question about the two big agent protocols is which one to use — and the answer is almost always "both," because they solve different problems: MCP connects an agent to its tools, A2A connects an agent to other agents.

The most common question about the two big agent protocols is which to use — and the answer is almost always both, because MCP connects an agent to its tools and A2A connects an agent to other agents.

Pratik Dhanave · ·5 min read

Cost Observability and AI FinOps

You cannot manage what you cannot see, and the difference between a team that controls its AI spend and one that is surprised by it is almost always whether they measure cost per feature, per user, and per call.

You cannot manage what you cannot see. The difference between a team that controls its AI spend and one surprised by it is whether they measure cost per feature, per user, and per call.

Pratik Dhanave · ·5 min read

Building a Context Pipeline

Everything in this series — the budget, the system prompt, retrieval, memory, tools, and compaction — comes together as a pipeline that assembles the right window on every single turn, deliberately rather than by accident.

The budget, system prompt, retrieval, memory, tools, and compaction come together as a pipeline that assembles the right window on every turn — deliberately rather than by accident.

Pratik Dhanave · ·5 min read

What Machine Learning Is — Programs That Learn from Data

Most of this blog's AI content sits on top of machine learning without ever defining it. This series fixes that from the ground up: before transformers and LLMs, before fine-tuning and RAG, there is the plain idea of a program that improves at a task by learning patterns from data instead of being explicitly told the rules. Understanding that idea — and its three core flavors — is the foundation everything else in modern AI is built on.

Before transformers and LLMs there is the plain idea of a program that learns patterns from data instead of being told the rules. What machine learning is, the vocabulary (features, model, training, generalization), and the three paradigms — supervised, unsupervised, reinforcement — the whole AI stack is built on.

Pratik Dhanave · ·6 min read

Contamination, Gaming, and Goodhart's Law

A benchmark score is trustworthy only if the model hasn't seen the answers and no one has optimized directly for the test. Both assumptions fail constantly. Training data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target and destroys its meaning. This post is about the ways evaluation gets corrupted — and how to defend against them.

A benchmark score is trustworthy only if the model hasn't seen the answers and no one optimized directly for the test — and both assumptions fail constantly. Training-data contamination inflates scores by rewarding memorization; optimizing for a benchmark turns it from a measure into a target. Here's how evaluation gets corrupted, and how to defend it.

Pratik Dhanave · ·6 min read

Why Code Actions Win

The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This post examines the evidence and the mechanism, so you understand not just that code actions win but why.

The claim that agents should write code isn't just aesthetic — Hugging Face backs it with measured results: code agents take fewer steps, make fewer LLM calls, and score better on hard benchmarks. This examines the evidence and the mechanism.

Pratik Dhanave · ·7 min read

Model Selection and Prompt Audits

Model selection is the most commonly botched cost decision, because the intuitive answer — pick the cheaper model — is measured by the wrong number. The right number is cost per completed task, and by that measure the more capable model often wins. Paired with it is the least-known lever of all: auditing prompts written for an older model against your current one.

Model selection is the most commonly botched cost decision, because the intuitive answer — pick the cheaper model — is measured by the wrong number. The right number is cost per completed task, priced on the tail not the median.

Pratik Dhanave · ·6 min read

Data: The Real Determinant

You can pick the perfect technique, tune every hyperparameter, and rent the best GPU — and still get a worse model than you started with, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones. This is the post that actually determines whether your fine-tune works.

You can pick the perfect technique and rent the best GPU, and still get a worse model, because your dataset was mediocre. Fine-tuning quality is decided overwhelmingly by data quality, and a few hundred excellent examples beat tens of thousands of sloppy ones.

Pratik Dhanave · ·6 min read

Gemma on Flutter

This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI. The plugin hides the runtime; you write an app.

This is where the concepts become code. With a quantized Gemma model and the flutter_gemma plugin, running a real LLM entirely on the user's phone is a handful of Dart calls — load the model, open a chat session, stream tokens into your UI.

Pratik Dhanave · ·6 min read

Speculative Decoding

Decode is slow because it's sequential — one token at a time, each waiting for the last. Speculative decoding cheats that limit with a beautiful trick: let a small, fast model guess several tokens ahead, then let the big model verify them all in a single pass. When the guesses are good, you get several tokens for the price of one — with mathematically identical output.

Decode is slow because it's sequential. Speculative decoding cheats that with a beautiful trick: a small fast model guesses several tokens ahead, the big model verifies them in one pass — several tokens for the price of one, with identical output.

Pratik Dhanave · ·5 min read

RAG vs Fine-Tuning vs Long-Context

The most common architecture mistake in applied AI is reaching for fine-tuning to fix a knowledge problem — so the single most useful rule here is that RAG is for knowledge and fine-tuning is for behavior, and long-context is a convenience, not a strategy.

The most common architecture mistake is reaching for fine-tuning to fix a knowledge problem — so the key rule: RAG is for knowledge, fine-tuning is for behavior, and long-context is a convenience, not a strategy.

Pratik Dhanave · ·5 min read

RAG, Fine-Tuning, and Self-Hosting Trade-offs

Some of the biggest cost decisions are architectural — whether to feed knowledge through retrieval or bake it into a model, whether to prompt a big model or fine-tune a small one, and whether to rent tokens or run your own — and each trade turns on your volume and your task.

Some of the biggest cost decisions are architectural — retrieval versus long context, prompting a big model versus fine-tuning a small one, and renting tokens versus self-hosting — each turning on your volume.

Pratik Dhanave · ·5 min read

Compaction, Summarization, and Long Context

When context threatens to overflow, you compress it; and when you have a huge window to spend, you still should not fill it — because a long context is not used as well as a short, focused one.

When context threatens to overflow you compress it; and when you have a huge window you still should not fill it — because a long context is not used as well as a short, focused one.

Pratik Dhanave · ·5 min read

Benchmarks and How They're Designed

The scores that dominate model announcements — MMLU, GSM8K, HumanEval, and the rest — are benchmarks: standardized public tests that let the whole field compare models on common ground. They've driven enormous progress, but a benchmark is only as good as its design, and a number without understanding of what it measures is easy to misread. Knowing how benchmarks are built, and what makes a good one, is how you read a leaderboard critically instead of credulously.

The scores in model announcements — MMLU, GSM8K, HumanEval — are benchmarks: standardized public tests for comparing models on common ground. They've driven huge progress, but a benchmark is only as good as its design, and a number without understanding is easy to misread. Here's how they're built and what makes a good one.

Pratik Dhanave · ·6 min read

Batching and Token Hygiene

The second-largest cost lever is almost embarrassingly simple: if no one is waiting on a response, run it in a batch for half off — and the discount is identical across all three major clouds, which makes it a safe architectural assumption. Then comes token hygiene, where the wins are real but one popular technique quietly costs more than it saves.

The second-largest cost lever is almost embarrassingly simple: if no one is waiting on a response, run it in a batch for half off — identical across all three major clouds. Then comes token hygiene, where one popular technique quietly costs more than it saves.

Pratik Dhanave · ·6 min read

Structured Outputs

This is the feature Pydantic AI is named for and built around: you declare a Pydantic model as your agent's output type, and you get back a validated instance of it — not a string to parse, not JSON to hope about, but a real typed object. It turns the single most brittle part of LLM applications into the most reliable.

This is the feature Pydantic AI is named for: you declare a Pydantic model as your agent's output type and get back a validated instance — not a string to parse. It turns the most brittle part of LLM applications into the most reliable.

Pratik Dhanave · ·6 min read

QLoRA and Quantized Fine-Tuning

LoRA shrank the trainable parameters, but you still had to hold the full base model in memory to train against it — and for a large model that alone needs serious hardware. QLoRA closes the gap: quantize the frozen base model to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss. It's what truly democratized fine-tuning.

LoRA shrank the trainable parameters, but you still had to hold the full base model in memory. QLoRA closes the gap: quantize the frozen base to 4-bit, train LoRA adapters on top, and fine-tune a large model on a single consumer GPU with almost no quality loss.

Pratik Dhanave · ·6 min read

Quantization

Quantization shrinks a model by storing its numbers in fewer bits — and because LLM decode is bottlenecked on moving those numbers from memory, making them smaller makes inference both cheaper to host and faster to run. It's the rare optimization that improves memory, cost, and speed at once, if you respect its limits on quality.

Quantization stores a model's numbers in fewer bits — and because decode is bottlenecked on moving those numbers from memory, making them smaller makes inference cheaper to host and faster to run, if you respect the limits on quality.

Pratik Dhanave · ·5 min read

Choosing a Model Platform: Bedrock vs watsonx vs NVIDIA NIM vs Vertex

The model platform decision is usually decided before you compare models at all — by which cloud you're already on, what governance you need, and whether you're renting inference or running it — and getting that framing right matters more than any benchmark.

The model-platform decision is usually settled before you compare models — by which cloud you're on, what governance you need, and whether you're renting inference or running it.

Pratik Dhanave · ·5 min read

Self-Play and Co-Evolution

The technique that produced superhuman game-playing — a system improving by competing against copies of itself — has an LLM analogue: models that generate their own training signal and bootstrap their way up without new human labels.

The technique that produced superhuman game-playing has an LLM analogue: models that generate their own training signal and bootstrap up without new human labels — and the reward-source problem at its center.

Pratik Dhanave · ·5 min read

Batching, Async, and Throughput

Not all AI work needs an answer this second, and for the work that can wait, batching and asynchronous processing buy meaningfully cheaper tokens in exchange for latency you were not using anyway.

Not all AI work needs an answer this second, and for the work that can wait, batching and asynchronous processing buy meaningfully cheaper tokens in exchange for latency you were not using anyway.

Pratik Dhanave · ·5 min read

Tools and Structured Context

Tool definitions and structured data quietly consume a large share of the context budget, and how you select, describe, and format them shapes both what fits and how well the model uses it.

Tool definitions and structured data quietly consume a large share of the context budget, and how you select, describe, and format them shapes both what fits and how well the model uses it.

Pratik Dhanave · ·5 min read

Building an MCP Client

A server is only half the story; the client is what connects to it, discovers its capabilities, and turns a model's intent into real tool calls.

Build an MCP client that launches a server, discovers its tools, and drives them from a language model — the core of what every MCP host does internally.

Pratik Dhanave · ·6 min read

Building an Eval Harness

Metrics and judges are ingredients; a harness is the kitchen. An evaluation harness is the system that takes a dataset of test cases, runs your LLM system over them, scores the outputs, and reports the results — reproducibly, every time. Building one well is what turns evaluation from a one-off spreadsheet into an engineering asset you run on every change, like a test suite.

Metrics and judges are ingredients; a harness is the kitchen. An eval harness takes a dataset of test cases, runs your system over them, scores the outputs, and reports results — reproducibly, every time. Building one well turns evaluation from a one-off spreadsheet into an engineering asset you run on every change.

Pratik Dhanave · ·7 min read

Prompt Caching: The Single Largest Lever

Of every AI cost lever, one dominates the rest, and all the major vendors agree on it: prompt caching. On long prompts and agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.

Of every AI cost lever, one dominates and all the major vendors agree on it: prompt caching. On agentic workloads the reported savings are the largest anywhere — because caching attacks the specific way agent costs explode. If you optimize one thing, optimize this.

Pratik Dhanave · ·6 min read

Models, Prompts, and Output Parsers

Every LangChain application, no matter how elaborate, is built from three humble pieces: a model you call, a prompt you send it, and a parser that turns its reply into something usable. Master these three and the rest of LangChain is just composing them — which is exactly what the framework is designed to let you do.

Every LangChain application, no matter how elaborate, is built from three humble pieces: a model you call, a prompt you send it, and a parser that turns its reply into something usable. Master these three and the rest is just composing them.

Pratik Dhanave · ·6 min read

Parameter-Efficient Fine-Tuning: LoRA

Full fine-tuning updates every weight in a model — billions of numbers — which needs enormous memory and produces a full-size copy per task. LoRA sidesteps all of it with one insight: the change a model needs for a task is "low-rank," so you can train a tiny pair of matrices instead of the whole model. It's the technique that put fine-tuning within reach of anyone with a single GPU.

Full fine-tuning updates every weight — billions of numbers — needing enormous memory. LoRA sidesteps it with one insight: the change a model needs for a task is low-rank, so you train a tiny pair of matrices instead of the whole model.

Pratik Dhanave · ·6 min read

Making Models Fit: Quantization for the Edge

On the cloud, quantization is an optimization you reach for to cut costs. On a phone, it's the difference between the model running and not running at all. Nearly every on-device LLM you'll ever ship is quantized, because full-precision weights simply don't fit — so understanding the bit-width trade-off is non-negotiable for edge AI.

On the cloud, quantization cuts costs. On a phone, it's the difference between the model running and not running at all — nearly every on-device LLM you'll ship is quantized, so the bit-width trade-off is non-negotiable for edge AI.

Pratik Dhanave · ·6 min read

Batching and Throughput

A single request leaves an expensive GPU almost entirely idle. Batching is how you fill it — and the leap from naive static batching to continuous batching is the single biggest throughput improvement in modern LLM serving, often several times more requests from the exact same hardware.

A single request leaves an expensive GPU almost entirely idle. Batching fills it — and the leap from static to continuous batching is the single biggest throughput improvement in modern LLM serving.

Pratik Dhanave · ·4 min read

Choosing an Agent Framework: MAF vs LangGraph vs ADK vs CrewAI

Four popular agent frameworks, four genuinely different philosophies — and the right choice is decided less by features than by how much control you want, how your team thinks, and what you're actually building.

Four popular agent frameworks, four genuinely different philosophies — the right choice is decided less by features than by how much control you want, how your team thinks, and what you're building.

Pratik Dhanave · ·5 min read

Caching: Not Paying Twice for the Same Work

A large share of what an AI system processes is repeated — the same system prompt, the same documents, the same questions — and caching is how you stop paying full price for work you have already done.

A large share of what an AI system processes is repeated — the same system prompt, the same documents, the same questions. Caching is how you stop paying full price for work you have already done.

Pratik Dhanave · ·5 min read

Memory and Conversation History

A conversation that never forgets eventually overflows, so managing what history an agent carries forward — and how it remembers across sessions — is one of the defining problems of context engineering.

A conversation that never forgets eventually overflows, so managing what history an agent carries forward — and how it remembers across sessions — is a defining problem of context engineering.

Pratik Dhanave · ·6 min read

LLM-as-a-Judge

When there's no reference answer and no rule to check, you can ask a strong model to grade the output — LLM-as-a-judge. It's the technique that made open-ended evaluation practical at scale, and it's also a minefield: judges have systematic biases, can be gamed, and agree with humans only when you design the grading carefully. Used well it's indispensable; used naively it produces confident numbers that mean nothing.

When there's no reference and no rule to check, you can ask a strong model to grade the output — LLM-as-a-judge. It made open-ended evaluation practical at scale, and it's a minefield: judges have systematic biases (position, verbosity, self-preference), can be gamed, and agree with humans only when you design the grading carefully.

Pratik Dhanave · ·5 min read

What Is LangChain?

LangChain is the framework everyone starts with and everyone has opinions about — a vast toolkit for building LLM applications whose real value isn't any one feature but the standardization it brings: one interface across every model, vector store, and tool, so you write your application once and swap the pieces underneath. Understanding what it is (and its relationship to LangGraph) cuts through the confusion.

LangChain is the framework everyone starts with and everyone has opinions about — a vast toolkit whose real value is the standardization it brings: one interface across every model, vector store, and tool, so you write your application once and swap the pieces underneath.

Pratik Dhanave · ·5 min read

What Is Pydantic AI?

Most agent frameworks treat the LLM's output as text you hope to parse. Pydantic AI treats it as typed, validated data — bringing the discipline that made Pydantic the backbone of Python data validation to the messy world of LLM agents. If you've ever wished your agent's output was a real typed object instead of a string you cross your fingers over, this framework was built for you.

Most agent frameworks treat the LLM's output as text you hope to parse. Pydantic AI treats it as typed, validated data — bringing the discipline that made Pydantic the backbone of Python data validation to the messy world of LLM agents.

Pratik Dhanave · ·6 min read

The Fine-Tuning Spectrum

"Fine-tuning" is not one thing — it's a family of techniques that range from cheaply nudging a model's output format to expensively rebuilding its knowledge base. Confusing them leads to using a sledgehammer for a thumbtack. Knowing where your task sits on the spectrum tells you which technique, how much data, and how much compute you actually need.

'Fine-tuning' is not one thing — it's a family from cheaply nudging output format to expensively rebuilding a model's knowledge. Knowing where your task sits on the spectrum tells you which technique, how much data, and how much compute you actually need.

Pratik Dhanave · ·6 min read

The KV Cache

The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in LLM serving, from how many users you can batch to why long contexts cost so much, traces back to this one data structure.

The KV cache is the optimization that makes LLM generation fast enough to be practical — and the memory hog that makes it expensive. Almost every hard problem in serving traces back to this one data structure.

Pratik Dhanave · ·5 min read

How to Make AI Architecture Decisions

Most AI architecture debates are settled by hype, familiarity, or whoever spoke last — this series settles them by requirements and trade-offs, starting with the meta-framework that every specific decision reduces to.

Most AI architecture debates are settled by hype or familiarity; this series settles them by requirements and trade-offs, starting with the meta-framework every specific decision reduces to.

Pratik Dhanave · ·5 min read

Query Transformation

The user's question is written to be asked, not to be searched, so the first thing an agentic RAG system should do is turn that question into queries that actually retrieve well.

The user's question is written to be asked, not searched, so the first thing an agentic RAG system should do is turn that question into queries that actually retrieve well.

Pratik Dhanave · ·5 min read

Prompt and Context Optimization

Since you pay for every input token, the fastest way to cut the cost of a call without changing the model is to send fewer tokens — and most production prompts are carrying far more than they need.

Since you pay for every input token, the fastest way to cut the cost of a call without changing the model is to send fewer tokens — and most production prompts carry far more than they need.

Pratik Dhanave · ·5 min read

Retrieval: Bringing in the Right Context

Retrieval is how you put external, current knowledge into a model's window, and doing it well is a context-engineering problem — the goal is not the most documents, but the right ones.

Retrieval is how you put external, current knowledge into a model's window, and doing it well is a context-engineering problem — the goal is not the most documents, but the right ones.

Pratik Dhanave · ·6 min read

What to Measure: A Taxonomy of Metrics

Before you can score an LLM, you have to decide what "good" even means for your task — and that choice determines everything downstream. Metrics fall into a few families, from exact string matching to reference overlap to semantic similarity to task-specific checks, each measuring something different and each with blind spots. Picking the wrong metric is worse than no metric: it gives you a confident number that points the wrong way.

Before you can score an LLM you must decide what "good" means — and that choice determines everything. Metrics fall into families (exact match, reference overlap, semantic similarity, task-specific), each measuring something different with different blind spots. Picking the wrong metric is worse than none: it points confidently the wrong way.

Pratik Dhanave · ·6 min read

What Fine-Tuning Is and When to Use It

Fine-tuning is the most misunderstood tool in the LLM toolkit. Reach for it to teach a model new facts and you'll waste weeks and get worse results than a day of RAG. Reach for it to change how a model behaves — its format, tone, or a narrow skill — and nothing else comes close. The whole discipline starts with knowing which problem you actually have.

Fine-tuning is the most misunderstood tool in the LLM toolkit. Reach for it to teach a model facts and you'll waste weeks; reach for it to change how a model behaves — format, tone, a narrow skill — and nothing else comes close. The discipline starts with knowing which problem you have.

Pratik Dhanave · ·6 min read

How LLM Inference Works

Running an LLM is not one computation — it's two very different ones stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization that follows makes sense only once you see that inference has these two phases with opposite bottlenecks.

Running an LLM is two very different computations stitched together: a compute-heavy pass over your prompt, then a long, memory-bound slog generating one token at a time. Almost every serving optimization makes sense only once you see these two phases.

Pratik Dhanave · ·5 min read

What Is LlamaIndex?

LlamaIndex began as the fastest way to build RAG and has grown into a full data framework for LLM applications — connect your data, index it, retrieve it, and reason over it, with agents and workflows on top.

LlamaIndex began as the fastest way to build RAG and has grown into a full data framework for LLM applications — connect your data, index it, retrieve it, and reason over it, with agents and workflows on top.

Pratik Dhanave · ·5 min read

Model Selection and Routing

The single biggest cost lever in most AI systems is not clever prompting — it is not using an expensive model for work a cheap one would do just as well.

The single biggest cost lever in most AI systems is not clever prompting — it is not using an expensive model for work a cheap one would do just as well. Right-sizing and routing models to tasks.

Pratik Dhanave · ·14 min read

AI-Assisted Code Review

The series finale — where LLM reviewers genuinely help on every pull request, where they quietly fail, and how to build a human-plus-AI workflow that speeds review up without letting judgment or accountability leak away.

The capstone: using AI/LLM reviewers well and where humans stay essential — AI as a tireless first pass that clears the noise, its limits (design judgment, contextual security, accountability, hallucinations), and a healthy human+AI division of labor.

Pratik Dhanave · ·6 min read

Why Evaluation Is the Bottleneck

Building something with an LLM is easy for a weekend and hard for a year. The wall almost everyone hits is not the model, the prompt, or the framework — it is knowing whether a change made things better or worse. Without a way to measure quality, every improvement is a guess and every deploy is a gamble. Evaluation is the discipline that turns "it seems better" into "it is better, by this much," and it is the real bottleneck in shipping AI systems.

Building with an LLM is easy for a weekend and hard for a year, and the wall almost everyone hits is knowing whether a change made things better or worse. Without measurement every improvement is a guess — evaluation is the discipline that turns "it seems better" into "it is better, by this much," and it's the real bottleneck in shipping AI.

Pratik Dhanave · ·5 min read

Why Naive RAG Falls Short

The retrieve-then-generate pipeline that launched a thousand demos hits a wall on real questions, and understanding exactly where it breaks is the case for making retrieval agentic.

The retrieve-then-generate pipeline that launched a thousand demos hits a wall on real questions. Understanding exactly where naive RAG breaks is the case for making retrieval agentic.

Pratik Dhanave · ·5 min read

Understanding the Token Economy

You cannot optimize what you cannot price, and pricing an AI system starts with understanding the token — what counts as one, why input and output cost differently, and how to compute the true cost of a request.

You cannot optimize what you cannot price. Pricing an AI system starts with the token — what counts as one, why input and output cost differently, and how to compute the true cost of a request.

Pratik Dhanave · ·5 min read

The Context Window as a Budget

Every token in the window costs money, adds latency, and competes for the model's attention, so the first skill of context engineering is treating context as a scarce budget to be spent deliberately.

Every token in the window costs money, adds latency, and competes for attention. The first skill of context engineering is treating context as a scarce budget to spend deliberately.

Pratik Dhanave · ·12 min read

Measuring and Scoring Red-Team Results

Turning red-team attacks into metrics you can act on and track over time — attack success rate, coverage, severity, and trend — plus the honest limits of what any of those numbers can tell you.

Turning attacks into metrics: attack success rate and why it's subtle, scoring success (rule/classifier/LLM-judge with its biases), coverage across the taxonomy, severity weighting, tracking trends per model/prompt version, and honest reporting of residual risk.

Pratik Dhanave · ·12 min read

Production on the NVIDIA Stack

Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.

Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.

Pratik Dhanave · ·5 min read

Why AI Costs Explode

An LLM demo costs almost nothing, so teams ship without a cost model — and then production traffic turns a rounding error into the biggest line item on the bill.

An LLM demo costs almost nothing, so teams ship without a cost model — then production traffic turns a rounding error into the biggest line item on the bill. Why AI costs explode, and how to see it coming.

Pratik Dhanave · ·5 min read

What Is Context Engineering?

Prompt engineering was about wording a single instruction well; context engineering is the broader discipline of deciding everything a model sees at inference time — and for agents, it is the discipline that matters most.

Prompt engineering was about wording one instruction well; context engineering is the broader discipline of deciding everything a model sees at inference — and for agents it is the skill that matters most.

Pratik Dhanave · ·13 min read

Automated Red-Teaming and Tooling

Scaling red-teaming beyond manual probing — the building blocks of an automated harness (seed library, mutation, orchestrator, scorer), LLM-driven adaptive attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and wiring it all into CI as a repeatable gate.

Scaling red-teaming: the harness building blocks (attack seeds, mutation, orchestrator, scorer), adaptive LLM-driven attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and integrating an automated red-team gate into CI.

Pratik Dhanave · ·15 min read

Self-Hosting and Optimizing Inference

When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.

Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.

Pratik Dhanave · ·6 min read

What Is the Model Context Protocol?

A model is only as useful as the context and tools it can reach, and MCP is the open standard that lets any AI app plug into any tool through one interface instead of a hundred bespoke integrations.

MCP turns the M×N mess of wiring every AI app to every tool into M+N: wrap each system as a server once, make each app a client once, and any app can use any system.

Pratik Dhanave · ·6 min read

What Are Self-Evolving Agents?

Most agents are frozen the moment they ship, repeating the same mistakes forever, and self-evolving agents are the attempt to break that ceiling by letting the system improve itself as it runs.

Most agents are frozen at deployment and repeat their mistakes forever. Self-evolving agents route their own experience back into their own behavior — here are the axes of change and the loop underneath them.

Pratik Dhanave · ·13 min read

Monitoring and Drift in Production

Governance doesn't stop at deploy. This is the NIST RMF MANAGE function in practice: what to monitor for an LLM system, how to detect the drift — including the silent kind where a provider swaps the model under you — and why the audit trail you log is the regulatory deliverable, not a debugging convenience.

Governance doesn't stop at deploy — the Manage function in production: what to monitor (operational, quality, safety), detecting drift including silent provider-model drift behind moving aliases, and the audit trail (versions + retrieved context, PII-redacted) that becomes regulatory evidence.

Pratik Dhanave · ·15 min read

Production AI

The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.

The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.

Pratik Dhanave · ·8 min read

Limits and the Frontier

Reasoning models are a genuine advance — and it's precisely because they're so impressive that their limits are worth stating plainly. A model that writes out careful, step-by-step reasoning invites you to trust the reasoning, and that trust is not always warranted. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve. Knowing the edges is part of using the capability wisely.

Reasoning models are a genuine advance — and precisely because they're so impressive, their limits are worth stating plainly. The chain of thought may not be why the model actually answered. More thinking eventually stops helping. And some problems no amount of test-time compute will solve.

Pratik Dhanave · ·14 min read

Bias, Fairness, and Explainability

The three trustworthy-AI properties regulators and users press on hardest — where bias enters a system, why the fairness definitions contradict each other so you must choose one deliberately, and why an explanation you can read is not the same as an explanation you can trust.

The trustworthy-AI properties regulators care about: where bias enters, why fairness notions conflict (you must choose one), disaggregated evaluation, mitigation with Fairlearn/AIF360, and explainability (SHAP/LIME) — with the honest caveat that LLM rationales are not faithful explanations.

Pratik Dhanave · ·12 min read

Insecure Output Handling and Excessive Agency

Two tightly-linked OWASP LLM risks that turn a clever prompt injection into real-world damage — and the Python patterns that shrink the blast radius: treat model output as untrusted input, and give agents the least agency they can get away with.

Two OWASP risks that turn an injection into damage: insecure output handling (model output is untrusted input — never eval/shell/SQL it unescaped) and excessive agency (least-privilege tools, allow-lists, human approval for irreversible actions, audit logs).

Pratik Dhanave · ·14 min read

Inference Optimization

Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.

Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.

Pratik Dhanave · ·7 min read

Using Reasoning Models Well

The habits that made you good at prompting standard models can actively hurt you with reasoning models. "Let's think step by step" is redundant — even counterproductive — when the model already thinks natively. Few-shot examples can constrain reasoning that's better left free. The shift from standard to reasoning models isn't just picking a different model; it's unlearning some prompting reflexes and learning where deep thinking actually earns its cost.

The habits that made you good at prompting standard models can actively hurt you with reasoning models. 'Let's think step by step' is redundant when the model already thinks natively. The shift isn't just picking a different model — it's unlearning some prompting reflexes and learning where deep thinking earns its cost.

Pratik Dhanave · ·14 min read

Evaluation and Quality Gates

Governance is only as real as your ability to measure it. This is the MEASURE function of an AI risk program made concrete — a versioned eval set, the metric families that matter for an LLM system, and a CI gate that fails the build when quality regresses instead of just logging a warning.

Governance enforced through evaluation — the Measure function made real: build a versioned eval set, pick the metric families (quality, faithfulness, safety, bias, PII, cost/latency), and turn eval into a CI quality gate that fails the build on regression.

Pratik Dhanave · ·13 min read

Data Security and Privacy

Part three of the AI Security Engineering series: protecting the data that flows through an LLM system — how sensitive information leaks out of prompts, logs, and retrieval, and the engineering controls (redaction, data minimization, per-user retrieval authz, residency choices) that actually stop it.

Protecting data in LLM systems: sensitive-information disclosure, PII in prompts and logs (your observability can be the leak), redaction with Presidio, data residency, and per-user access control on the retrieval layer so RAG doesn't leak across tenants.

Pratik Dhanave · ·13 min read

Evaluating AI Systems

How to know whether an LLM system actually works — building an eval dataset, the four metric families (deterministic checks, text overlap, embedding similarity, LLM-as-judge) in Go, task-specific eval for RAG and classification, and wiring a scored regression gate into CI so you measure instead of vibe.

How to know whether an LLM system works when outputs are non-deterministic: build an eval dataset, score with deterministic checks, embedding similarity, and LLM-as-judge (with its biases), evaluate RAG and classification, and gate regressions in CI.

Pratik Dhanave · ·7 min read

The Economics of Thinking

Test-time compute reframes a question engineers rarely had to ask before: how much is a correct answer worth? Because thinking now costs money and time in direct proportion to how much of it you do, reasoning is no longer free — it's a purchase. A reasoning model can generate many times more tokens working through a problem than a standard model uses to answer it, and you pay for every one. Deciding when that's worth it is the core practical skill of the reasoning era.

Test-time compute reframes a question engineers rarely had to ask: how much is a correct answer worth? Thinking now costs money and time in direct proportion to how much of it you do. A reasoning model can generate many times more tokens than a standard model — and you pay for every one.

Pratik Dhanave · ·14 min read

Prompt Injection and Jailbreaks

Why the #1 risk on the OWASP LLM Top 10 has no clean fix — the model can't tell your instructions apart from the text it reads — and the defense-in-depth pattern that actually shrinks the blast radius.

The #1 LLM risk: direct vs indirect prompt injection (the latter is the real threat for RAG and agents), jailbreak families, and honestly-rated defense-in-depth — delimiting untrusted data, least-privilege tools, human-in-the-loop, and detection's limits.

Pratik Dhanave · ·7 min read

Inference-Time Techniques: Spending Compute for Accuracy

A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model, reasoning or not, to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps, pruning the bad branches. These techniques share one shape — do more work at inference, then choose well — and understanding them turns test-time compute from a model property into a toolkit you control.

A reasoning model thinks harder on its own — but you can spend test-time compute on top of any model to squeeze out more accuracy. Generate many answers and vote. Generate many and pick the best with a verifier. Search a tree of reasoning steps. These techniques share one shape: do more work, then choose well.

Pratik Dhanave · ·11 min read

What AI Red Teaming Is

The opening post of a hands-on series for builders: what it means to systematically stress-test an AI system — model, prompts, retrieval, tools, and guardrails — to surface its failures before adversaries or ordinary users do, how that differs from pentesting and robustness testing, and the frameworks and ethics that keep the work rigorous.

The opener to a defensive AI red-teaming series: stress-testing AI systems (models + prompts + RAG + tools + guardrails) to find failures before adversaries do — how it differs from pentesting, the frameworks (OWASP GenAI, MITRE ATLAS, NIST AI 100-2), and ethical rules of engagement.

Pratik Dhanave · ·14 min read

AI Risk Management with the NIST AI RMF

Turning the four functions of the NIST AI Risk Management Framework — GOVERN, MAP, MEASURE, MANAGE — into something an engineering team can actually run: a risk taxonomy for LLM apps, a versioned risk register in code, and the eval hooks that keep MEASURE honest.

Make the NIST AI Risk Management Framework actionable: the Govern/Map/Measure/Manage functions in your own words, a risk taxonomy for LLM apps, and a versioned risk register (YAML/Python) capturing per-use-case risks, likelihood/impact, mitigations, and owners.

Pratik Dhanave · ·11 min read

The AI Security Landscape

Why LLM and agent applications open a genuinely new attack surface, the mental models to reason about it (OWASP Top 10 for LLM Applications, MITRE ATLAS, NIST AI RMF), and how to threat-model an AI system before you write a line of defensive code.

The opener to a series on securing LLM and agent systems: why AI apps have a new attack surface (instructions and data share one channel, tools grant real power), the OWASP Top 10 for LLM Applications, MITRE ATLAS, and how to threat-model an AI system.

Pratik Dhanave · ·10 min read

Calling watsonx.ai from Python

Your first real inference calls with the ibm-watsonx-ai SDK — credentials, ModelInference, the generate and chat APIs, streaming, token usage, error handling, and the langchain-ibm path.

Make your first watsonx.ai calls in Python: credentials (IAM key + project_id + regional url), ModelInference with generate_text vs the modern chat API, decoding methods and parameters, streaming, token usage, and the langchain-ibm ChatWatsonx path.

Pratik Dhanave · ·10 min read

Calling NIM from Python

Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.

Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.

Pratik Dhanave · ·9 min read

Streaming and Token Usage

How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.

Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.

Pratik Dhanave · ·13 min read

Response Evaluation: Match, ROUGE, and LLM-as-Judge

How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating Agents in Go.

How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating...

Pratik Dhanave · ·8 min read

How Reasoning Models Are Trained

You can't teach deep reasoning by showing a model more examples of good reasoning — because the best reasoning for a hard problem often isn't in any dataset, and imitation caps a model at the quality of what it imitates. The breakthrough behind modern reasoning models was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers. That shift — from imitation to reinforcement on verifiable outcomes — is why reasoning models can think in ways no one wrote down.

You can't teach deep reasoning by showing more examples — imitation caps a model at what it imitates. The breakthrough was to stop imitating and start rewarding: let the model try to solve problems, check whether it got them right, and reinforce whatever thinking led to correct answers.

Pratik Dhanave · ·12 min read

What AI Governance Is

A working definition of AI governance for engineers — what it means, how it differs from security and compliance, why you already own a slice of it, and the frameworks and lifecycle map that anchor the rest of this series.

The opener to a governance series for engineers: what AI governance is and how it differs from security and compliance, why engineers produce the evidence (docs, evals, monitors, audit trails), and the frameworks used throughout — NIST AI RMF, the EU AI Act, and ISO/IEC 42001.

Pratik Dhanave · ·9 min read

The IBM watsonx Platform

A Python engineer's map of IBM watsonx — what watsonx.ai, watsonx.governance, watsonx.data and watsonx Orchestrate actually are, why enterprises pick them, and the smallest amount of `ibm-watsonx-ai` code that gets a foundation model answering you.

The opener to a series on building LLM and agent applications on IBM watsonx from Python: how watsonx.ai (Granite + third-party models), watsonx.governance, watsonx.data and Orchestrate fit together, and why the ibm-watsonx-ai SDK and langchain-ibm make it Python-native.

Pratik Dhanave · ·8 min read

The NVIDIA AI Stack

A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.

The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.

Pratik Dhanave · ·11 min read

Calling a Model with the Converse API

Your first real inference call in Go against Amazon Bedrock — using the unified, model-agnostic Converse API and the AWS SDK for Go v2, from client construction to reading tokens back off the response.

Your first real inference call on Bedrock in Go via the unified Converse API: building the client, the ConverseInput message/content-block union, extracting the assistant text, and reading stop reason and token usage — with the content-block union explained.

Pratik Dhanave · ·7 min read

Test-Time Compute: The New Scaling Axis

The dominant story of AI progress for years was training-time scale: bigger models, more data, more training compute. Test-time compute is a second, independent axis — spend more computation when you run the model, not when you train it, and get better answers on hard problems. It reframes a trained model not as a fixed-capability artifact but as one whose performance you can dial up per query by letting it think more.

The dominant story of AI progress was training-time scale: bigger models, more data. Test-time compute is a second, independent axis — spend more computation when you run the model, not when you train it, and get better answers on hard problems. It reframes a model as one whose performance you can dial up per query.

Pratik Dhanave · ·9 min read

What Amazon Bedrock Is

The opener for a Go series on building LLM and agent applications with Amazon Bedrock — what the service actually is, why it sits between your Go code and a dozen foundation models, and which aws-sdk-go-v2 packages you will lean on for the rest of the way.

The opener to a series on building LLM and agent applications on Amazon Bedrock in Go: what Bedrock actually is, what it adds over calling a provider API directly (one API across models, IAM auth, data residency), and the aws-sdk-go-v2 packages you'll use.

Pratik Dhanave · ·7 min read

Chain of Thought: The Idea That Started It

The observation that launched reasoning models was almost embarrassingly simple: if you ask a language model to "think step by step" before answering, it gets dramatically better at hard problems — with no change to the model at all. That a few words of prompting could unlock reasoning already latent in the model was a clue about something deep: the ability to reason was there, waiting to be elicited, and giving the model room to work was the key.

The observation that launched reasoning models was almost embarrassingly simple: ask a model to 'think step by step' and it gets dramatically better at hard problems, with no change to the model. That a few words could unlock latent reasoning was a clue about something deep.

Pratik Dhanave · ·7 min read

What Reasoning Models Are

For years, the way to make a language model better was to make it bigger — more parameters, more training data. Reasoning models represent a different bet: instead of only scaling what the model knows, scale how much it thinks before answering. A reasoning model spends extra computation at inference time working through a problem step by step before committing to an answer — and on hard problems, that changes what's possible.

For years, making a language model better meant making it bigger. Reasoning models represent a different bet: scale how much the model thinks before answering, not just what it knows. A reasoning model spends extra computation working through a problem step by step — and on hard problems, that changes what's possible.

Pratik Dhanave · ·15 min read

Prompt Engineering as Engineering

Treating the prompt as a real engineering artifact — grounded in how a next-token predictor actually works — with roles, specificity, few-shot examples, decomposition, chain-of-thought, grounding, temperature, injection defense, and versioned Go templates you can test.

Prompt engineering as a real engineering discipline: roles, specificity, few-shot, decomposition, chain-of-thought and its cost, grounding, and prompt injection — plus building prompts as versioned, testable Go text/template templates you treat like code.

Pratik Dhanave · ·13 min read

Structured Output and Tool Calling

From-scratch Go for the two mechanisms that turn an LLM from a text generator into a component you can wire into real software — schema-constrained JSON and function calling — both spoken over the same OpenAI-compatible chat JSON.

Getting reliable machine-readable output from an LLM in Go: structured output (json-schema mode, decode into a typed struct, validate with a bounded retry) and tool/function calling (the full round-trip loop, decoding tool arguments, returning results tied to the call id).

Pratik Dhanave · ·7 min read

LLM-Guided Portfolio Allocation

A hybrid allocator where a quantitative core owns the money, a language model only whispers tilts, and hard constraints plus a human bound everything before a single order goes out.

A hybrid allocator: quantitative signals set the baseline (mean-variance / risk parity) while an LLM proposes tilts from qualitative context, constrained by risk limits and human review.

Pratik Dhanave · ·11 min read

Calling an LLM from Go

Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.

Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.

Pratik Dhanave · ·12 min read

Tokens and Tokenization

The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.

The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.

Pratik Dhanave · ·12 min read

How LLMs Work, Enough to Build With Them

The working mental model an AI engineer needs — next-token prediction, attention, training, and sampling — without the transformer math, and with every fact tied back to a decision you make in code.

The working mental model an AI engineer needs — next-token prediction, attention at an intuition level, pretraining vs post-training, and sampling — with every fact tied back to a concrete decision you make in code.

Pratik Dhanave · ·12 min read

What AI Engineering Is

The opener to a from-scratch series on building applications on top of foundation models in Go — what AI engineering actually is, how it differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.

The opener to a from-scratch AI-engineering-in-Go series: what AI engineering actually is, how building on foundation models differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.

Pratik Dhanave · ·8 min read

Generative AI and LLMs in Finance: Patterns and Guardrails

Large language models earn their place in regulated finance as fast, well-supervised assistants — not as autonomous agents with a hand on the money.

Where LLMs fit in regulated finance: document analysis/extraction, retrieval-grounded Q&A, and agentic decision support — plus the guardrails (grounding, human-in-the-loop, output validation, audit trails) that keep them…

Pratik Dhanave · ·6 min read

Interpreting Deep Networks and LLMs

The hardest interpretability frontier is the models that matter most right now: deep neural networks and large language models, with billions of parameters and no obvious place to look for "the reason." The attribution methods that work on tabular models strain here, and a new field — mechanistic interpretability — is trying to reverse-engineer what these networks actually compute. This post covers how we peer inside deep models and LLMs, and why it's both crucial and genuinely unsolved.

The hardest frontier: deep nets and LLMs with billions of parameters and no obvious place to find 'the reason'. Saliency maps (fragile), attention (not explanation), probing, and mechanistic interpretability's attempt to reverse-engineer circuits — plus the honest truth that we largely cannot yet explain why frontier LLMs behave as they do.

Pratik Dhanave · ·5 min read

Multi-Agent Systems in ADK: Coordinators, sub_agents, and LLM-Driven Delegation

How one agent routes work to specialists — and why the description field is the most important string you write.

Agent hierarchies and LLM-driven delegation: sub_agents, how the description field drives auto-transfer, and coordinator/dispatcher patterns — contrasted with deterministic workflow agents.

Pratik Dhanave · ·8 min read

Vision-Language Models

The multimodal capability people actually experience — uploading a photo and asking a chatbot what's in it, or having it read a screenshot, explain a diagram, or extract data from a chart — comes from vision-language models: LLMs that can see. The clever part is how it's done. Rather than build a seeing-and-reasoning model from scratch, you take a language model that already reasons brilliantly and give it eyes, by connecting a vision encoder to it. Understanding how that connection works explains the multimodal AI most people use.

The multimodal capability people actually experience — uploading a photo and asking a chatbot what's in it — comes from vision-language models: LLMs that can see. The clever part is how it's done: take a language model that already reasons brilliantly and give it eyes by connecting a vision encoder to it.

Pratik Dhanave · ·9 min read

Building Reliable Agents

The demo is easy; the production agent is hard. An agent that works impressively in a demo can fail unpredictably in the real world — looping, hallucinating, taking wrong actions, or racking up huge costs — because the same flexibility that makes agents powerful makes them unreliable. Building agents that actually work in production is a discipline of managing that unreliability: adding guardrails, evaluating rigorously, and, most importantly, knowing when not to use an agent at all. This closing post is about that discipline.

The demo is easy; the production agent is hard. The same flexibility that makes agents powerful makes them unreliable — looping, hallucinating, taking wrong actions, racking up costs. Building agents that actually work is a discipline of managing that unreliability, and knowing when NOT to use an agent at all.

Pratik Dhanave · ·8 min read

Multi-Agent Patterns

The instinct, once single agents work, is to build teams of them — a researcher agent, a writer agent, a critic agent, all collaborating like a little organization. It's an appealing vision, and sometimes exactly right. But multi-agent systems are also where a lot of complexity and cost hides, and the honest guidance is more restrained than the hype: use multiple agents when the problem genuinely calls for it, and prefer a single well-designed agent when it doesn't. Understanding the multi-agent patterns — and their real tradeoffs — is what lets you make that call well.

The instinct, once single agents work, is to build teams of them — a researcher, a writer, a critic, collaborating like an organization. Sometimes that's right. But multi-agent systems are also where a lot of complexity and cost hides, and the honest guidance is restrained: use multiple agents when the problem genuinely calls for it, and prefer a single well-designed agent when it doesn't.

Pratik Dhanave · ·8 min read

Reflection and Self-Correction

The first output is rarely the best output — a truth as old as writing, and one that applies to agents too. An agent that acts once and moves on repeats its mistakes; an agent that looks back at what it did, judges whether it worked, and tries again can dramatically improve. Reflection — the agent evaluating and correcting its own work — is what turns a one-shot attempt into an iterative process that gets better, and it's one of the most powerful patterns for making agents reliable on hard tasks.

The first output is rarely the best output. An agent that acts once and moves on repeats its mistakes; an agent that looks back at what it did, judges whether it worked, and tries again can dramatically improve. Reflection turns a one-shot attempt into an iterative process that gets better.

Pratik Dhanave · ·8 min read

Agent Memory

An LLM is, fundamentally, stateless — it remembers nothing between calls except what you put in its context window. For an agent that takes many steps or works across many sessions, that's a serious problem: without memory, every step starts from scratch, and nothing is ever learned. Memory is how agents overcome statelessness — holding the context of the current task, and carrying knowledge across tasks and time. Understanding the kinds of agent memory, and their limits, is essential to building agents that can handle real, extended work.

An LLM is fundamentally stateless — it remembers nothing between calls except what you put in its context. For an agent that takes many steps or works across sessions, that's a serious problem. Memory is how agents overcome statelessness — holding the current task's context, and carrying knowledge across tasks and time.

Pratik Dhanave · ·8 min read

Planning and Decomposition

Ask an agent to "research this market and write a report" and it faces the same problem a person would: the task is too big to do in one leap. The answer, for agents as for people, is to break it down — decompose the goal into steps, and work through them. Planning is how agents handle complexity that the basic reason-act loop alone would fumble, and the patterns for doing it — from planning upfront to decomposing on the fly — are among the most important in agent design.

Ask an agent to 'research this market and write a report' and it faces the same problem a person would: the task is too big to do in one leap. The answer, for agents as for people, is to break it down. Planning is how agents handle complexity that the basic reason-act loop alone would fumble.

Pratik Dhanave · ·9 min read

Tool Use

An LLM on its own can only do one thing: generate text. It can't search the web, run code, query a database, check the current time, or send a message — it can only produce words. Tools are what break that confinement, turning a model that can only talk into an agent that can act. Tool use is arguably the single most important capability that makes agents possible, and understanding how it works — and how to design tools well — is central to building effective agents.

An LLM on its own can only do one thing: generate text. Tools are what break that confinement, turning a model that can only talk into an agent that can act. Tool use is arguably the single most important capability that makes agents possible.

Pratik Dhanave · ·8 min read

The Core Agent Loop

Strip away the frameworks, the tooling, and the jargon, and every LLM agent reduces to one simple loop: think about what to do, do it, look at what happened, repeat. This reason-act-observe cycle — crystallized by the ReAct pattern — is the beating heart of every agent, and understanding it deeply is understanding agents themselves. Once you see the loop clearly, agent frameworks stop being mysterious: they're all just implementations of this same fundamental cycle.

Strip away the frameworks and every LLM agent reduces to one simple loop: think about what to do, do it, look at what happened, repeat. This reason-act-observe cycle — crystallized by the ReAct pattern — is the beating heart of every agent, and understanding it deeply is understanding agents themselves.

Pratik Dhanave · ·9 min read

What an AI Agent Is

"Agent" has become one of the most overused and least precise words in AI — applied to everything from a chatbot with a system prompt to a fully autonomous system that writes and ships code. Cutting through the hype requires a clear definition: an agent is a system where an LLM decides its own actions in a loop, using tools, until a goal is met. That one distinction — the model choosing what to do next, rather than following a fixed script — is what separates a genuine agent from a workflow, and it's where both the power and the difficulty come from.

'Agent' has become one of the most overused words in AI. Cutting through the hype requires a clear definition: an agent is a system where an LLM decides its own actions in a loop, using tools, until a goal is met. That one distinction — the model choosing what to do next, rather than following a fixed script — is where both the power and the difficulty come from.

All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse the blog or explore related topics using the tag cloud above.