Articles about LLMs — exploring patterns, best practices, and real-world implementations in production systems.
45 posts tagged with llms. ← All posts
The series finale — where LLM reviewers genuinely help on every pull request, where they quietly fail, and how to build a human-plus-AI workflow that speeds review up without letting judgment or accountability leak away.
The capstone: using AI/LLM reviewers well and where humans stay essential — AI as a tireless first pass that clears the noise, its limits (design judgment, contextual security, accountability, hallucinations), and a healthy human+AI division of labor.
Turning red-team attacks into metrics you can act on and track over time — attack success rate, coverage, severity, and trend — plus the honest limits of what any of those numbers can tell you.
Turning attacks into metrics: attack success rate and why it's subtle, scoring success (rule/classifier/LLM-judge with its biases), coverage across the taxonomy, severity weighting, tracking trends per model/prompt version, and honest reporting of residual risk.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
Scaling red-teaming beyond manual probing — the building blocks of an automated harness (seed library, mutation, orchestrator, scorer), LLM-driven adaptive attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and wiring it all into CI as a repeatable gate.
Scaling red-teaming: the harness building blocks (attack seeds, mutation, orchestrator, scorer), adaptive LLM-driven attackers, the real tools by role (PyRIT, garak, promptfoo, Giskard), and integrating an automated red-team gate into CI.
When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.
Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.
Governance doesn't stop at deploy. This is the NIST RMF MANAGE function in practice: what to monitor for an LLM system, how to detect the drift — including the silent kind where a provider swaps the model under you — and why the audit trail you log is the regulatory deliverable, not a debugging convenience.
Governance doesn't stop at deploy — the Manage function in production: what to monitor (operational, quality, safety), detecting drift including silent provider-model drift behind moving aliases, and the audit trail (versions + retrieved context, PII-redacted) that becomes regulatory evidence.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
The three trustworthy-AI properties regulators and users press on hardest — where bias enters a system, why the fairness definitions contradict each other so you must choose one deliberately, and why an explanation you can read is not the same as an explanation you can trust.
The trustworthy-AI properties regulators care about: where bias enters, why fairness notions conflict (you must choose one), disaggregated evaluation, mitigation with Fairlearn/AIF360, and explainability (SHAP/LIME) — with the honest caveat that LLM rationales are not faithful explanations.
Two tightly-linked OWASP LLM risks that turn a clever prompt injection into real-world damage — and the Python patterns that shrink the blast radius: treat model output as untrusted input, and give agents the least agency they can get away with.
Two OWASP risks that turn an injection into damage: insecure output handling (model output is untrusted input — never eval/shell/SQL it unescaped) and excessive agency (least-privilege tools, allow-lists, human approval for irreversible actions, audit logs).
Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.
Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.
Governance is only as real as your ability to measure it. This is the MEASURE function of an AI risk program made concrete — a versioned eval set, the metric families that matter for an LLM system, and a CI gate that fails the build when quality regresses instead of just logging a warning.
Governance enforced through evaluation — the Measure function made real: build a versioned eval set, pick the metric families (quality, faithfulness, safety, bias, PII, cost/latency), and turn eval into a CI quality gate that fails the build on regression.
Part three of the AI Security Engineering series: protecting the data that flows through an LLM system — how sensitive information leaks out of prompts, logs, and retrieval, and the engineering controls (redaction, data minimization, per-user retrieval authz, residency choices) that actually stop it.
Protecting data in LLM systems: sensitive-information disclosure, PII in prompts and logs (your observability can be the leak), redaction with Presidio, data residency, and per-user access control on the retrieval layer so RAG doesn't leak across tenants.
How to know whether an LLM system actually works — building an eval dataset, the four metric families (deterministic checks, text overlap, embedding similarity, LLM-as-judge) in Go, task-specific eval for RAG and classification, and wiring a scored regression gate into CI so you measure instead of vibe.
How to know whether an LLM system works when outputs are non-deterministic: build an eval dataset, score with deterministic checks, embedding similarity, and LLM-as-judge (with its biases), evaluate RAG and classification, and gate regressions in CI.
Why the #1 risk on the OWASP LLM Top 10 has no clean fix — the model can't tell your instructions apart from the text it reads — and the defense-in-depth pattern that actually shrinks the blast radius.
The #1 LLM risk: direct vs indirect prompt injection (the latter is the real threat for RAG and agents), jailbreak families, and honestly-rated defense-in-depth — delimiting untrusted data, least-privilege tools, human-in-the-loop, and detection's limits.
The opening post of a hands-on series for builders: what it means to systematically stress-test an AI system — model, prompts, retrieval, tools, and guardrails — to surface its failures before adversaries or ordinary users do, how that differs from pentesting and robustness testing, and the frameworks and ethics that keep the work rigorous.
The opener to a defensive AI red-teaming series: stress-testing AI systems (models + prompts + RAG + tools + guardrails) to find failures before adversaries do — how it differs from pentesting, the frameworks (OWASP GenAI, MITRE ATLAS, NIST AI 100-2), and ethical rules of engagement.
Turning the four functions of the NIST AI Risk Management Framework — GOVERN, MAP, MEASURE, MANAGE — into something an engineering team can actually run: a risk taxonomy for LLM apps, a versioned risk register in code, and the eval hooks that keep MEASURE honest.
Make the NIST AI Risk Management Framework actionable: the Govern/Map/Measure/Manage functions in your own words, a risk taxonomy for LLM apps, and a versioned risk register (YAML/Python) capturing per-use-case risks, likelihood/impact, mitigations, and owners.
Why LLM and agent applications open a genuinely new attack surface, the mental models to reason about it (OWASP Top 10 for LLM Applications, MITRE ATLAS, NIST AI RMF), and how to threat-model an AI system before you write a line of defensive code.
The opener to a series on securing LLM and agent systems: why AI apps have a new attack surface (instructions and data share one channel, tools grant real power), the OWASP Top 10 for LLM Applications, MITRE ATLAS, and how to threat-model an AI system.
Your first real inference calls with the ibm-watsonx-ai SDK — credentials, ModelInference, the generate and chat APIs, streaming, token usage, error handling, and the langchain-ibm path.
Make your first watsonx.ai calls in Python: credentials (IAM key + project_id + regional url), ModelInference with generate_text vs the modern chat API, decoding methods and parameters, streaming, token usage, and the langchain-ibm ChatWatsonx path.
Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.
Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.
How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.
Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating Agents in Go.
How to score an agent's final answer against a reference — from exact string match, through ROUGE-1 unigram overlap, to an LLM judge — with original Go you can drop into a test suite. Part 5 of Evaluating...
A working definition of AI governance for engineers — what it means, how it differs from security and compliance, why you already own a slice of it, and the frameworks and lifecycle map that anchor the rest of this series.
The opener to a governance series for engineers: what AI governance is and how it differs from security and compliance, why engineers produce the evidence (docs, evals, monitors, audit trails), and the frameworks used throughout — NIST AI RMF, the EU AI Act, and ISO/IEC 42001.
A Python engineer's map of IBM watsonx — what watsonx.ai, watsonx.governance, watsonx.data and watsonx Orchestrate actually are, why enterprises pick them, and the smallest amount of `ibm-watsonx-ai` code that gets a foundation model answering you.
The opener to a series on building LLM and agent applications on IBM watsonx from Python: how watsonx.ai (Granite + third-party models), watsonx.governance, watsonx.data and Orchestrate fit together, and why the ibm-watsonx-ai SDK and langchain-ibm make it Python-native.
A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.
The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.
Your first real inference call in Go against Amazon Bedrock — using the unified, model-agnostic Converse API and the AWS SDK for Go v2, from client construction to reading tokens back off the response.
Your first real inference call on Bedrock in Go via the unified Converse API: building the client, the ConverseInput message/content-block union, extracting the assistant text, and reading stop reason and token usage — with the content-block union explained.
The opener for a Go series on building LLM and agent applications with Amazon Bedrock — what the service actually is, why it sits between your Go code and a dozen foundation models, and which aws-sdk-go-v2 packages you will lean on for the rest of the way.
The opener to a series on building LLM and agent applications on Amazon Bedrock in Go: what Bedrock actually is, what it adds over calling a provider API directly (one API across models, IAM auth, data residency), and the aws-sdk-go-v2 packages you'll use.
Treating the prompt as a real engineering artifact — grounded in how a next-token predictor actually works — with roles, specificity, few-shot examples, decomposition, chain-of-thought, grounding, temperature, injection defense, and versioned Go templates you can test.
Prompt engineering as a real engineering discipline: roles, specificity, few-shot, decomposition, chain-of-thought and its cost, grounding, and prompt injection — plus building prompts as versioned, testable Go text/template templates you treat like code.
From-scratch Go for the two mechanisms that turn an LLM from a text generator into a component you can wire into real software — schema-constrained JSON and function calling — both spoken over the same OpenAI-compatible chat JSON.
Getting reliable machine-readable output from an LLM in Go: structured output (json-schema mode, decode into a typed struct, validate with a bounded retry) and tool/function calling (the full round-trip loop, decoding tool arguments, returning results tied to the call id).
A hybrid allocator where a quantitative core owns the money, a language model only whispers tilts, and hard constraints plus a human bound everything before a single order goes out.
A hybrid allocator: quantitative signals set the baseline (mean-variance / risk parity) while an LLM proposes tilts from qualitative context, constrained by risk limits and human review.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.
Turning a firehose of unstructured headlines into a disciplined, point-in-time trading signal.
Turning unstructured news into a tradable signal with LLMs: entity/event extraction, sentiment scoring, aggregation to a per-asset signal, latency and point-in-time constraints, and combining qualitative signals with a q…
The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.
The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.
The working mental model an AI engineer needs — next-token prediction, attention, training, and sampling — without the transformer math, and with every fact tied back to a decision you make in code.
The working mental model an AI engineer needs — next-token prediction, attention at an intuition level, pretraining vs post-training, and sampling — with every fact tied back to a concrete decision you make in code.
The opener to a from-scratch series on building applications on top of foundation models in Go — what AI engineering actually is, how it differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
The opener to a from-scratch AI-engineering-in-Go series: what AI engineering actually is, how building on foundation models differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
Large language models earn their place in regulated finance as fast, well-supervised assistants — not as autonomous agents with a hand on the money.
Where LLMs fit in regulated finance: document analysis/extraction, retrieval-grounded Q&A, and agentic decision support — plus the guardrails (grounding, human-in-the-loop, output validation, audit trails) that keep them…
How one agent routes work to specialists — and why the description field is the most important string you write.
Agent hierarchies and LLM-driven delegation: sub_agents, how the description field drives auto-transfer, and coordinator/dispatcher patterns — contrasted with deterministic workflow agents.
description, instruction, generation params, and structured output — the dials on almost every agent you'll build
The four knobs on almost every ADK agent: description (for delegation), instruction with {state} templating, generation params, and structured output — Python Pydantic model vs Go genai.Schema.
Agents return a clean 200 OK and still be wrong, unsafe, or expensive. Why agentic AI needs a new observability layer — LLM-as-judge, safety metrics, and the four lifecycle stages.
Single-turn evals check one decision. Multi-turn evals check the whole trajectory. A Python harness with three evaluators, an offline test suite, and the judge prompt that actually works.
PROVIDER=ollama, granite4.1:3b, zero API keys, no Azure account. How to make a multi-agent project that demonstrates enterprise patterns without requiring enterprise infrastructure.
How to port ADK's model hard-codes to Microsoft Agent Framework's provider factory pattern.
Zero-code LLM provider swaps across environments: Ollama for dev, OpenAI for staging, Azure Foundry for prod. Same agents, different models.
How to instrument multi-agent systems with OpenTelemetry, propagate trace context across an in-memory bus, and build a layered evaluation pipeline — from real-time policy gates to async LLM-as-judge to SLO-based trust scoring. Everything I learned building Genie.
Every LLM-backed agent in this platform has a deterministic rule-based fallback. The case always finalises. The fallback isn't a workaround — it's the contract.
PAN check-digit validation, Aadhaar offline KYC, DigiLocker, PEP/sanctions — all in Go code, not in a prompt. The LLM's job is to translate the verdict into something a human can read.
Not every query needs the production agent. A cost-aware dispatcher decides whether to route to the cheap-and-fast agent or the expensive-and-thorough one. Same UX, dramatically lower bill.
All posts on this site are written by Pratik Dhanave, an Agentic AI Architect with 7+ years building production distributed systems, multi-agent AI platforms, and cloud-native infrastructure. About the author → Each article includes working code, architecture diagrams, and references to the specific frameworks and standards discussed. Browse all posts or explore related topics using the tag cloud above.