Taking a watsonx.ai system from a notebook to production in Python — deployment spaces, reliability with retries and fallbacks, cost and throughput control, observability wired to watsonx.governance, and a hardening checklist.
Run a watsonx system in production from Python: IBM Cloud vs Cloud Pak for Data, project_id vs deployment spaces, reliability (tenacity retries, IAM token refresh, fallback), token-based cost, observability wired to watsonx.governance monitors, and securing IAM credentials.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
Why enterprises pick watsonx for its governance story, what watsonx.governance actually gives a Python team, and how to wire monitoring, factsheets, and generative-quality metrics into an LLM feature — honestly, as the platform-heavy work it is.
Governance is why many enterprises pick watsonx: model inventory, automated factsheets, monitoring (quality, drift, bias, and generative metrics like faithfulness), explainability, and EU AI Act-aligned risk workflows — with the ibm-watsonx-gov evaluation SDK and what to log.
When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.
Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.
Taking an Amazon Bedrock Go service from a working prototype to something you can run on-call — least-privilege IAM, credentials without static keys, tuning the SDK's built-in retryer, tracking token cost, and wiring up logging and metrics with aws-sdk-go-v2.
Taking a Bedrock Go service to production: least-privilege IAM and role-based credentials, tuning the SDK's built-in retryer for throttling, token-based cost tracking, and observability via model-invocation logging, structured metrics, and request IDs.
Adding safety to a watsonx app in Python with two complementary layers — the built-in HAP and PII moderations that watsonx.ai applies to generation, and IBM's dedicated Granite Guardian risk-detection models run as classifiers around your main model to catch harm, jailbreaks, and RAG hallucination before a response reaches a user.
Add safety to a watsonx app in Python with two layers: built-in HAP/PII moderations on generation, and Granite Guardian — IBM's dedicated risk models — run as a classifier to screen input and check RAG groundedness, wrapped in a guarded-generate flow.
Adding a safety layer to an NVIDIA-stack Python app with NeMo Guardrails — and why running it in-process, with no HTTP boundary, is the quiet advantage Python gives you over a separate guardrails server.
Add safety with NeMo Guardrails, in-process in Python: the rail types (input/output/dialog/retrieval), the config.yml + Colang model, loading with RailsConfig/LLMRails backed by a NIM model, and detecting when a rail trips instead of treating a refusal as a normal answer.
How to put Amazon Bedrock Guardrails in front of a model from Go — attaching one to a Converse call, screening raw text with ApplyGuardrail, and reading whether the guardrail actually intervened.
Guardrails for Amazon Bedrock in Go: content filters, denied topics, PII/sensitive-information filters, and contextual grounding — attaching a guardrail to a Converse call and screening arbitrary text with ApplyGuardrail, checking for intervention.
The last post in the series: what changes when the LLM system you built across posts 1-14 has to run for real — reliability, security, cost, observability, evaluation gates, and versioning, from a Go engineer's seat, with code where it earns its place.
The capstone: running an LLM system in production from a Go engineer's seat — reliability (timeouts, retries, fallbacks), security (injection, least-privilege tools, secrets), cost and observability, CI eval gates, and versioning models and prompts.
Assembling a full retrieval-augmented generation pipeline in Python on watsonx.ai — ingest and chunk documents, embed them with slate, retrieve by cosine, rerank for precision, then generate a grounded, cited answer with a Granite model, shown both from scratch and with langchain-ibm.
Build a full RAG pipeline in Python on watsonx: chunk (with Docling for parsing), embed and store, retrieve, rerank, and generate a grounded, cited answer with a Granite model — shown both from scratch and via the idiomatic langchain-ibm assembly, with notes on scaling to watsonx.data/Milvus.
Assembling a full retrieval-augmented generation pipeline in Python — chunk and embed with NeMo Retriever, shortlist by cosine, sharpen with a reranker, then ground a ChatNVIDIA answer that cites its sources — first from scratch, then the idiomatic LangChain wiring.
Build a full RAG pipeline in Python on NVIDIA's stack: chunk and embed with NeMo Retriever, shortlist by cosine, rerank, then generate a grounded, cited answer with ChatNVIDIA — shown both from scratch and the idiomatic LangChain assembly, with notes on scaling to a vector DB.
How to invoke a managed Agent for Amazon Bedrock from Go — where the server owns the reason-act loop, and your job is to call InvokeAgent, range the event stream, accumulate the answer chunks, and read the trace for observability.
Agents for Amazon Bedrock from Go: the managed reason-act loop that runs server-side (vs the DIY Converse loop), invoking an agent alias with InvokeAgent, streaming the response and trace events, and keeping multi-turn state with a SessionId.
Making an LLM system faster and cheaper without touching the weights — the levers an application engineer actually controls, from streaming and caching to token trimming, model routing, and Go's real superpower: concurrency with a rate limiter.
Make LLM systems faster and cheaper without retraining: TTFT vs throughput, exact and semantic response caching, prompt caching, token reduction, model routing and cascades, and Go concurrency with a worker pool and rate limiter.
Building RAG's retrieval core with watsonx.ai from Python — turning a corpus into vectors with IBM's slate embedding models, scoring a query against them, and then sharpening the shortlist with a reranking model so the LLM gets the right passages, not just plausible ones.
Use watsonx.ai's slate embedding models and reranking from Python to build RAG's retrieval core: embed_documents vs embed_query, numpy cosine scoring, and a two-stage retrieve-then-rerank pipeline — plus the langchain-ibm WatsonxEmbeddings/WatsonxRerank path.
Building RAG's retrieval core in Python — turning a corpus and a query into vectors with NeMo Retriever embedding NIMs, scoring by cosine similarity, then sharpening the shortlist with a cross-encoder reranker NIM.
Use NeMo Retriever from Python to build RAG's retrieval core: NVIDIAEmbeddings (embed_documents vs embed_query for the asymmetric passage/query distinction) with cosine scoring, and NVIDIARerank.compress_documents for a two-stage retrieve-then-rerank pipeline.
How to query a Knowledge Base for Amazon Bedrock from Go — the managed retrieve-then-read layer — using both the low-level Retrieve call and the one-shot RetrieveAndGenerate, with citations wired through.
RAG on Bedrock in Go with Knowledge Bases: the retrieve-then-read pattern via Retrieve, the one-shot managed path via RetrieveAndGenerate with citations, and when to reach for each — plus reading grounding so you keep RAG's trust benefit.
How to know whether an LLM system actually works — building an eval dataset, the four metric families (deterministic checks, text overlap, embedding similarity, LLM-as-judge) in Go, task-specific eval for RAG and classification, and wiring a scored regression gate into CI so you measure instead of vibe.
How to know whether an LLM system works when outputs are non-deterministic: build an eval dataset, score with deterministic checks, embedding similarity, and LLM-as-judge (with its biases), evaluate RAG and classification, and gate regressions in CI.
Letting a Granite model on watsonx.ai invoke your Python functions — the full request-and-response loop with the first-party `ibm-watsonx-ai` chat API, plus the shorter LangChain path with `ChatWatsonx.bind_tools`.
Function/tool calling against watsonx.ai Granite models in Python: the chat() round-trip loop (json.loads the arguments, echo tool_call_id, cap iterations), handling multiple tool calls, and the LangChain ChatWatsonx.bind_tools path — Granite models are tool-trained.
Driving function calling against NVIDIA NIM models from Python — the full request-execute-respond loop with the plain `openai` client, then the same thing automated by `ChatNVIDIA.bind_tools`.
Function/tool calling against NIM models in Python: the openai round-trip loop (json.loads the arguments string, echo tool_call_id, cap the iterations) and the LangChain ChatNVIDIA.bind_tools path — with the honest caveat that model support varies.
How to give an Amazon Bedrock model real Go functions — declaring tools, catching the tool-use stop reason, executing your code, and returning results — using the full round-trip loop in aws-sdk-go-v2.
Giving a Bedrock model tools in Go via the Converse API: declaring a ToolConfiguration, the ToolUse round-trip loop, echoing ToolUseId, returning tool results as a user message, and handling parallel tool calls.
Give the hand-rolled Go agent from post 11 a memory it can carry between turns and a plan it can follow across many steps — a compacting conversation buffer, retrieval over the post-8 vector store, and a plan-then-execute-then-reflect loop, all built from scratch.
Give the agent memory and planning in Go: a compacting short-term conversation buffer, long-term memory as timestamped embeddings in the vector store, and planning — plan-then-execute, reflection and re-planning when observations contradict the plan, and task decomposition.
Your first real inference calls with the ibm-watsonx-ai SDK — credentials, ModelInference, the generate and chat APIs, streaming, token usage, error handling, and the langchain-ibm path.
Make your first watsonx.ai calls in Python: credentials (IAM key + project_id + regional url), ModelInference with generate_text vs the modern chat API, decoding methods and parameters, streaming, token usage, and the langchain-ibm ChatWatsonx path.
Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.
Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.
How to stream Amazon Bedrock responses token-by-token with the aws-sdk-go-v2 Converse API, decode the event stream with a double type-switch, and account for tokens and cost from the metadata event — accurately, in Go.
Streaming responses and accounting for tokens and cost on Bedrock in Go: ranging the ConverseStream event stream, the nested delta unions, checking stream.Err(), and computing cost from the metadata usage event with a formula you fill in.
Building a real agent loop in Go by hand — an LLM in a loop that picks tools, runs them, reads the results, and repeats until the task is done — so you can see there is no magic behind LangGraph, MAF, or ADK.
Build a minimal but real agent loop in Go by hand: an Agent with a tool registry and a reason-act Run loop, an iteration budget, validation against hallucinated tools, feeding tool errors back as observations, and parallel tool calls — the loop frameworks formalize, demystified.
A Python engineer's map of IBM watsonx — what watsonx.ai, watsonx.governance, watsonx.data and watsonx Orchestrate actually are, why enterprises pick them, and the smallest amount of `ibm-watsonx-ai` code that gets a foundation model answering you.
The opener to a series on building LLM and agent applications on IBM watsonx from Python: how watsonx.ai (Granite + third-party models), watsonx.governance, watsonx.data and Orchestrate fit together, and why the ibm-watsonx-ai SDK and langchain-ibm make it Python-native.
A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.
The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.
Your first real inference call in Go against Amazon Bedrock — using the unified, model-agnostic Converse API and the AWS SDK for Go v2, from client construction to reading tokens back off the response.
Your first real inference call on Bedrock in Go via the unified Converse API: building the client, the ConverseInput message/content-block union, extracting the assistant text, and reading stop reason and token usage — with the content-block union explained.
Why the naive RAG pipeline from post 9 underperforms in production, and the concrete, evaluation-driven fixes — structure-aware chunking, hybrid search, reranking, query transformation, and deliberate context construction — each explained with the reasoning and a real Go sketch.
Why naive RAG underperforms and the techniques that fix it: measure recall@k first, then structure-aware chunking, hybrid dense+BM25 search fused with RRF, over-retrieve-then-rerank, query transformation (HyDE, multi-query), and deliberate context construction against lost-in-the-middle.
The opener for a Go series on building LLM and agent applications with Amazon Bedrock — what the service actually is, why it sits between your Go code and a dozen foundation models, and which aws-sdk-go-v2 packages you will lean on for the rest of the way.
The opener to a series on building LLM and agent applications on Amazon Bedrock in Go: what Bedrock actually is, what it adds over calling a provider API directly (one API across models, IAM auth, data residency), and the aws-sdk-go-v2 packages you'll use.
Wire the embedding client, vector store, and chat client from the last five posts into one working RAG pipeline in Go — ingest and chunk documents, retrieve the top matches for a question, inject them as grounded context, and generate a cited answer, all from scratch.
Assemble embeddings and vector search into a working RAG pipeline in Go: chunk documents, embed and store them, retrieve the top-k for a query, augment the prompt with grounded context (and cite sources), then generate — a baseline end-to-end Answer() built from scratch.
Build a working in-memory vector store and exact k-nearest-neighbor search in Go by hand — no vector database — then understand precisely what HNSW, FAISS, and pgvector optimize when brute force finally runs out of road.
Build an in-memory vector store and exact k-NN search in Go by hand: a VectorStore with Add and Search, top-k selection with container/heap, normalize-on-insert, an honest look at when brute force is right, and when ANN (HNSW, FAISS, pgvector) earns its keep.
Turn text into a `[]float32` that places meaning in space — what an embedding is, cosine similarity implemented by hand in Go, calling an OpenAI-compatible /embeddings endpoint with net/http, and a worked pairwise-similarity example that scores related sentences higher.
Turn text into a []float32 that places meaning in space — what an embedding is, cosine similarity implemented by hand in Go, calling an OpenAI-compatible /embeddings endpoint with net/http, and a worked pairwise-similarity example that scores related sentences higher.
Treating the prompt as a real engineering artifact — grounded in how a next-token predictor actually works — with roles, specificity, few-shot examples, decomposition, chain-of-thought, grounding, temperature, injection defense, and versioned Go templates you can test.
Prompt engineering as a real engineering discipline: roles, specificity, few-shot, decomposition, chain-of-thought and its cost, grounding, and prompt injection — plus building prompts as versioned, testable Go text/template templates you treat like code.
From-scratch Go for the two mechanisms that turn an LLM from a text generator into a component you can wire into real software — schema-constrained JSON and function calling — both spoken over the same OpenAI-compatible chat JSON.
Getting reliable machine-readable output from an LLM in Go: structured output (json-schema mode, decode into a typed struct, validate with a bounded retry) and tool/function calling (the full round-trip loop, decoding tool arguments, returning results tied to the call id).
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming — no framework required.
Make your first model call from scratch with net/http and encoding/json — the chat/messages API shape, a typed client with a Bearer key and context timeout, robust error handling, and server-sent-event streaming.
The unit a language model actually reads is neither a word nor a character — it is a token, and once you see the world the way the model does, half of its strange behavior stops being strange.
The unit a language model actually reads is neither a word nor a character but a token. How byte-pair encoding builds a vocabulary, why tokenization explains half of an LLM's strange behavior, and how to count tokens exactly in Go.
The working mental model an AI engineer needs — next-token prediction, attention, training, and sampling — without the transformer math, and with every fact tied back to a decision you make in code.
The working mental model an AI engineer needs — next-token prediction, attention at an intuition level, pretraining vs post-training, and sampling — with every fact tied back to a concrete decision you make in code.
The opener to a from-scratch series on building applications on top of foundation models in Go — what AI engineering actually is, how it differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.
The opener to a from-scratch AI-engineering-in-Go series: what AI engineering actually is, how building on foundation models differs from traditional ML and from ordinary software, and why Go is a serious language for the systems around the model.