A Python engineer's map of NVIDIA's AI platform — NIM inference, NeMo Retriever, NeMo Guardrails, and Triton with TensorRT-LLM — and why the same code runs against the hosted API Catalog or your own self-hosted microservice.
The opener to a series on building LLM and agent applications on NVIDIA's AI stack from Python: how NIM, the API Catalog, NeMo Retriever, NeMo Guardrails, Triton and TensorRT-LLM fit together — and why the openai client and langchain-nvidia-ai-endpoints make it Python-native.
Your first real NVIDIA NIM calls in Python, two idiomatic ways: the plain OpenAI SDK pointed at NVIDIA's endpoint, and the official LangChain integration — with error handling, streaming, and the one change that makes the same code run against a self-hosted model.
Make your first NIM calls in Python two ways: the standard openai client pointed at the API Catalog with an nvapi- key, and the official ChatNVIDIA from langchain-nvidia-ai-endpoints — with streaming, token usage, typed errors, and the base_url swap to a self-hosted NIM.
Driving function calling against NVIDIA NIM models from Python — the full request-execute-respond loop with the plain `openai` client, then the same thing automated by `ChatNVIDIA.bind_tools`.
Function/tool calling against NIM models in Python: the openai round-trip loop (json.loads the arguments string, echo tool_call_id, cap the iterations) and the LangChain ChatNVIDIA.bind_tools path — with the honest caveat that model support varies.
Building RAG's retrieval core in Python — turning a corpus and a query into vectors with NeMo Retriever embedding NIMs, scoring by cosine similarity, then sharpening the shortlist with a cross-encoder reranker NIM.
Use NeMo Retriever from Python to build RAG's retrieval core: NVIDIAEmbeddings (embed_documents vs embed_query for the asymmetric passage/query distinction) with cosine scoring, and NVIDIARerank.compress_documents for a two-stage retrieve-then-rerank pipeline.
Assembling a full retrieval-augmented generation pipeline in Python — chunk and embed with NeMo Retriever, shortlist by cosine, sharpen with a reranker, then ground a ChatNVIDIA answer that cites its sources — first from scratch, then the idiomatic LangChain wiring.
Build a full RAG pipeline in Python on NVIDIA's stack: chunk and embed with NeMo Retriever, shortlist by cosine, rerank, then generate a grounded, cited answer with ChatNVIDIA — shown both from scratch and the idiomatic LangChain assembly, with notes on scaling to a vector DB.
Adding a safety layer to an NVIDIA-stack Python app with NeMo Guardrails — and why running it in-process, with no HTTP boundary, is the quiet advantage Python gives you over a separate guardrails server.
Add safety with NeMo Guardrails, in-process in Python: the rail types (input/output/dialog/retrieval), the config.yml + Colang model, loading with RailsConfig/LLMRails backed by a NIM model, and detecting when a rail trips instead of treating a refusal as a normal answer.
When to graduate from the hosted API Catalog to inference you run yourself — deploying a NIM container, reaching for Triton and its Python client, the TensorRT-LLM levers that raise throughput and cut latency, and the one base_url change that carries your Python client across unchanged.
Move from the hosted API Catalog to self-hosted inference: run a NIM container (your Python client works unchanged by swapping base_url), use Triton directly via tritonclient, and understand the TensorRT-LLM levers — quantization, in-flight batching, paged KV cache, tensor parallelism.
Taking an NVIDIA-stack LLM system from a working prototype to something you trust in production — reliability, cost and throughput, observability, and security — all from Python, with the OpenAI-compatible surface keeping the code stable whether you burst to the API Catalog or run your own NIM.
Run an NVIDIA-stack LLM system in production from Python: hosted vs self-hosted vs hybrid, reliability (client retries, tenacity backoff, readiness probes, fallback), the GPU-hours cost model, Prometheus observability across the pipeline, and securing nvapi-/NGC keys.
This series is part of a larger body of work by Pratik Dhanave, an Agentic AI Architect writing about production AI systems, distributed systems, and cloud-native engineering. Explore all course series, browse every post, or find topics via the tag index.