Production on watsonx

Everything in this series so far has run against a development project: a project_id, an API key in your shell, and a ModelInference call in a script. That is exactly right for building. It is not what you run in production. This finale is about the gap between the two — the choices, the failure modes, and the Python patterns that keep a watsonx.ai system dependable, affordable, observable, and secure once real traffic hits it.

We will lean on everything the earlier posts built: calling models (post 2), tool calling with Granite (post 3), embeddings and reranking (post 4), RAG with Docling (post 5), guardrails and Granite Guardian (post 6), and watsonx.governance (post 7). Production is where those pieces stop being separate demos and become one traced, monitored request path.

From project to space: how prod differs from dev

In development you scope work to a project and pass project_id. In production you promote the asset — a deployed prompt template, a deployed model, or an AI service — into a deployment space and address it by space_id. A space is watsonx.ai’s production container: it holds deployed assets, controls who can invoke them, and separates a running service from the experimentation that produced it.

The SDK addresses one or the other on the same APIClient.

from ibm_watsonx_ai import APIClient, Credentials

credentials = Credentials(
    url=os.environ["WATSONX_URL"],
    api_key=os.environ["WATSONX_APIKEY"],
)
client = APIClient(credentials)

# development
client.set.default_project(os.environ["WATSONX_PROJECT_ID"])

# production — the same client, a different scope
client.set.default_space(os.environ["WATSONX_SPACE_ID"])

Once a prompt template or model is deployed into a space, you invoke it through the deployment rather than naming a raw model_id, which lets the platform version and govern what actually runs.

# invoke a deployed asset by its deployment id, not a bare model_id
resp = client.deployments.generate(
    deployment_id=os.environ["WATSONX_DEPLOYMENT_ID"],
    params={"prompt_variables": {"question": user_question}},
)

Beyond deployed prompt templates, watsonx.ai lets you deploy an AI service — your own Python function, packaged and run inside the space — so orchestration like “retrieve, rerank, guard, generate” lives behind one governed endpoint instead of in a client script. Check the ibm-watsonx-ai deployment docs for the exact packaging call in your SDK version, since the AI-service surface is still evolving.

The gotcha: promoting from dev to prod is not swapping project_id for space_id in a string. The asset itself has to be moved — promoted or deployed into the space — before a space_id can reach it. A space with no deployed asset returns nothing no matter how correct your credentials are. Treat “promote the asset into the space” as a real deployment step, not a config edit.

SaaS or Cloud Pak for Data: where watsonx runs

Where your production endpoint physically lives is a first-class decision, and it changes only the url and the operational model — the SDK calls stay the same.

• IBM Cloud (SaaS). watsonx.ai as a managed service, addressed by a regional URL like https://us-south.ml.cloud.ibm.com. IBM runs the infrastructure; you pick the region, which fixes where inference happens and where data is processed.
• Cloud Pak for Data (CPD). watsonx.ai running on OpenShift in your own environment — on-premises or in your cloud tenancy — for hybrid deployments and stricter data residency or isolation requirements. Here the url points at your CPD cluster, and authentication uses your CPD credentials rather than an IBM Cloud IAM key.

Region and deployment flavor together decide data residency. If a workload must keep data inside a jurisdiction, choosing the right region (SaaS) or hosting on CPD is the control — not anything you can patch later in code. Decide it before you build the endpoint, because moving a production space across regions or platforms is a migration, not a setting.

Reliability: timeouts, retries, and fallback

Networks are unreliable, services rate-limit, and models occasionally return errors. A production caller assumes every request can fail and decides in advance what happens next.

The SDK exposes its failures through the hierarchy post 2 introduced: WMLClientError for client-side and configuration problems, and its subclass ApiRequestFailure when the service returns a non-success status (the message carries the HTTP status and the server’s error payload). Retry the transient ones — timeouts, 429 rate limits, 5xx — and fail fast on the permanent ones like a bad model_id or missing access.

tenacity handles the retry loop cleanly. Use exponential backoff with jitter so many clients backing off at once do not synchronize into a thundering herd.

from tenacity import (
    retry, stop_after_attempt, wait_exponential_jitter,
    retry_if_exception, before_sleep_log,
)
from ibm_watsonx_ai.wml_client_error import ApiRequestFailure
import logging

log = logging.getLogger("watsonx")

def is_transient(exc: BaseException) -> bool:
    # retry rate limits and server errors; give up on client mistakes
    if isinstance(exc, ApiRequestFailure):
        status = getattr(exc, "response", None) and getattr(exc.response, "status_code", None)
        return status in (429, 500, 502, 503, 504)
    return isinstance(exc, (TimeoutError, ConnectionError))

@retry(
    retry=retry_if_exception(is_transient),
    wait=wait_exponential_jitter(initial=1, max=30),
    stop=stop_after_attempt(5),
    before_sleep=before_sleep_log(log, logging.WARNING),
    reraise=True,
)
def generate_with_retry(model, prompt: str) -> dict:
    return model.generate(prompt=prompt)

Two more reliability habits. First, bound every call with a timeout — the SDK accepts request timeouts, and an unbounded call is a hung worker waiting to happen. Second, have a fallback: when the primary path is exhausted, degrade gracefully rather than 500 the user. A smaller Granite model, a cached previous answer, or a plain “try again shortly” is almost always better than an error page.

from ibm_watsonx_ai.foundation_models import ModelInference

def answer(prompt: str) -> str:
    try:
        return generate_with_retry(primary_model, prompt)["results"][0]["generated_text"]
    except ApiRequestFailure:
        log.warning("primary model failed after retries; falling back")
        cached = cache.get(prompt)
        if cached:
            return cached
        # a smaller, cheaper model as a last resort
        return fallback_model.generate_text(prompt=prompt)

The gotcha: the IAM bearer token behind your API key is short-lived, and the SDK refreshes it for you — you never write token-refresh code (post 2). What you do own is the long-lived API key itself, which expires or gets rotated on your organization’s schedule. Store it in a secret manager, not in code or a baked image, and read it at startup so a rotation is a secret update and a restart, not a redeploy. Do not try to “help” the SDK by caching tokens yourself; you will only reintroduce the expiry bug it already solved.

Cost and throughput

watsonx.ai bills on tokens processed, so cost control starts with token accounting. As post 2 showed, generate() returns per-call counts and chat() returns a usage object — read them on every production call and record them.

resp = model.generate(prompt=prompt)
r = resp["results"][0]
input_tokens = r["input_token_count"]
output_tokens = r["generated_token_count"]
# chat(): resp["usage"]["prompt_tokens"] / ["completion_tokens"] / ["total_tokens"]

From there, four levers move the bill and the latency:

• Right-size the model. A granite-3-8b-instruct answers most tasks; reserve a larger model for the queries that genuinely need it. Routing simple traffic to the smaller model is usually the biggest single saving.
• Trim the input. In a RAG path (post 5), fewer, better-reranked chunks (post 4) mean fewer input tokens per call — reranking pays for itself in prompt size.
• Batch where the API allows it instead of firing one request per item in a Python loop.
• Cap concurrency client-side. Do not let a burst of users translate into an unbounded fan-out at the service; a semaphore keeps you inside your plan’s limits and turns 429s into short waits.

import asyncio

sem = asyncio.Semaphore(8)  # at most 8 in-flight calls

async def bounded_generate(model, prompt: str) -> str:
    async with sem:
        # ModelInference calls are synchronous; run them off the event loop
        resp = await asyncio.to_thread(generate_with_retry, model, prompt)
        return resp["results"][0]["generated_text"]

On actual prices: watsonx.ai pricing is plan- and region-specific and changes over time, so this post cites the pricing page rather than quoting a number — check it for your plan and region and compute your own per-request cost from the token counts above.

The gotcha: pin your model version and prompt-template version. Foundation models are updated and eventually deprecated on a published schedule (post 2), and a prompt template evolves as you tune it. If production silently floats to a new model build or an edited template, your outputs — and your token usage — shift underneath you with no code change. Pin the versions you validated, and treat a model or template upgrade as a change you test and roll out deliberately, not a surprise.

Observability and governance

You cannot operate what you cannot see. Two layers matter: your own telemetry, and watsonx.governance.

Structured logging and metrics. Give every request a correlation id and log each stage of the pipeline with it, so one line of grep reconstructs a full RAG → rerank → guardrail → generate trace. Record latency, token counts, and outcome as structured fields, and derive your operational metrics — p95 latency, tokens per request, error rate — from them.

import logging, time, uuid

log = logging.getLogger("watsonx")

def handle(question: str) -> str:
    rid = uuid.uuid4().hex[:12]
    t0 = time.monotonic()
    log.info("request start", extra={"rid": rid, "stage": "start"})

    chunks = retrieve(question)                       # post 5: RAG
    log.info("retrieved", extra={"rid": rid, "stage": "retrieve", "n": len(chunks)})

    top = rerank(question, chunks)                    # post 4: rerank
    if not guardrails_ok(question, top):              # post 6: Granite Guardian
        log.warning("blocked by guardrail", extra={"rid": rid, "stage": "guard"})
        return "I can't help with that request."

    resp = model.generate(prompt=build_prompt(question, top))
    r = resp["results"][0]
    log.info("generated", extra={
        "rid": rid, "stage": "generate",
        "input_tokens": r["input_token_count"],
        "output_tokens": r["generated_token_count"],
        "latency_ms": round((time.monotonic() - t0) * 1000),
    })
    return r["generated_text"]

watsonx.governance. Post 7 covered factsheets and monitors; production is where they earn their keep. A factsheet records the model, prompt template, and evaluation lineage of what you deployed — auditable evidence of what is running. Monitors (via watsonx.governance / OpenScale) watch the live endpoint for quality, drift, and generative-AI risks like toxicity or hallucination, so a regression shows up as an alert instead of a customer complaint. The ibm-aigov-facts-client package (AIGovFactsClient) is the Python entry point for wiring factsheets to a deployment; consult the watsonx.governance docs for the monitor configuration that matches your evaluation setup.

The gotcha: shipping without governance monitors wired to the production endpoint means you are blind to drift and hallucination in prod. Local guardrails (post 6) block bad inputs and outputs request-by-request, but only continuous monitoring tells you the aggregate quality is sliding — a model update degrading answers, inputs drifting away from what you evaluated on. In a governance-first platform, connecting monitors is not an optional nicety; it is the difference between knowing your system’s health and hoping.

Security

The threats in production are credential leaks and injection, and both have concrete Python-side defenses.

• Secrets belong in a secret manager, never in code. The IAM API key and the project/space id should be read from IBM Cloud Secrets Manager, a Kubernetes secret, HashiCorp Vault, or the platform’s equivalent — injected as environment variables or fetched at startup. No key in git, no key in a container image, no key in a log line.
• Least privilege. Provision the production API key (or service id) with only the access the endpoint needs — invoke a specific space and its deployments, nothing more. A key scoped to one space cannot be used to wander the account if it leaks.
• Injection is a live surface. RAG (post 5) puts retrieved document text into the prompt, and tool calling (post 3) lets model output trigger actions — both are paths for prompt injection and untrusted content. Keep the guardrails from post 6 on the input and the output, treat tool arguments as untrusted (validate before executing, gate side effects), and never let retrieved text silently override your system instructions.

Least privilege plus a secret manager plus guardrails on both ends is the baseline; none of the three substitutes for the others.

Wrapping it in a FastAPI endpoint

Putting the pieces together, a minimal production endpoint reads config from the environment at startup, applies the retry/fallback logic, and returns a clean response.

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from ibm_watsonx_ai.wml_client_error import WMLClientError

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
async def ask(q: Query):
    try:
        return {"answer": await bounded_generate(primary_model, q.question)}
    except WMLClientError as exc:
        log.error("request failed", exc_info=exc)
        raise HTTPException(status_code=502, detail="Model service unavailable")

@app.get("/healthz")
def healthz():
    return {"status": "ok"}

The health check lets your orchestrator restart a stuck worker; the WMLClientError catch turns SDK failures into a proper HTTP status instead of a stack trace on the wire.

Production checklist

Key takeaways

• Dev is a project; prod is a space. Promote the asset into a deployment space and invoke it by space_id / deployment id — moving the asset is a real step, not a string swap.
• Placement is a design decision. SaaS region or Cloud Pak for Data fixes data residency, and only the url and auth model change in code.
• Assume every call fails. Timeouts, tenacity retries with backoff and jitter for transient errors, and a smaller-model or cached fallback keep the endpoint up.
• The SDK refreshes IAM tokens; you protect the API key. Store it in a secret manager, scoped least-privilege, and rotate it out of band.
• Cost tracks tokens. Read input_token_count / generated_token_count, right-size the model, cap concurrency, and compute price from the region-specific pricing page — never a guessed number.
• Governance is not optional in production. Wire factsheets and drift/quality monitors, or you are blind to regressions that request-level guardrails cannot see.

This is the last post in the series, so here is the whole arc: we started with the watsonx platform (post 1), made real inference calls with ModelInference and ChatWatsonx (post 2), gave Granite tools (post 3), added embeddings and reranking (post 4), built RAG with Docling (post 5), put guardrails and Granite Guardian around it (post 6), brought it under watsonx.governance (post 7), and now run it in production. The through-line is watsonx’s enterprise, governance-first posture: the platform is built so the same system you demo can be deployed, secured, monitored, and audited — which is exactly what production asks of it.

Further reading

• Deploying assets and deployment spaces in watsonx.ai — spaces, deployed prompt templates, and AI services.
• IBM Cloud Pak for Data documentation — on-premises and hybrid watsonx deployment.
• watsonx.ai pricing — plan- and region-specific rates; compute your own per-request cost.
• ibm-watsonx-ai SDK documentation — APIClient, set.default_space, deployments, and error classes.
• IBM watsonx.governance documentation — factsheets and production model monitors.
• tenacity documentation — retry, wait_exponential_jitter, and stop conditions.
• FastAPI documentation — building and serving the endpoint.