<!-- Markdown version for AI agents and readers. Canonical: https://pratikdhanave.com/blog/posts/ai-eng-go-09-rag.html -->

# Retrieval-Augmented Generation

*Wire the embedding client, vector store, and chat client from the last five posts into one working RAG pipeline in Go — ingest and chunk documents, retrieve the top matches for a question, inject them as grounded context, and generate a cited answer, all from scratch.*

---

A language model knows a great deal, but it does not know two things that matter most to you: whatever happened after its training cutoff, and anything private — your handbook, your tickets, your codebase, this quarter's numbers. Ask it about those and it will answer anyway, confidently, from the fuzzy parametric memory baked into its weights. Sometimes that memory is right. Often it is stale, generic, or invented outright.

The naive fix is to paste everything the model might need into the prompt. That collapses fast. Post 3 showed that context is measured in tokens and every token costs money and latency, and the window has a hard ceiling — you cannot stuff a 40,000-page wiki into a request, and you would not want to pay for it if you could. Most of those pages are irrelevant to any single question anyway.

**Retrieval-Augmented Generation (RAG)** is the disciplined version of "paste it into the prompt." Instead of the whole corpus, you retrieve only the handful of passages actually relevant to the question, put *those* in the context, and instruct the model to answer from them. The model's fluency does the writing; your documents supply the facts. You have already built every piece this needs. This post assembles them.

---

## The four stages, and the parts you already have

RAG is a pipeline with four stages. Two happen ahead of time, two happen per question:

```text
INGEST (once, offline)          RETRIEVE + AUGMENT + GENERATE (per query)
─────────────────────           ────────────────────────────────────────
load docs                       embed the question      (post 7)
  → chunk them                    → Search top-k        (post 8)
  → embed each chunk  (post 7)    → build a prompt with the chunks
  → Add to store      (post 8)    → Chat to answer      (post 4)
```

The mapping to earlier posts is exact, so we reuse rather than reinvent:

- **Post 4** gave us the `llm` package: a `Chat(ctx, []Message)` method and `Message{Role, Content}`.
- **Post 7** gave us the `embed` package: `Embed(ctx, []string) ([][]float32, error)` against an OpenAI-compatible `/embeddings` endpoint, plus `normalize`.
- **Post 8** gave us the `vstore` package: `New(dim)`, `Add(id, text, vector)`, and `Search(query, k) []Result` doing exact k-NN.

Nothing below replaces those. The new work is the connective tissue: chunking, prompt assembly, and the small `Answer` function that runs the whole loop.

---

## Stage 1: Ingest — load and chunk

Ingestion turns raw documents into searchable records. The load step is boring on purpose — read files, strip nothing you need, hand back strings. The interesting decision is **chunking**: splitting each document into smaller pieces before you embed it.

Why not embed a whole document as one vector? Two reasons pull in the same direction. First, an embedding model has an input limit and squeezes everything you give it into a *single* fixed-length vector — feed it a 20-page document and the resulting vector is an averaged blur where no specific fact stands out. A query about one sentence on page 12 will match it weakly if at all. Second, retrieval feeds the chat model, which has its own context budget (post 3 again); you want to inject a few tight, on-topic passages, not whole documents that spend your token budget on irrelevance. Chunking gives retrieval something specific to match and generation something small to read.

The simplest chunker that works splits on a size budget with a small **overlap** so a fact spanning a boundary survives in at least one chunk. We measure in words here for clarity; a production chunker would count tokens (post 3) since that is what the limits are actually in.

```go
package rag

import "strings"

// Chunk is a slice of a source document, tagged with where it came from.
type Chunk struct {
	DocID string
	Index int    // position of this chunk within its document
	Text  string
}

// chunkDocument splits text into ~size-word chunks that overlap by `overlap`
// words, so a sentence straddling a boundary lands whole in one of them.
func chunkDocument(docID, text string, size, overlap int) []Chunk {
	words := strings.Fields(text)
	if len(words) == 0 {
		return nil
	}
	if overlap >= size {
		overlap = size / 2 // guard: overlap must be smaller than the window
	}

	var chunks []Chunk
	step := size - overlap
	for start, idx := 0, 0; start < len(words); start, idx = start+step, idx+1 {
		end := start + size
		if end > len(words) {
			end = len(words)
		}
		chunks = append(chunks, Chunk{
			DocID: docID,
			Index: idx,
			Text:  strings.Join(words[start:end], " "),
		})
		if end == len(words) {
			break
		}
	}
	return chunks
}
```

The `step := size - overlap` is the whole trick: each window advances by less than its width, so consecutive chunks share their edges. With `size=120, overlap=20`, chunk 0 covers words 0–120, chunk 1 covers 100–220, and the 20-word seam appears in both.

**The gotcha:** chunking strategy dominates RAG quality more than any other single knob, and it is a genuine trade-off with no free setting. Chunks that are too **big** blur many topics into one averaged vector, so retrieval can't distinguish them and each hit wastes context tokens on text the question didn't ask about. Chunks that are too **small** sever the context a fact needs to make sense — a pronoun whose antecedent got left in the previous chunk, a number whose units are a sentence away. Splitting on natural boundaries (paragraphs, headings, sentences) beats a blind word count, and the overlap is your cheap insurance against a fact landing exactly on a seam. There is no universal right size; measure retrieval quality on your own documents.

---

## Stage 1, continued: embed each chunk and Add to the store

With chunks in hand, ingestion finishes by embedding them (post 7) and adding each to the store (post 8). Embedding is batched — one HTTP round-trip for many chunks — exactly as post 7's `Embed` was built to do.

```go
import (
	"context"
	"fmt"
	"strconv"

	"example.com/embed"
	"example.com/vstore"
)

// Pipeline holds the pieces wired together for one corpus.
type Pipeline struct {
	Embedder *embed.Client
	Store    *vstore.VectorStore
	TopK     int
}

// Ingest chunks every document, embeds the chunks in one batch, and stores them.
func (p *Pipeline) Ingest(ctx context.Context, docs map[string]string) error {
	var chunks []Chunk
	for docID, text := range docs {
		chunks = append(chunks, chunkDocument(docID, text, 120, 20)...)
	}
	if len(chunks) == 0 {
		return fmt.Errorf("no chunks produced from %d docs", len(docs))
	}

	texts := make([]string, len(chunks))
	for i, c := range chunks {
		texts[i] = c.Text
	}

	vecs, err := p.Embedder.Embed(ctx, texts) // one batched call for all chunks
	if err != nil {
		return fmt.Errorf("embed chunks: %w", err)
	}

	for i, c := range chunks {
		id := c.DocID + "#" + strconv.Itoa(c.Index) // e.g. "handbook#3"
		if err := p.Store.Add(id, c.Text, vecs[i]); err != nil {
			return fmt.Errorf("add chunk %s: %w", id, err)
		}
	}
	return nil
}
```

The record ID `docID#index` is deliberate: it survives into the retrieval results, so when we cite sources later we can point back at the exact chunk and its parent document. The store normalizes on `Add` for us (post 8), so nothing extra is needed here.

---

## Stage 2: Retrieve — embed the query, search top-k

Retrieval is two lines of real work. Embed the question with the **same** embedder, then ask the store for the top-k nearest chunks.

```go
// retrieve embeds the question and returns the top-k most similar chunks.
func (p *Pipeline) retrieve(ctx context.Context, question string) ([]vstore.Result, error) {
	qvecs, err := p.Embedder.Embed(ctx, []string{question})
	if err != nil {
		return nil, fmt.Errorf("embed query: %w", err)
	}
	return p.Store.Search(qvecs[0], p.TopK), nil
}
```

**The gotcha:** the query must be embedded with the *same model* as the chunks. This is post 7's warning made load-bearing — a vector from `text-embedding-3-small` and a vector from any other model are coordinates in incompatible spaces, and their cosine similarity is meaningless noise. Because both the ingest path and this retrieve path go through the same `p.Embedder`, they can't drift apart by accident. The bug appears when someone re-embeds the corpus with a newer model but leaves the query path pointed at the old one; retrieval silently returns garbage that still *looks* like ranked results. Store the model name with your vectors so a mismatch is detectable rather than invisible.

---

## Stage 3: Augment — build a grounded prompt

Now the connective step that makes RAG *retrieval-augmented* rather than plain chat. We assemble a prompt that hands the retrieved chunks to the model as context and instructs it to answer *only* from them.

```go
import "strings"

// buildPrompt turns retrieved chunks into a system+user message pair that
// grounds the model in the context and forbids it from guessing.
func buildPrompt(question string, hits []vstore.Result) []llm.Message {
	var ctxBuilder strings.Builder
	for i, h := range hits {
		fmt.Fprintf(&ctxBuilder, "[%d] (source: %s)\n%s\n\n", i+1, h.ID, h.Text)
	}

	system := "You are a precise assistant. Answer the question using ONLY the " +
		"numbered context passages below. If the answer is not contained in the " +
		"context, reply exactly: \"I don't know based on the provided context.\" " +
		"Cite the passages you used by their number, like [1] or [2]."

	user := fmt.Sprintf("Context:\n%s\nQuestion: %s", ctxBuilder.String(), question)

	return []llm.Message{
		{Role: "system", Content: system},
		{Role: "user", Content: user},
	}
}
```

Two design choices carry the weight. Numbering each passage (`[1]`, `[2]`) and tagging it with its source ID lets the model cite specifically and lets *you* verify the citation against the original chunk. And the explicit escape hatch — reply "I don't know" when the context lacks the answer — is what turns off the model's instinct to fill silence with plausible fiction.

**The gotcha:** you must instruct the model *both* to ground in the context *and* to admit when the answer isn't there. Give it retrieved passages without the "only from context" constraint and it treats them as a suggestion, blending them with parametric memory and hallucinating the difference. Give it the grounding rule without the "I don't know" escape and it will torture an irrelevant passage into a wrong answer rather than decline. Both instructions are load-bearing; dropping either one reintroduces exactly the failure mode RAG exists to prevent.

---

## Stage 4: Generate — call the chat client and cite sources

The last stage calls post 4's `Chat` and packages the answer with the sources that produced it, so the caller can audit where the answer came from.

```go
// Answer is the whole pipeline: retrieve, augment, generate, cite.
type Answer struct {
	Text    string          // the model's grounded reply
	Sources []vstore.Result // the chunks that were injected, with scores
}

func (p *Pipeline) Answer(ctx context.Context, chat *llm.Client, question string) (Answer, error) {
	hits, err := p.retrieve(ctx, question)
	if err != nil {
		return Answer{}, err
	}
	if len(hits) == 0 {
		return Answer{Text: "I don't know based on the provided context."}, nil
	}

	messages := buildPrompt(question, hits)
	resp, err := chat.Chat(ctx, messages)
	if err != nil {
		return Answer{}, fmt.Errorf("generate: %w", err)
	}
	if len(resp.Choices) == 0 {
		return Answer{}, fmt.Errorf("model returned no choices")
	}

	return Answer{
		Text:    resp.Choices[0].Message.Content,
		Sources: hits,
	}, nil
}
```

Returning `Sources` alongside `Text` is not decoration. It is what makes a RAG answer *trustworthy in the way a raw LLM answer is not*: every claim can be traced to a passage a human can read. When the model cites `[2]`, the caller can print `hits[1]` and check.

---

## The whole thing, wired end to end

Here is ingest-then-ask in one runnable `main`, reusing the three packages unchanged.

```go
package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"example.com/embed"
	"example.com/llm"
	"example.com/rag"
	"example.com/vstore"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
	defer cancel()

	embedder, err := embed.NewClient()
	if err != nil {
		log.Fatal(err)
	}
	chat, err := llm.NewClient()
	if err != nil {
		log.Fatal(err)
	}

	const dim = 1536 // must match the embedding model's output dimension
	pipe := &rag.Pipeline{
		Embedder: embedder,
		Store:    vstore.New(dim),
		TopK:     4,
	}

	docs := map[string]string{
		"handbook": "Employees accrue 1.5 vacation days per month, up to 24 days " +
			"per year. Unused days carry over, but the balance is capped at 30 days. " +
			"Requests must be submitted two weeks in advance through the HR portal.",
		"security": "All laptops must have full-disk encryption enabled. Report a " +
			"lost or stolen device to security@example.com within 24 hours.",
	}
	if err := pipe.Ingest(ctx, docs); err != nil {
		log.Fatal(err)
	}

	ans, err := pipe.Answer(ctx, chat, "How many vacation days can I carry over at most?")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(ans.Text)
	fmt.Println("\nSources:")
	for _, s := range ans.Sources {
		fmt.Printf("  [%.3f] %s\n", s.Score, s.ID)
	}
}
```

With a real embedding model and chat model behind the clients, the model answers from the handbook chunk — not from whatever a generic HR policy sounds like — and shows its work:

```text
The balance is capped at 30 days, so you can carry over at most 30 vacation days. [1]

Sources:
  [0.734] handbook#0
  [0.291] security#0
```

Note that `security#0` was retrieved even though it has nothing to do with vacation — top-k always returns k results. Its low score (0.291) is the tell, and the grounding instruction is why it didn't poison the answer.

---

## Keeping it honest: this is baseline RAG

What you have built is real and it works, but it is deliberately the *naive* baseline. Its weaknesses are worth naming plainly, because post 10 exists to fix them:

**The gotcha:** retrieved is not the same as relevant. `Search` returns exactly k chunks whether or not any of them is actually on-topic — for a question your corpus can't answer, it still hands back the k *least bad* vectors, and a model told to answer from context may over-trust them. The fix is to check scores: apply a similarity floor (e.g. drop hits below 0.4) and, if nothing clears the bar, return "I don't know" without ever calling the chat model. That single threshold turns "always answers, sometimes wrong" into "declines when it should."

| Limitation of this baseline | What a better version does (post 10) |
|---|---|
| Fixed word-count chunks | Semantic / structure-aware splitting |
| Top-k with no score floor | Threshold filtering; return "don't know" on weak hits |
| Raw query embedded as-is | Query rewriting, hybrid keyword + vector search |
| Chunks injected in retrieval order | Re-ranking so the best passage leads |

None of that changes the shape you built. It refines each stage. Get the baseline correct first — grounded, cited, honest about ignorance — and every later improvement is a swap of one stage's implementation, not a rewrite.

---

## Key takeaways

- **RAG exists because LLMs have a knowledge cutoff and no access to your private or recent data**, and stuffing the whole corpus into the prompt is impossible and expensive (post 3). Retrieve the few relevant chunks instead.
- **The pipeline is four stages**: ingest (chunk + embed + store), retrieve (embed query + search), augment (build a grounded prompt), generate (chat). Each maps directly to a package from posts 4, 7, and 8 — you reuse, you don't reinvent.
- **Chunking dominates quality.** Too big blurs relevance and burns tokens; too small severs context. Overlap insures against facts landing on a seam; natural boundaries beat blind counts.
- **Grounding needs two instructions, not one**: answer *only* from context, *and* say "I don't know" when it isn't there. Drop either and the model falls back on parametric memory and hallucinates.
- **Embed the query with the same model as the chunks**, and **return the sources** so every answer is auditable. Retrieved ≠ relevant — check scores with a floor, because top-k always returns k results.

---

## Further reading

- [Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (arXiv:2005.11401)](https://arxiv.org/abs/2005.11401) — the original 2020 RAG paper that named and framed the pattern.
- [OpenAI — Retrieval and RAG guidance](https://platform.openai.com/docs/guides/retrieval) — a provider's take on retrieval, chunking, and grounding against your own data.
- [Cohere — Retrieval-Augmented Generation](https://docs.cohere.com/docs/retrieval-augmented-generation-rag) — a second provider's overview, including re-ranking, which post 10 builds toward.
- [Go `strings` package documentation](https://pkg.go.dev/strings) — `Fields`, `Join`, and `Builder`, used by the chunker and prompt assembly.

---

Author: Pratik Dhanave (https://pratikdhanave.com) | Published: 2026-08-05
Canonical: https://pratikdhanave.com/blog/posts/ai-eng-go-09-rag.html
Attribution: please link the canonical URL when citing or quoting this post.
