Retrieval-Augmented Generation

Wire the embedding client, vector store, and chat client from the last five posts into one working RAG pipeline in Go — ingest and chunk documents, retrieve the top matches for a question, inject them as grounded context, and generate a cited answer, all from scratch.

By Pratik Dhanave · · 13 min read
#AI Engineering#RAG#Go#Vector Search#Backend Engineering#Embeddings#Vector Databases#Prompt Engineering
Part 9 of 15 AI Engineering in Go
← Part 8: Vector Search from Scratch Part 10: Making RAG Good →

A language model knows a great deal, but it does not know two things that matter most to you: whatever happened after its training cutoff, and anything private — your handbook, your tickets, your codebase, this quarter’s numbers. Ask it about those and it will answer anyway, confidently, from the fuzzy parametric memory baked into its weights. Sometimes that memory is right. Often it is stale, generic, or invented outright.

The naive fix is to paste everything the model might need into the prompt. That collapses fast. Post 3 showed that context is measured in tokens and every token costs money and latency, and the window has a hard ceiling — you cannot stuff a 40,000-page wiki into a request, and you would not want to pay for it if you could. Most of those pages are irrelevant to any single question anyway.

Retrieval-Augmented Generation (RAG) is the disciplined version of “paste it into the prompt.” Instead of the whole corpus, you retrieve only the handful of passages actually relevant to the question, put those in the context, and instruct the model to answer from them. The model’s fluency does the writing; your documents supply the facts. You have already built every piece this needs. This post assembles them.

The four stages, and the parts you already have

RAG is a pipeline with four stages. Two happen ahead of time, two happen per question:

INGEST (once, offline)          RETRIEVE + AUGMENT + GENERATE (per query)
─────────────────────           ────────────────────────────────────────
load docs                       embed the question      (post 7)
  → chunk them                    → Search top-k        (post 8)
  → embed each chunk  (post 7)    → build a prompt with the chunks
  → Add to store      (post 8)    → Chat to answer      (post 4)

The mapping to earlier posts is exact, so we reuse rather than reinvent:

Nothing below replaces those. The new work is the connective tissue: chunking, prompt assembly, and the small Answer function that runs the whole loop.

Stage 1: Ingest — load and chunk

Ingestion turns raw documents into searchable records. The load step is boring on purpose — read files, strip nothing you need, hand back strings. The interesting decision is chunking: splitting each document into smaller pieces before you embed it.

Why not embed a whole document as one vector? Two reasons pull in the same direction. First, an embedding model has an input limit and squeezes everything you give it into a single fixed-length vector — feed it a 20-page document and the resulting vector is an averaged blur where no specific fact stands out. A query about one sentence on page 12 will match it weakly if at all. Second, retrieval feeds the chat model, which has its own context budget (post 3 again); you want to inject a few tight, on-topic passages, not whole documents that spend your token budget on irrelevance. Chunking gives retrieval something specific to match and generation something small to read.

The simplest chunker that works splits on a size budget with a small overlap so a fact spanning a boundary survives in at least one chunk. We measure in words here for clarity; a production chunker would count tokens (post 3) since that is what the limits are actually in.

package rag

import "strings"

// Chunk is a slice of a source document, tagged with where it came from.
type Chunk struct {
    DocID string
    Index int    // position of this chunk within its document
    Text  string
}

// chunkDocument splits text into ~size-word chunks that overlap by `overlap`
// words, so a sentence straddling a boundary lands whole in one of them.
func chunkDocument(docID, text string, size, overlap int) []Chunk {
    words := strings.Fields(text)
    if len(words) == 0 {
        return nil
    }
    if overlap >= size {
        overlap = size / 2 // guard: overlap must be smaller than the window
    }

    var chunks []Chunk
    step := size - overlap
    for start, idx := 0, 0; start < len(words); start, idx = start+step, idx+1 {
        end := start + size
        if end > len(words) {
            end = len(words)
        }
        chunks = append(chunks, Chunk{
            DocID: docID,
            Index: idx,
            Text:  strings.Join(words[start:end], " "),
        })
        if end == len(words) {
            break
        }
    }
    return chunks
}

The step := size - overlap is the whole trick: each window advances by less than its width, so consecutive chunks share their edges. With size=120, overlap=20, chunk 0 covers words 0–120, chunk 1 covers 100–220, and the 20-word seam appears in both.

The gotcha: chunking strategy dominates RAG quality more than any other single knob, and it is a genuine trade-off with no free setting. Chunks that are too big blur many topics into one averaged vector, so retrieval can’t distinguish them and each hit wastes context tokens on text the question didn’t ask about. Chunks that are too small sever the context a fact needs to make sense — a pronoun whose antecedent got left in the previous chunk, a number whose units are a sentence away. Splitting on natural boundaries (paragraphs, headings, sentences) beats a blind word count, and the overlap is your cheap insurance against a fact landing exactly on a seam. There is no universal right size; measure retrieval quality on your own documents.

Stage 1, continued: embed each chunk and Add to the store

With chunks in hand, ingestion finishes by embedding them (post 7) and adding each to the store (post 8). Embedding is batched — one HTTP round-trip for many chunks — exactly as post 7’s Embed was built to do.

import (
    "context"
    "fmt"
    "strconv"

    "example.com/embed"
    "example.com/vstore"
)

// Pipeline holds the pieces wired together for one corpus.
type Pipeline struct {
    Embedder *embed.Client
    Store    *vstore.VectorStore
    TopK     int
}

// Ingest chunks every document, embeds the chunks in one batch, and stores them.
func (p *Pipeline) Ingest(ctx context.Context, docs map[string]string) error {
    var chunks []Chunk
    for docID, text := range docs {
        chunks = append(chunks, chunkDocument(docID, text, 120, 20)...)
    }
    if len(chunks) == 0 {
        return fmt.Errorf("no chunks produced from %d docs", len(docs))
    }

    texts := make([]string, len(chunks))
    for i, c := range chunks {
        texts[i] = c.Text
    }

    vecs, err := p.Embedder.Embed(ctx, texts) // one batched call for all chunks
    if err != nil {
        return fmt.Errorf("embed chunks: %w", err)
    }

    for i, c := range chunks {
        id := c.DocID + "#" + strconv.Itoa(c.Index) // e.g. "handbook#3"
        if err := p.Store.Add(id, c.Text, vecs[i]); err != nil {
            return fmt.Errorf("add chunk %s: %w", id, err)
        }
    }
    return nil
}

The record ID docID#index is deliberate: it survives into the retrieval results, so when we cite sources later we can point back at the exact chunk and its parent document. The store normalizes on Add for us (post 8), so nothing extra is needed here.

Stage 2: Retrieve — embed the query, search top-k

Retrieval is two lines of real work. Embed the question with the same embedder, then ask the store for the top-k nearest chunks.

// retrieve embeds the question and returns the top-k most similar chunks.
func (p *Pipeline) retrieve(ctx context.Context, question string) ([]vstore.Result, error) {
    qvecs, err := p.Embedder.Embed(ctx, []string{question})
    if err != nil {
        return nil, fmt.Errorf("embed query: %w", err)
    }
    return p.Store.Search(qvecs[0], p.TopK), nil
}

The gotcha: the query must be embedded with the same model as the chunks. This is post 7’s warning made load-bearing — a vector from text-embedding-3-small and a vector from any other model are coordinates in incompatible spaces, and their cosine similarity is meaningless noise. Because both the ingest path and this retrieve path go through the same p.Embedder, they can’t drift apart by accident. The bug appears when someone re-embeds the corpus with a newer model but leaves the query path pointed at the old one; retrieval silently returns garbage that still looks like ranked results. Store the model name with your vectors so a mismatch is detectable rather than invisible.

Stage 3: Augment — build a grounded prompt

Now the connective step that makes RAG retrieval-augmented rather than plain chat. We assemble a prompt that hands the retrieved chunks to the model as context and instructs it to answer only from them.

import "strings"

// buildPrompt turns retrieved chunks into a system+user message pair that
// grounds the model in the context and forbids it from guessing.
func buildPrompt(question string, hits []vstore.Result) []llm.Message {
    var ctxBuilder strings.Builder
    for i, h := range hits {
        fmt.Fprintf(&ctxBuilder, "[%d] (source: %s)\n%s\n\n", i+1, h.ID, h.Text)
    }

    system := "You are a precise assistant. Answer the question using ONLY the " +
        "numbered context passages below. If the answer is not contained in the " +
        "context, reply exactly: \"I don't know based on the provided context.\" " +
        "Cite the passages you used by their number, like [1] or [2]."

    user := fmt.Sprintf("Context:\n%s\nQuestion: %s", ctxBuilder.String(), question)

    return []llm.Message{
        {Role: "system", Content: system},
        {Role: "user", Content: user},
    }
}

Two design choices carry the weight. Numbering each passage ([1], [2]) and tagging it with its source ID lets the model cite specifically and lets you verify the citation against the original chunk. And the explicit escape hatch — reply “I don’t know” when the context lacks the answer — is what turns off the model’s instinct to fill silence with plausible fiction.

The gotcha: you must instruct the model both to ground in the context and to admit when the answer isn’t there. Give it retrieved passages without the “only from context” constraint and it treats them as a suggestion, blending them with parametric memory and hallucinating the difference. Give it the grounding rule without the “I don’t know” escape and it will torture an irrelevant passage into a wrong answer rather than decline. Both instructions are load-bearing; dropping either one reintroduces exactly the failure mode RAG exists to prevent.

Stage 4: Generate — call the chat client and cite sources

The last stage calls post 4’s Chat and packages the answer with the sources that produced it, so the caller can audit where the answer came from.

// Answer is the whole pipeline: retrieve, augment, generate, cite.
type Answer struct {
    Text    string          // the model's grounded reply
    Sources []vstore.Result // the chunks that were injected, with scores
}

func (p *Pipeline) Answer(ctx context.Context, chat *llm.Client, question string) (Answer, error) {
    hits, err := p.retrieve(ctx, question)
    if err != nil {
        return Answer{}, err
    }
    if len(hits) == 0 {
        return Answer{Text: "I don't know based on the provided context."}, nil
    }

    messages := buildPrompt(question, hits)
    resp, err := chat.Chat(ctx, messages)
    if err != nil {
        return Answer{}, fmt.Errorf("generate: %w", err)
    }
    if len(resp.Choices) == 0 {
        return Answer{}, fmt.Errorf("model returned no choices")
    }

    return Answer{
        Text:    resp.Choices[0].Message.Content,
        Sources: hits,
    }, nil
}

Returning Sources alongside Text is not decoration. It is what makes a RAG answer trustworthy in the way a raw LLM answer is not: every claim can be traced to a passage a human can read. When the model cites [2], the caller can print hits[1] and check.

The whole thing, wired end to end

Here is ingest-then-ask in one runnable main, reusing the three packages unchanged.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "example.com/embed"
    "example.com/llm"
    "example.com/rag"
    "example.com/vstore"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
    defer cancel()

    embedder, err := embed.NewClient()
    if err != nil {
        log.Fatal(err)
    }
    chat, err := llm.NewClient()
    if err != nil {
        log.Fatal(err)
    }

    const dim = 1536 // must match the embedding model's output dimension
    pipe := &rag.Pipeline{
        Embedder: embedder,
        Store:    vstore.New(dim),
        TopK:     4,
    }

    docs := map[string]string{
        "handbook": "Employees accrue 1.5 vacation days per month, up to 24 days " +
            "per year. Unused days carry over, but the balance is capped at 30 days. " +
            "Requests must be submitted two weeks in advance through the HR portal.",
        "security": "All laptops must have full-disk encryption enabled. Report a " +
            "lost or stolen device to security@example.com within 24 hours.",
    }
    if err := pipe.Ingest(ctx, docs); err != nil {
        log.Fatal(err)
    }

    ans, err := pipe.Answer(ctx, chat, "How many vacation days can I carry over at most?")
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println(ans.Text)
    fmt.Println("\nSources:")
    for _, s := range ans.Sources {
        fmt.Printf("  [%.3f] %s\n", s.Score, s.ID)
    }
}

With a real embedding model and chat model behind the clients, the model answers from the handbook chunk — not from whatever a generic HR policy sounds like — and shows its work:

The balance is capped at 30 days, so you can carry over at most 30 vacation days. [1]

Sources:
  [0.734] handbook#0
  [0.291] security#0

Note that security#0 was retrieved even though it has nothing to do with vacation — top-k always returns k results. Its low score (0.291) is the tell, and the grounding instruction is why it didn’t poison the answer.

Keeping it honest: this is baseline RAG

What you have built is real and it works, but it is deliberately the naive baseline. Its weaknesses are worth naming plainly, because post 10 exists to fix them:

The gotcha: retrieved is not the same as relevant. Search returns exactly k chunks whether or not any of them is actually on-topic — for a question your corpus can’t answer, it still hands back the k least bad vectors, and a model told to answer from context may over-trust them. The fix is to check scores: apply a similarity floor (e.g. drop hits below 0.4) and, if nothing clears the bar, return “I don’t know” without ever calling the chat model. That single threshold turns “always answers, sometimes wrong” into “declines when it should.”

Keeping it honest: this is baseline RAG
Limitation of this baseline What a better version does (post 10)
Fixed word-count chunks Semantic / structure-aware splitting
Top-k with no score floor Threshold filtering; return “don’t know” on weak hits
Raw query embedded as-is Query rewriting, hybrid keyword + vector search
Chunks injected in retrieval order Re-ranking so the best passage leads

None of that changes the shape you built. It refines each stage. Get the baseline correct first — grounded, cited, honest about ignorance — and every later improvement is a swap of one stage’s implementation, not a rewrite.

Key takeaways

Further reading

Written by Pratik Dhanave ← All posts

Find me on GitHub · LinkedIn · Acknowledgments