Most RAG Problems Are Retrieval Problems

A support assistant returns a confidently wrong answer. The team's first move is to rewrite the system prompt. Their second is to try a bigger model. Both are usually wasted effort, because the model answered correctly given what it was shown. It just wasn't shown the right passage.

Before you touch the prompt, check whether the correct source document made it into the context at all. In our experience that single question explains the majority of RAG failures.

Measure retrieval separately

You cannot debug a pipeline you only measure end to end. Split the metrics:

Recall@k: how often the passage containing the answer appears in the top k retrieved chunks. If this is 0.7, your ceiling is 70% no matter which model you use.

Groundedness: given the retrieved context, does the answer stick to it, or invent? This is the model's responsibility.

Answer accuracy: the end-to-end number everyone quotes, and the least diagnostic of the three.

Build a set of fifty real questions with the passage that answers each one, and score recall first. If recall is low, nothing downstream can save you.

Chunking is where most recall is lost

The default, splitting every document into 512-token windows with a bit of overlap, is convenient and frequently wrong. It cuts tables in half, separates a heading from the paragraph it governs, and strips the context that made a sentence meaningful.

Chunk along the document's real structure instead. Markdown and HTML give you headings; use them, and prepend the heading path to each chunk so a fragment carries its own context. Keep tables intact even when they exceed your target size, because half a table is worse than none. For transcripts and tickets, split on speaker turns rather than character counts.

Attach metadata while you have it: source, section, date, author, permissions. You will want to filter on all of it later.

Hybrid search, always

Pure vector search is excellent at paraphrase and terrible at exact identifiers. Ask about error code E-4471 or part number AX-9, and embeddings will happily return chunks about similar-sounding codes.

Run keyword search alongside vector search and fuse the results. This is a few hours of work and routinely moves recall by ten points or more, because the two methods fail on different queries. Real corpora contain product names, SKUs, and acronyms, and those are exactly where semantic similarity breaks down.

Rerank

Retrieval gets you a rough candidate set. A cross-encoder reranker reads the query and each candidate together and reorders them properly. Retrieve twenty, rerank, keep five.

Per hour of engineering, this is the highest-leverage change available in most RAG systems. It also lets you send less context to the model, which improves both accuracy and cost, because long contexts dilute attention, and a precise five beats a mediocre twenty.

Filter permissions at query time

An assistant that can surface a document its user could not open directly is a data breach with a friendly interface.

Enforce access control inside the retrieval query, not by asking the model to be discreet. Mirror your existing identity model, index the permission metadata, and filter before the reranker ever sees a candidate. Post-hoc filtering of a generated answer is not a control. By then the content has already been read into the context.

Keep the index honest

An index that drifts from the source of truth produces answers that are confidently out of date. Two requirements:

Deletions must propagate. If a document is removed or unshared, it has to leave the index. This is a compliance obligation, not a nicety.

Updates must be incremental. Re-embedding an entire corpus nightly is expensive and slow; sync deltas from webhooks or change tracking, and monitor freshness as a metric with an alert on it.

Cite, and check the citations

Every claim should point to the passage that supports it, for two reasons. Users can verify, which is where trust actually comes from. And you can grade automatically, with a cheap, deterministic check that flags any statement whose cited source doesn't contain it catches a large class of quiet failures.

The order to work in

If retrieval quality is poor, work in this sequence: fix chunking, add hybrid search, add a reranker, filter permissions properly, then verify freshness. Only after all of that does prompt engineering start to pay, and only then is a bigger model worth its cost.

The unglamorous truth is that RAG is mostly a search problem wearing an AI costume. Teams that treat it that way ship assistants people trust.