RAG Systems
Retrieval-augmented generation: embedding strategy, vector stores, and context management.
Chunking is the decision that matters most
Most RAG quality problems trace back to chunking, not the model or the vector store. Chunks that split a table row, a code block, or a legal clause in half will retrieve cleanly and answer wrong. We chunk by document structure first (headings, sections, list boundaries) and only fall back to fixed-size windows with overlap when structure isn't available.
Retrieval beyond a single similarity search
Pure cosine-similarity search over one index handles the easy 70% of queries. The remaining 30% — questions that need an exact figure, a recent update, or a cross-document comparison — usually need hybrid retrieval: a keyword/BM25 pass alongside vector search, re-ranked by a cheaper cross-encoder before the top results reach the generation prompt.
- Hybrid search (vector + keyword) beats vector-only on numeric and named-entity queries
- Re-ranking the top ~20 candidates down to ~5 before generation improves answer precision more than a bigger embedding model does
- Metadata filters (date, source, department) cut irrelevant retrieval before similarity scoring runs at all
Keeping answers grounded
We require the generation step to cite which retrieved chunk supports each claim, and we validate that cited chunk IDs actually exist in the retrieved set before returning an answer. This turns 'the model might be making it up' into a checkable property of the response.
Working through something like this?
Get a scored blueprint in the Architecture Lab, or talk it through with the Architect Assistant first.