RAG Done Right: Grounded Answers From Your Data

Why most RAG systems give wrong answers

Retrieval-Augmented Generation (RAG) is the standard pattern for building AI systems that answer questions from your own data. The idea is simple: retrieve relevant documents, feed them to an LLM, and generate a grounded answer.

In practice, most RAG implementations fail for one reason: bad retrieval. If the wrong chunks come back from the database, no model can save the answer. The LLM will confidently hallucinate based on irrelevant context, or miss the answer entirely because it was never retrieved.

Building RAG that works means investing in the retrieval pipeline, not just the model.

Chunking strategy matters more than you think

The default approach of splitting documents into fixed-size chunks of 500-1000 tokens works for prototypes. It fails for production.

Better chunking strategies:

• Semantic chunking: Split at natural boundaries like headings, paragraphs, or topic shifts instead of token counts. Each chunk represents one coherent idea.
• Contextual chunking: Include document metadata (title, section, date, source) in each chunk so the retriever and model know where the information came from.
• Overlapping chunks: For technical documents, overlap chunks by 10-20% to avoid splitting critical information across chunk boundaries.
• Hierarchical chunking: Store both small chunks (for precise retrieval) and larger parent sections (for context). Retrieve small, expand to large.

The right strategy depends on your data. Financial documents need different chunking than support tickets or legal contracts.

Embeddings are not one-size-fits-all

The embedding model determines how well your queries match your documents. A general-purpose embedding model will not capture domain-specific semantics.

What to consider:

• Domain fine-tuning: Fine-tune embeddings on your domain data so technical terms, acronyms, and jargon map to the right vectors.
• Query expansion: Before embedding the user query, expand it with synonyms, related terms, or a rephrased version to improve recall.
• Hybrid search: Combine dense vector search with traditional keyword (BM25) search. Vectors capture semantics. Keywords capture exact matches like product names, error codes, and IDs.

In our projects, hybrid search consistently outperforms pure vector search by 15-25% on recall metrics.

Re-ranking is non-negotiable

Initial retrieval should cast a wide net. Get 20-50 candidate chunks, then re-rank them with a cross-encoder model that considers both the query and chunk together.

Why re-ranking matters:

• The initial embedding search is fast but approximate. It prioritizes speed over precision.
• A cross-encoder re-ranker is slower but much more accurate at determining true relevance.
• The combination gives you both speed (fast initial retrieval) and accuracy (precise final selection).

We typically retrieve 30 chunks and re-rank to the top 5-8 for the LLM context. This single step improves answer accuracy more than any model upgrade.

Always cite sources

Every answer should include citations linking back to the source document and section. This serves three purposes:

• Trust: Users can verify the answer against the original source.
• Debugging: When answers are wrong, citations show exactly which retrieval led the model astray.
• Honesty: If the system cannot find a relevant source, it should say so instead of making something up.

We implement citations by passing chunk metadata (document title, page, section) into the LLM prompt and requiring citations in the output format.

Build an evaluation harness

You cannot improve what you do not measure. A RAG evaluation harness should track:

• Retrieval metrics: Precision, recall, and MRR (Mean Reciprocal Rank) for your retrieval pipeline against a labeled test set.
• Generation metrics: Faithfulness (is the answer grounded in the retrieved context?), relevance (does it answer the question?), and completeness.
• End-to-end metrics: User satisfaction, correction rate, and time to correct answer.

Build a test set of 100+ question-answer pairs from your real data. Run your RAG pipeline against this set after every change. Track trends over time.

The evaluation-first approach

We build RAG systems the same way we build all software: test-first.

1. Create the test set. Collect real questions, expected answers, and source documents.
2. Build the simplest pipeline. Basic chunking, standard embeddings, no re-ranking.
3. Measure the baseline. How bad is “simple”? This tells you where to invest.
4. Improve iteratively. Better chunking, hybrid search, re-ranking, query expansion. Measure after each change.
5. Stop when it is good enough. Perfect is the enemy of production.

Need RAG that actually works?

We build production RAG systems for document processing, knowledge management, and compliance. If your current RAG demo is not translating to real results, we can help.

Book a free strategy call with our engineering team.

Chunking Strategy Is Everything

The biggest factor in RAG quality is how you chunk your documents. We’ve tested three approaches across dozens of projects: fixed-size (fast, decent), sentence-based (better coherence) and semantic chunking (best quality, slower). For most use cases, recursive character splitting with 512-token chunks and 50-token overlap gives the best balance.

Don’t Skip Reranking

Vector similarity search gets you 80% of the way. A cross-encoder reranker (like Cohere Rerank or a fine-tuned MiniLM) takes you to 95%. The two-stage approach — fast retrieval with vector search, then precise reranking of top-K results — is the standard pattern we use in production.

Evaluation Framework

We use the RAGAS framework to measure: context precision (are we retrieving the right chunks?), context recall (did we miss anything?), faithfulness (is the answer grounded in the context?) and answer relevance (does it actually answer the question?). Build your eval dataset first, then build your RAG system.