Retrieval-augmented generation (RAG) is pitched as the fix for LLM hallucination: ground the model in your own documents, and it can only answer from what's true. In practice, most RAG chatbots we're brought in to fix still hallucinate - just less obviously.
The root cause is almost always retrieval quality, not the model. Chunking documents by fixed character counts breaks context mid-sentence. Embedding models tuned for general web text miss domain-specific terminology. And most implementations retrieve the top-k chunks by similarity alone, with no re-ranking step to catch cases where the most similar chunk isn't the most relevant one.
Our fix is usually three changes: semantic chunking that respects document structure (headings, tables, sections), a re-ranking model between retrieval and generation, and an explicit 'I don't know' path when retrieved context doesn't actually answer the question - so the model stops filling gaps with plausible-sounding fiction.
We also instrument every RAG deployment with retrieval-quality monitoring, not just output monitoring. If the wrong chunks are being retrieved, no amount of prompt engineering on the generation side will fix it - you're solving the wrong half of the pipeline.
