Sourced RAG architecture
A RAG that invents is worse than a search engine.
The real problem
A search engine that finds nothing tells you so. A badly built RAG instead produces a plausible answer with sources that do not support it. That is a reliability regression dressed up as an ergonomics win.
The trap is that the demo always works. Defects surface on questions whose answer is not in the corpus — exactly the ones the user cannot check.
How I solve it
Three non-negotiables:
- Refusal is a valid answer. If coverage of the retained passages falls below threshold, the system says it does not know. Refusal rate is a tracked metric, not a defect to hide.
- Hybrid retrieval: FTS5 full-text and vector search fused by RRF. Lexical catches what semantic misses on proper nouns and exact references.
- Cross-encoder reranking after initial recall: you pay the cost on 30 candidates, not on 1.7 million.
A trap learned in the field
Ranking documents by raw hit count makes doorstop novels beat the actual manuals. A 900-page book mechanically contains more occurrences of any term than a dense 120-page reference — without being more relevant.
Ranking is now by distinct-term coverage. Same corpus, same query, entirely different results.
Second trap, on migration: before moving 1.7 million vectors to a different embedding backend, I measured cosine similarity between old and new on the real corpus. cos = 1.0000 — so zero re-encoding needed. Without that measurement, it was several days of compute spent for nothing.
What it delivers, measured
- 25,376 documents, 1,824,420 chunks, a 20.4 GB database
- Nightly regression gate: coverage, refusal rate, 0 hallucination — a score under baseline fails the night
- 1,968 declared tests
- Cross-lingual: question in French, sources in English
Where it runs in production
- Library Brain — 100% local RAG
- Brain Distiller — Library distillation
- dream-x-world — Persistent lore engine
Your data cannot leave the building?
Describe the use case, the data involved and the available hardware. I answer personally within 48 business hours.