RAG starts with source quality
If documents are duplicated, obsolete, contradictory, or missing metadata, retrieval will faithfully surface bad information. Source ownership and freshness rules matter before prompt tuning.
The ingestion pipeline should preserve identifiers, dates, document type, access metadata, and enough structure to trace an answer back to the source.
Retrieval has to be measured
Build a representative set of real questions and expected source material. Measure whether the correct chunks are retrieved before blaming the language model for a bad answer.
Hybrid search, reranking, chunking strategy, metadata filters, and query rewriting can be compared against the same evaluation set.
Permissions come before prompts
A useful answer is still a security incident if the user should not have seen the source. Access control has to be applied during retrieval, not only described in a system prompt.
The design should also log which sources were used so access issues can be audited.
When not to use RAG
Do not add RAG when the answer can be produced deterministically from a database query or API, when source content is too unstable to govern, or when the workflow requires strict transactional correctness rather than natural-language synthesis.
Sometimes search plus a conventional interface is the better product.
When NOT to use an autonomous agent
If an action can move money, delete data, approve a regulated decision, or affect a customer account, broad autonomous tool access may be the wrong default. Use explicit tools, narrow permissions, validation, and human approval for sensitive steps.
Automation value does not require giving the model unrestricted authority.
The item that is almost always missing: evaluation
Without repeatable evaluations, every prompt or model change becomes a subjective demo. A small but representative evaluation suite allows the team to compare versions and catch regressions before users do.
Pair offline evaluation with production traces, user feedback, latency, cost, and escalation rates.
Frequently asked questions
Does RAG eliminate hallucinations?
No. It can ground answers in retrieved sources and make evidence visible, but the system still needs evaluation, constraints, and fallback behavior.
Do we need a vector database?
Not always. The retrieval design depends on scale, metadata, latency, existing search infrastructure, and whether lexical or hybrid search performs better on real questions.