A question that keeps coming up in my work: when should a system use RAG (Retrieval-Augmented Generation), and when should it just put the relevant content directly in the prompt?
After about six months of building both approaches into different systems, here is the rough decision framework I use:
Direct context wins when the relevant information is small and stable. If the content you need in the prompt fits in 4-8K tokens and does not change often, put it in a system prompt and call it a day. You will have faster responses, simpler architecture, no retrieval quality issues, and no vector database to maintain.
RAG wins when the information is large, changing, or user-specific. A large knowledge base, a per-customer dataset, or content that updates daily — these are RAG territory. The retrieval latency and infrastructure complexity are worth it because the alternative is unmaintainable.
Hybrid wins more often than either pure approach. In practice, many of my production systems use a small stable system prompt (identity, rules, common patterns) combined with RAG for dynamic content. Both techniques have different strengths; using them together is not a compromise, it is a better architecture.
The failure mode I see most often: teams defaulting to RAG for everything because it is the sophisticated-sounding choice. For small, stable knowledge bases, RAG adds complexity without adding value. Start simple. Add retrieval only when the simple approach genuinely fails.
Leave a Reply