What is Retrieval-Augmented Generation (RAG)?

Definition

Retrieval-Augmented Generation (RAG) is an architecture pattern in which a system retrieves relevant passages from an approved knowledge base—such as internal wikis, policies, or support tickets—and passes them as context to a language model. The model then generates an answer that cites or reflects that material instead of relying solely on weights learned during training.

The name captures two phases: retrieval (find the right evidence) and generation (turn evidence into a helpful answer). Neither replaces the other. Weak retrieval caps quality no matter how strong the model is; weak generation wastes good passages behind vague or non-committal wording.

Why organizations use RAG

• Grounding: Responses can be tied to specific documents or chunks, which supports fact-checking, internal citations, and audit trails.
• Freshness: Updating the index refreshes what the application can surface without retraining the base large language model.
• Domain fit: Proprietary terminology, product names, and procedures appear when they exist in the corpus—areas where generic pretrained models are often thin.
• Control: Access rules, sensitivity labels, and document classes can be enforced at retrieval time, not only at the model API boundary.

How the pipeline is usually structured

Typical building blocks include:

• Ingestion: Connectors pull content from file shares, knowledge bases, ticketing systems, or approved web crawls.
• Chunking: Documents are split into segments small enough to fit context windows and large enough to preserve meaning; metadata (source, owner, classification) travels with each chunk.
• Indexing: Chunks are embedded into a vector index, often combined with hybrid search (keyword plus semantic) to handle exact identifiers like SKU codes alongside conceptual questions.
• Query: The user question (and sometimes conversation history) is embedded or reformulated; top-k passages are retrieved with filters for tenant, team, or clearance.
• Prompt assembly: System instructions and retrieved evidence are separated so the model can treat evidence as attributable text, not as override instructions.
• Inference and logging: On-premises or private-cloud runtimes generate the answer; logs record which chunk IDs were used for traceability.

Enterprise considerations

Production RAG is as much about governance and ownership as about vectors. Clear answers are needed for: who may add a corpus, how often it reindexes, how retention and legal holds apply to source documents, and how to handle access revocation when a user leaves a project. Without that, assistants can silently serve stale policy text or over-retrieve from low-trust sources.

Teams also align RAG with identity: retrieval should mirror the same folder, space, and ticket permissions a human would have in the source system. Treating the vector index as a flat global store is a common source of compliance incidents.

Limits to plan for

RAG does not guarantee correctness. If the corpus is incomplete, outdated, or internally inconsistent, the model can still sound confident. Poor chunk boundaries (splitting tables mid-row, merging unrelated topics), missing metadata, or weak relevance scoring can surface the wrong passages at the top.

Security teams treat RAG as an application attack surface: retrieved text can carry indirect prompt injection (instructions hidden in documents). Defenses include trust tiers for sources, chunk-level hygiene, prompt separation patterns, and policy layers on downstream tools—not only perimeter network controls.

When RAG is the right tool

RAG fits internal assistants, support copilots, compliance Q&A, and operational search where answers should trace to source material. It is a weaker fit when the task is pure reasoning with no stable document base, when latency budgets are extremely tight, or when the knowledge is fundamentally dynamic and never written down—in those cases teams often combine RAG with specialized models, caching, or human workflows.

Summary

RAG is best understood as search plus generation under governance: retrieval constrains where information may come from; the model explains and synthesizes within that boundary. Done well, it aligns generative AI with enterprise expectations for provenance, control, and auditability.