What Is RAG in AI? Retrieval-Augmented Generation Explained
What RAG means, in plain terms
A language model knows two kinds of things: what it absorbed during training and what is in the prompt in front of it. Training knowledge is broad but frozen at a cutoff date, and it never included your contracts, your help center or last week's price list. Ask about those and the model either says it doesn't know or, worse, produces a fluent guess.
RAG closes that gap through the prompt. The three words describe the three moves:
- Retrieval. A search step finds the passages in your content that are most relevant to the question.
- Augmented. Those passages are inserted into the prompt next to the question, usually with an instruction to answer only from them and cite them.
- Generation. The model writes the answer, now with the facts in front of it.
The usual analogy is an open-book exam. The model still does the reading and the writing, but it works from the right pages instead of from memory. The term comes from a 2020 paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", by Patrick Lewis and co-authors at Facebook AI Research, University College London and New York University. The original work trained the retriever and the generator together. What the industry calls RAG today is looser: any system that retrieves content at question time and feeds it to a general-purpose model.
When people say "RAG in generative AI", they mean this pattern applied to chat assistants, search, copilots and document tools. It is the default way to make a general model useful on private or fast-changing knowledge.
How RAG works, step by step
A RAG system has two halves. Indexing happens ahead of time, whenever content changes. Answering happens on every question, usually within seconds.
- Collect and parse the sourcesPull content from where it lives: PDFs, wikis, help centers, ticket systems, databases. Extract clean text and keep the structure that carries meaning, such as headings, tables and page numbers. Poor parsing of PDFs and tables is a frequent cause of wrong answers later.
- Split into chunksCut each document into passages along natural boundaries such as sections or paragraphs; a few hundred tokens per chunk is a common starting point, to be tuned on your own test questions. Chunks that are too small lose context; chunks that are too large bury the relevant sentence and waste prompt space. Attach metadata to each chunk: source, date, product, language and who may see it.
- Embed and indexAn embedding model turns each chunk into a vector, a list of numbers positioned so that texts with similar meaning sit close together. Store the vectors in an index, usually next to a classic keyword index over the same chunks.
- Retrieve for each questionEmbed the user's question, find the nearest chunks, and combine that with keyword matches. This combination is called hybrid search, and it matters because vectors catch paraphrases while keywords catch exact terms such as part numbers, error codes and names. Filter by the user's permissions before anything else.
- Rerank and trimA reranking model reads the question together with each candidate passage and scores how well it actually answers. Keep the top handful. This step can improve answer quality more than switching to a larger language model.
- Generate with instructionsPut the question and the chosen passages in the prompt, with rules: answer only from the sources, cite them, and say so when they do not contain the answer.
- Check, cite and logShow the sources next to the answer so users can verify it. Log the question, the retrieved passages and the response, because that record is how you find out which half failed when an answer is wrong.
The component-level view, with the design choices at each stage, is in our RAG architecture guide. If you plan to build on a framework, see RAG with LangChain and our comparison of the best RAG tools.
What RAG fixes and what it does not
RAG is popular because it solves several problems at once without retraining a model. It also has limits that no amount of prompt tuning removes.
| RAG helps with | RAG does not help with |
|---|---|
| Private knowledge. The model can answer from documents it never saw in training. | Bad sources. Outdated, contradictory or missing documents produce outdated, contradictory or missing answers. |
| Freshness. Update the index and the next answer reflects it. No retraining. | Aggregation. "How many contracts expire next quarter?" needs a database query, not five retrieved passages. |
| Verifiability. Answers can cite the passages they came from. | Whole-corpus reasoning. "Summarize the themes across all 4,000 tickets" does not fit in a handful of chunks. |
| Access control. Retrieval can filter by user, so one model serves people with different permissions. | Behavior. Tone, output format and task skill come from prompts or fine-tuning, not from retrieved facts. |
| Fewer invented facts. The model has the real text in front of it. | All hallucinations. The model can still misread a passage or answer from memory when retrieval comes back empty. |
The right-hand column is why mature systems rarely stop at plain RAG. Aggregate questions get routed to SQL or an analytics tool. Multi-step questions go to an agent that can search several times and call other tools; our articles on RAG agents and RAG vs agentic AI cover that step. For connecting a model to live systems and actions, as opposed to documents, see RAG vs MCP. And for the remaining wrong answers, RAG hallucinations explains where they come from and how to reduce them.
RAG vs fine-tuning vs long context
These three are the usual candidates when a general model doesn't know enough about your business. They are not interchangeable, and the decision rule is short: if the model lacks facts, retrieve them; if it lacks a behavior, train it; if the facts are few and stable, just put them in the prompt.
| RAG | Fine-tuning | Long context | |
|---|---|---|---|
| What it does | Searches your content per question and adds the best passages to the prompt | Trains the model further on your examples, changing its weights | Puts the whole document set into every prompt |
| Best for | Large, changing or permissioned knowledge | Consistent format, tone or a narrow task at high volume | A small, stable set of documents; one-off analysis of a long file |
| Keeping knowledge current | Re-index the changed documents | Retrain | Edit the prompt content |
| Citations | Natural: each passage has a source | Not available: knowledge is blended into weights | Possible, but harder to verify across a long prompt |
| Per-user access control | Yes, by filtering retrieval | No, everyone gets the same model | Only by assembling a different prompt per user |
| Cost per request | Low to moderate: a few passages plus a search | Low: short prompts, sometimes a smaller model | High: you pay for the full corpus in tokens every time, less with prompt caching |
| Upfront work | Ingestion pipeline, index, evaluation | Labeled training data, training runs, re-evaluation on every base model change | Almost none |
| Main risk | Retrieval misses the right passage | Knowledge goes stale; quality regressions are hard to trace | Cost, latency and missed details as the prompt grows |
The options combine well. A common production setup retrieves passages with RAG, uses a generous context window so it can pass more of them, and fine-tunes a small model only for a narrow sub-task such as classifying incoming questions. A practical order is to start with long context while the corpus is small, move to RAG when cost, latency or permissions start to hurt, and consider fine-tuning last. The detailed comparison, with cost examples, is in RAG vs fine-tuning.
The components of a RAG system
If you are scoping a build or reading a vendor proposal, these are the parts you should expect to see named. A proposal that only mentions "a vector database and an LLM" is describing a demo.
| Component | What it does | Decision to make |
|---|---|---|
| Connectors and parsers | Pull content from sources and extract clean text, tables and structure | Which sources first; how to handle scans, tables and attachments |
| Chunker | Splits documents into retrievable passages with metadata | Chunk size and boundaries; what metadata to keep |
| Embedding model | Turns text into vectors for meaning-based search | Hosted or self-hosted; language coverage; changing it later means re-embedding everything |
| Index | Stores vectors and keywords and searches them fast | A vector extension in your existing database, your search engine, or a dedicated vector database |
| Retriever and reranker | Finds candidates, applies filters, orders them by relevance | Hybrid search or vector only; how many passages to pass on |
| Language model and prompt | Writes the answer from the passages | Which model; how strict the "answer only from sources" rule is |
| Access control | Ensures users only retrieve what they may open | How source permissions are synced into the index |
| Evaluation and monitoring | Measures retrieval and answer quality before and after release | Who owns the test set and the release gate |
Product teams adding this to a multi-tenant application have extra concerns, such as tenant isolation and per-customer indexes, covered in RAG pipeline for SaaS. If you would rather buy than assemble, compare enterprise RAG platforms and RAG-as-a-service providers first.
Where RAG fails
When a RAG answer is wrong, the cause is usually in one of five places. Knowing them lets you ask the right question in a review: was the right passage retrieved or not?
- The passage was never retrieved. The answer exists in the corpus, but search ranked other chunks higher. Typical causes are vector-only search missing exact terms, questions phrased very differently from the documents, and no reranking.
- The chunk lost its meaning. A table was split from its header, or a clause from the definition it depends on. The retrieved text is technically relevant and practically useless.
- The index is stale or duplicated. Three versions of a policy are indexed and the model quotes the oldest. Deleted documents still answer questions.
- The model ignored or misread the sources. The right passage was in the prompt, and the answer still contradicts it or adds unsupported detail. Stricter instructions, fewer and better passages, and a check of answers against sources reduce this.
- Security was treated as a prompt. Permissions enforced by telling the model "don't reveal HR documents" will leak. So will a system that follows instructions hidden inside a retrieved document, which is known as indirect prompt injection. Filter in retrieval, and treat retrieved text as data, never as commands.
A useful evaluation set has real questions, the expected answer and the passages that contain it. It lets you score retrieval (did the right passage reach the prompt?) apart from generation (is the answer supported by the passages, and is it correct?). Without that split, a team can spend weeks tuning prompts to fix what is actually a search problem. Our RAG evaluation guide covers metrics and tooling.
Built by Gilzor
Results we’ve shipped
Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.
What RAG costs, at a glance
RAG has a build cost and a running cost, and they behave differently. The build is mostly engineering time. The running cost scales with usage and corpus size.
| Cost line | When you pay | What drives it |
|---|---|---|
| Ingestion and parsing | Build, then on every content change | Number and messiness of sources; scans and tables cost the most effort |
| Embeddings | Once per chunk, again if you change the embedding model | Corpus size; usually a small line compared with generation |
| Index hosting | Monthly | Number of vectors, replicas, managed or self-hosted |
| Generation | Per request | Model tier, passages per prompt, answer length, request volume |
| Reranking and guardrails | Per request | Extra model calls; they add latency as well as cost |
| Evaluation and maintenance | Ongoing | Someone reviewing samples, growing the test set and re-running it on every change |
For the build, our 2026 estimates with a Central/Eastern European or Latin American vendor are about $15,000–40,000 for a prototype on one source, $40,000–120,000 for a production system for one use case with permissions, evaluation and monitoring, and $120,000–300,000 and more for a multi-source enterprise system. US onshore agencies usually quote about 2–2.5 times as much. These are estimates, not quotes. The biggest swing factors are the number of sources, how strict access control must be, and whether the system runs in your own infrastructure. Our RAG development cost guide breaks the numbers down, and RAG development walks through the project phases.
Is RAG the right fit? A quick check
Tick what is true for your use case. The more boxes you tick, the stronger the case for RAG over a plain model call or fine-tuning.
RAG fit check
Typical products that tick most of these boxes are customer support assistants, internal knowledge search, sales and onboarding copilots, and contract or policy lookup. RAG use cases goes through them by industry with what each needs to work.
Where to go next
This page is the overview. Each topic has its own guide:
- Design: RAG architecture, RAG pipeline for SaaS, RAG with LangChain.
- Choosing an approach: RAG vs fine-tuning, RAG vs agentic AI, RAG vs MCP.
- Agents: RAG agents, multi-agent RAG, RAG agent frameworks.
- Quality: RAG evaluation, RAG hallucinations.
- Budget and delivery: RAG development cost, RAG development, RAG use cases.
- Tools and vendors: RAG tools, enterprise RAG platforms, RAG-as-a-service providers, RAG development companies.
FAQ
What does RAG stand for in AI?
What is RAG in simple terms?
Is RAG the same as a large language model?
Does RAG stop hallucinations?
Do I need a vector database for RAG?
Is RAG still needed now that models have very long context windows?
How much does it cost to build a RAG system?
Where Gilzor fits
If this overview left you with "we probably need RAG, but for which questions and over which sources?", that is the right place to start a conversation. Gilzor builds custom RAG systems, RAG applications and chatbots, adds RAG to existing products, and consults on and evaluates systems that are already running. Our AI work includes an AI payroll platform with PDF data extraction that cut salary calculation errors by 17% and QLode, an AI fashion commerce product with image and text search. We have launched 70+ products, and 85% of clients come back. The team works from Poland with partial overlap with the US working day, and a 2-week fixed-price Discovery Sprint is the usual way to turn a use case into a scoped first version. See our RAG development services, or RAG consulting if you want a second opinion on an approach before you build.
No sales pitch
Get a straight answer for your project
Tell us what you’re building. We’ll reply with options, a rough cost and timeline. If we’re not the right fit, we’ll say so.
CTO of Gilzor. Responsible for architecture and the engineering standards our teams work by.
LinkedIn →Gilzor · AI/ML Solutions partner
Need a team for your AI product?
Services
AI/ML SolutionsComputer vision, NLP, predictive analytics, automation.→By company type
Selected projects
The team behind them
