RAG Architecture: Components, Patterns and Trade-Offs

What a RAG architecture is

Retrieval-augmented generation (RAG) is a way to make a language model answer from your documents instead of from its training data alone. The term comes from a 2020 paper by Lewis and colleagues from Facebook AI Research, University College London and New York University. A RAG architecture is the set of components that makes this work, and it always has the same shape: two pipelines that share one index.

The ingestion pipeline runs offline, whenever documents are added or changed. It parses each document, splits it into chunks, converts the chunks into embeddings and writes them to an index together with metadata. The query pipeline runs online, on every request. It takes the user's question, retrieves the most relevant chunks the user is allowed to see, reranks them, assembles a prompt, has the model generate an answer and returns it with citations. Evaluation, tracing and caching wrap both pipelines. New to the idea itself? Start with what RAG is in AI.

RAG architecture diagram

The diagram below is the reference architecture most production systems converge on.

The ingestion side: where answer quality is set

The model can only use what retrieval hands it, and retrieval can only find what ingestion stored well.

Parsing

Parsing turns files into text the rest of the pipeline can use. Plain HTML and Markdown are easy. PDFs with multi-column layouts, tables, scanned pages, slides and spreadsheets are not. A table flattened into a stream of numbers loses the link between a value and its row and column headers, and no later stage can restore it. Decide per source type: a text extractor for clean digital files, OCR or a layout-aware parser for scans and complex PDFs, and a rule for tables. Check parser output by eye on your ugliest documents first.

Chunking

Chunks are the unit of retrieval. Too large, and one embedding has to represent several topics, so matches get vague and you spend context on irrelevant text. Too small, and a chunk loses the surrounding meaning: "the limit is 30 days" is useless without knowing which limit. The main options:

• Fixed size with overlap. Simple and predictable. A few hundred tokens with a small overlap is a common starting point. It cuts through sentences and tables without noticing.
• Structure-aware. Split on headings, sections, list items or clauses. Usually the best default for documentation, policies and contracts, because authors already grouped the content by topic.
• Small chunks, larger context. Search over small passages, then pass the model the parent section they came from. This buys precise matching and enough context, at the cost of a more complex index.

Whichever you choose, attach metadata to every chunk: source document, title and heading path, URL or page number, last-modified date, document type, language and access rights. You will filter, cite and debug with these fields.

Embeddings

An embedding model maps each chunk to a vector so that texts with similar meaning land close together. The choice affects retrieval quality, language coverage, cost and vector size. Two rules matter more than the leaderboard position of any model. First, test candidates on your own questions and documents, because rankings on public benchmarks do not transfer reliably to a specific domain. Second, treat the model as part of your data schema: vectors from different models are not comparable, so changing the model means re-embedding the whole corpus. Store the model name and version with every vector and keep the original chunk text so that a re-index is a batch job, not a rebuild.

The index

The index holds vectors for semantic search, usually an inverted index for keyword search, and the metadata for filters. Vector search at scale uses approximate nearest-neighbor algorithms such as HNSW, which give up a guarantee of finding the exact nearest matches in exchange for speed. The balance between recall and speed is tunable, which matters when a chunk you can see in the database never shows up in results. Plan the update path too: how a changed document replaces its old chunks, and how a deleted or restricted document disappears from the index.

The query side: from question to cited answer

Query processing

Raw user input is often a poor search query. In a conversation, "and what about for contractors?" means nothing without the previous turns, so a model call rewrites it into a standalone question first. Other useful steps: splitting a compound question into parts, extracting filters (a product name, a date range) and routing to the right index when you have several. Each step adds latency, so add them when your evaluation shows the failure they fix.

Vector search vs hybrid search

Vector search matches meaning. It finds a passage about cancelling a subscription when the user asks how to stop being billed. It is weaker on exact strings: part numbers, error codes, people's names, internal acronyms. Keyword search, typically BM25, is the opposite. Hybrid search runs both and merges the two result lists, often with reciprocal rank fusion, which combines rankings without needing the two scoring scales to be comparable. For business content full of identifiers and jargon, hybrid is the safer default.

Reranking

First-stage retrieval is built for speed across the whole corpus, so its ordering is rough. A reranker is a second model that reads the question and each candidate together and scores how well the passage answers it. A common starting point is to retrieve a wide set, for example 50 to 100 candidates, and keep the best 5 to 10 after reranking, then tune both numbers on your evaluation set. Reranking can be one of the largest quality gains you can add to a basic pipeline. It costs extra latency and a per-query fee or GPU time.

Prompt assembly

Prompt assembly decides what the model reads: instructions, the retrieved passages with source identifiers, conversation history and the question. More context is not better. Research on long contexts, including the 2023 "Lost in the Middle" study by Liu and colleagues, found that models use information at the start and end of the context more reliably than information in the middle. So send fewer, better passages, put the strongest ones first, remove near-duplicates and set a token budget. Tell the model what to do when the passages do not contain the answer: say so, and do not fill the gap from memory.

Generation and citations

Ask the model to cite the source ID of each passage it used, then resolve those IDs to titles and links in your own code. Citations let users verify an answer and let you audit one. They are not proof of correctness: a model can attach a real source to a claim the source does not make. For answers where an error is costly, add a check that each cited passage supports the sentence it is attached to, and treat "no relevant passage found" as a valid outcome with its own response. Our guide to RAG hallucinations covers these checks in detail.

The parts that are not on the happy path

Permissions

If your corpus contains anything not every user may read, access control belongs in retrieval. Store access rights (users, groups, tenant) on every chunk at ingestion and apply them as a filter inside the search query, using the identity of the person asking. A chunk that is never retrieved cannot leak. A prompt instruction such as "do not reveal salary data" can. The hard part is sync: when someone loses access in the source system, the index has to reflect it quickly.

Caching

Three layers are worth considering. Cache embeddings so unchanged chunks are not re-embedded on every ingestion run. Use the model provider's prompt caching for the stable prefix of your prompt if your provider offers it. Cache full answers only for frequent, non-personal questions, and make the cache key include the user's permission scope and the index version. An answer cache that ignores permissions is a data leak.

Evaluation

Measure the two halves separately. Retrieval metrics ask whether the right passages were found: recall at k, precision, rank of the first relevant result. Generation metrics ask whether the answer is faithful to the passages, relevant to the question and correct. A wrong answer with the right passages retrieved is a prompt or model problem. A wrong answer with the wrong passages is an ingestion or search problem. Without the split you cannot tell which to fix. As a starting point, collect 50 to 200 real questions with expected answers and source passages, and run them on every change. More in our guide to RAG evaluation.

Observability

Log each request as a trace: the original and rewritten query, the filters applied, the IDs and scores of retrieved and reranked chunks, the final prompt, the answer, token counts, latency per stage and user feedback. This trace is how you find out which box in the diagram failed. These logs contain customer data, so give them access controls and a retention period.

RAG architecture patterns compared

Four patterns cover most systems in production. Each adds capability and cost to the one before.

Agentic RAG architecture: where an LLM agent fits

In an agent RAG architecture, retrieval stops being a fixed step and becomes a tool. The LLM agent receives the question, decides whether to search at all, writes its own queries, chooses between sources (the document index, a SQL database, an API), reads the results and decides whether to answer or search again. The agent sits where "query processing" was in the diagram and can loop through retrieve and rerank several times.

That loop is the benefit and the cost. A question like "which of our enterprise customers are on a contract that predates the new data policy" needs a database lookup and a document search, which a single retrieval pass cannot do. But each loop adds model calls, so latency and cost per question vary, and a test that passed yesterday can take a different path today. Cap the number of steps, log every tool call and keep the simple path for simple questions. See what a RAG agent is and RAG vs agentic AI for the full comparison.

How to choose your architecture

1. Collect real questions firstGather the questions your users actually ask (50 to 200 is a workable starting set), with the correct answers and the documents that contain them. This set tells you which pattern you need and later becomes your evaluation set.
2. Classify the questionsSingle-fact lookups point to hybrid search with reranking. Questions that connect many documents point to graph-based retrieval. Multi-step questions and those needing live data point to an agent.
3. Audit the sourcesList formats, volume, update frequency and who may read what. Scanned PDFs, heavy tables and per-document permissions each add components before any model is involved.
4. Build the baseline and measure itStart with structure-aware chunking, hybrid search and a reranker. Score retrieval and answers separately against your question set.
5. Add complexity only where the numbers ask for itQuery rewriting, a graph, an agent loop: add each one to fix a measured failure, and confirm the score moved.

Built by Gilzor

Results we’ve shipped

70+products launched

98%delivered on time

85%clients come back

5.0 ★on Clutch

Aviation33%faster baggage weigh-inWeb and Mobile application for Airlines→E-commerce · AITaking an AI-built e-commerce SEO platform to production→Healthcare+30%lab test ordersMedical Lab Web and Native Mobile Applications→

Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.

See how we’d approach yours →

Mistakes that are expensive to undo

• Adding permissions after launch. If access rights are not in the chunk metadata, adding them means new connectors, a full re-index and a review of every cache and log. Design them in, even if the first corpus is public.
• Throwing away structure at ingestion. If you store only chunk text, you cannot cite a page, filter by date or re-chunk without parsing everything again. Keep the parsed document, the heading path and the source reference.
• Coupling to one embedding model without a re-index path. Models get replaced and retired. Without stored source text and a versioned index, a model change becomes a migration project.
• No evaluation set. Every later decision, from chunk size to model choice, becomes opinion.
• Starting with the most complex pattern. A graph or a multi-agent setup built before a measured baseline hides basic retrieval defects under layers that are slow to debug. Earn each layer.
• Ignoring deletes and updates. An index that only ever grows will eventually answer from a withdrawn policy. Document removal and permission revocation need a tested path from day one.

RAG architecture review checklist

Tick what is already true for your design or your running system.

FAQ

Where Gilzor fits

Architecture decisions are cheapest before the first index is built. Gilzor offers RAG development services: we design and build custom RAG systems, add retrieval to existing products, and review architectures that already run, with RAG consulting for teams that want a second opinion on patterns, components and evaluation before they commit. Our AI work includes an AI payroll platform that extracts data from PDFs and cut salary calculation errors by 17%, and image and text search with recommendations for the QLode fashion commerce product. The team works from Poland with partial overlap with the US working day, and a 2-week fixed-price Discovery Sprint is a practical way to turn your questions, sources and access rules into an architecture and a scoped first version.