RAG Agent: What It Is and How Agentic RAG Works

What is a RAG agent?

Retrieval-augmented generation (RAG) gives a language model facts it was not trained on by searching your documents or data and placing the results in the prompt. If the term is new, start with what RAG is in AI. In the classic design, your code runs the search. The model only sees the results and writes the answer.

A RAG agent moves that control to the model. Retrieval becomes a tool the model can call, the same way it can call a calculator or an API. On each turn, the model reads the question and everything gathered so far, then picks the next action: answer now, search a knowledge base, rewrite the query and search again, query a database, or check a draft answer against the sources. Your code executes the action, returns the result, and the model decides again. People also call this agentic RAG. "RAG AI agent" and "AI agent with RAG" mean the same thing.

In practice, a RAG agent can do four things a fixed pipeline cannot:

• Decide whether to retrieve. "Thanks, that helps" needs no search. "What is our refund window for annual plans?" does.
• Choose where to look. Product documentation, a ticket history, a SQL database and the public web hold different kinds of answers.
• Change the query. Users write short, vague or multi-part questions. The agent can rephrase them, split them and add terms the documents actually use.
• Judge the results and its own answer. If the retrieved passages are off topic, or the draft makes a claim no passage supports, the agent can try again instead of answering anyway.

RAG agent vs a fixed RAG pipeline

The difference is who owns the control flow. In a pipeline, the developer wrote it in advance. In an agent, the model chooses it at run time, inside limits the developer sets.

This is a spectrum, not two camps. A pipeline with one fixed query-rewriting step is slightly agentic. A system where the model plans ten steps across five tools is fully agentic. Most production systems sit in between: a mostly fixed flow with one or two points where the model makes a decision.

How agentic RAG works: the loop

Under every RAG agent is the same loop. Your code sends the model the question, the conversation so far and a list of tool definitions (a name, a description and the parameters for each). The model replies with either a final answer or a request to call a tool. Your code runs the tool, appends the output to the conversation and calls the model again. The loop ends when the model answers or when your code stops it.

Three details decide how well this works. First, the tool descriptions are prompts. The model picks a source by reading its description, so "search_docs: searches documents" produces worse routing than a description that says what the index contains, what it does not, and what a good query looks like. Second, permissions belong inside the tool. The search function must filter by the current user's access rights before anything reaches the model. Third, the stopping rule belongs in code. A prompt that says "use at most three searches" is a suggestion. A counter is a guarantee.

Four common agentic RAG patterns

You do not have to adopt a fully autonomous agent. These four patterns can be added to a pipeline one at a time, and each fixes a specific failure.

1. Router

A router is one decision at the start: does this question need retrieval, and from where? A small, fast model or a classifier can make it. Routing fixes two problems: wasted searches on messages that need none, and wrong answers caused by searching the wrong source, such as looking for last month's revenue in a documentation index when the number lives in a database. The Adaptive-RAG paper (Jeong and colleagues, 2024) describes a version that classifies questions by complexity and sends them to no retrieval, single-step retrieval or multi-step retrieval.

The risk is that a routing mistake is silent. If the router says "no retrieval needed" for a question about your refund policy, the model answers from its general knowledge and nothing downstream catches it. Test the router on its own, with a labeled set of questions and the route each should take.

2. Query rewriting and decomposition

Users do not write good search queries. In a conversation, "and what about for enterprise?" means nothing without the previous turns. A rewriting step turns it into a standalone query, such as "refund policy for enterprise annual plans". Decomposition goes further and splits "compare the data retention terms in our 2024 and 2026 contracts" into two searches, one per contract, because a single search tends to return passages from only one of them.

Rewriting is the cheapest pattern and often the first one worth adding. Its failure mode is drift: the rewritten query quietly drops a constraint (a date, a product name, a negation) and the search returns confident, relevant-looking passages about the wrong thing. Log the original and the rewritten query side by side.

3. Corrective and self-check loops

Here the system grades its own work at two points. After retrieval, a grader decides whether the passages actually address the question. If they do not, the agent rewrites the query, tries another source, or says it cannot find the answer. After generation, a second check compares each claim in the draft with the retrieved text and sends unsupported drafts back.

Two research papers gave these ideas their names. Corrective RAG (Yan and colleagues, 2024) adds a lightweight retrieval evaluator that scores the retrieved documents and triggers different actions, including a web search, when confidence is low. Self-RAG (Asai and colleagues, 2023) trains the model to emit special reflection tokens that say when to retrieve and whether its output is supported by the passages. In products, both are usually approximated with ordinary model calls and a grading prompt.

The limit is that the grader is also a model. It can pass a wrong answer or reject a correct one, and it does so more often when it is the same model with a similar prompt. A grader reduces unsupported answers; it does not remove them. Measure it against human labels before you trust it as a gate.

4. Multi-step retrieval

Some questions cannot be answered with one search because the second query depends on the first result. "Which of our suppliers are affected by the regulation mentioned in the March board memo?" needs the memo first, then the regulation's name, then the supplier contracts. The agent alternates between reasoning and searching until it has the chain, an approach popularized by the ReAct paper (Yao and colleagues, 2022).

This is the most capable pattern and the most expensive one. Errors compound: a wrong passage in step one steers every later step. It also has the least predictable latency, which is why step limits matter most here.

What an agent costs: latency and tokens

Each decision in the loop is a model call, and calls in a loop run one after another. Two effects follow.

Latency adds up. If one model call takes about two seconds in your stack, a pipeline answers in roughly two seconds plus the search. An agent that routes, searches, grades, searches again, grades again, drafts and checks makes five model calls in sequence, so the same question takes around ten seconds. The numbers are an illustration, but the multiplication is real. Streaming helps only at the end, because the user sees nothing while the agent is still deciding. Showing progress ("Searching contracts...") makes the wait tolerable but not shorter.

Input tokens grow faster than the number of steps. In a tool-calling loop, each model call receives the whole conversation so far, including every earlier search result. Take a 1,000-token prompt and three searches that each return 3,000 tokens. The four model calls read 1,000, 4,000, 7,000 and 10,000 tokens: 22,000 input tokens in total. A pipeline that searched once would have read 4,000. That is more than five times the input for three times the retrieval.

You can bring both down:

• Use a small model for routing and grading and keep the large one for the final answer.
• Return less per search. Rerank and pass the best three passages, not the top twenty.
• Run independent searches in parallel. The two halves of a comparison question do not depend on each other.
• Drop or summarize old tool results once the agent has moved on, so they are not resent on every call.
• Use prompt caching where your model provider offers it, so the repeated start of the conversation is billed at a lower rate.
• Cap steps and tokens per question and answer with what you have when the cap is reached.

Budget for the average and the tail separately. The average question may take two calls. The hard ones that hit the cap decide your 95th percentile latency and your worst-case bill.

When a plain pipeline is the better choice

An agent is the right tool for a specific set of failures. If you do not have those failures, it adds cost and new ways to be wrong. Stay with a fixed pipeline when:

• Most questions are lookups. One fact, one policy, one how-to, answered from one body of content. A single good search already finds the passage.
• The latency budget is tight. Voice assistants, autocomplete and in-app help that must respond in a second or two have no room for several sequential model calls.
• Volume is high and margins are thin. At millions of questions a month, a threefold token increase is a budget line someone has to approve.
• You need the same path every time. In regulated or audited settings, a fixed sequence is easier to explain, reproduce and sign off than a path the model chose.
• Retrieval itself is weak. If the right passage is not in the index, or chunking splits it in half, an agent will search three times and fail three times. Fix ingestion, chunking, hybrid search and reranking first. They are cheaper and they help every question.

Consider an agent when your evaluation shows the pipeline failing on a recognizable class of questions: comparisons across documents, questions whose second step depends on the first, questions that mix documents with structured data, or follow-ups that only make sense in context. The decision rule is simple: build the pipeline, measure it, sort the failures by cause, and add the one pattern that addresses the largest group.

Built by Gilzor

Results we’ve shipped

70+products launched

98%delivered on time

85%clients come back

5.0 ★on Clutch

Aviation33%faster baggage weigh-inWeb and Mobile application for Airlines→E-commerce · AITaking an AI-built e-commerce SEO platform to production→Healthcare+30%lab test ordersMedical Lab Web and Native Mobile Applications→

Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.

See how we’d approach yours →

How to test a RAG agent

A pipeline has two things to score: did retrieval find the right passages, and is the answer correct and grounded in them. An agent adds a third: the path. Two runs can reach the same correct answer, one in two steps and one in nine. Our guide to RAG evaluation covers the retrieval and answer metrics in depth; the steps below add what is specific to agents.

1. Build an evaluation set that includes the hard casesCollect real questions with expected answers and the source passages that support them. Add the types an agent is meant to fix (multi-part, multi-source, follow-ups), questions that need no retrieval, and questions your data cannot answer, where the right response is "I don't know".
2. Run the fixed pipeline as a baselineScore it on the same set. Every agent feature now has to beat a number, and you can see what each one costs in latency and tokens for the points it adds.
3. Test each decision on its ownLabel the correct route for each question and measure routing accuracy. Compare the grader's verdicts with human verdicts. Check rewritten queries for dropped constraints. A component that is 90% right on its own cannot be fixed by the steps after it.
4. Score the final answerMeasure correctness against the expected answer and faithfulness: whether every claim is supported by a retrieved passage. Check that citations point to passages that actually contain the claim.
5. Score the pathRecord steps, tool calls, tokens and time per question. Track how often the agent retrieves when it should not, skips retrieval when it should not, repeats the same query, or hits the step limit.
6. Repeat runsThe same question can take different paths on different runs. Run the set several times and look at the spread, not one lucky pass.
7. Trace everything in productionLog each model call, tool call and result with the question it belongs to. When a user reports a bad answer, you need to see which step went wrong. Add those cases to the evaluation set and rerun it before every release.

RAG agent readiness checklist

Tick what is already true. It shows whether an agent is the next step for your system or whether something earlier needs work first.

If you are choosing tooling for the loop, graph and tracing, see our comparison of the best RAG agent frameworks.

FAQ

Where Gilzor fits

The decision this guide keeps returning to, a pipeline or an agent and which pattern first, is the one we help teams make before they spend on a build. Gilzor offers RAG development services: custom RAG systems, applications and chatbots, RAG added to an existing product, and consulting on and evaluation of systems you already run. Our AI work so far includes an AI payroll platform with PDF data extraction that cut salary calculation errors by 17%, image and text search with recommendations for the QLode fashion commerce product, and two AI editor apps with more than 4 million downloads combined. The team works from offices in Warsaw (Poland) and Limassol (Cyprus), with partial overlap with the US working day. A 2-week fixed-price Discovery Sprint is the usual way to start: it ends with the question set, the baseline design and a scoped first version you can measure.