RAG vs Agentic AI: The Difference and When to Use Each
The short answer
RAG (retrieval-augmented generation) answers questions from your data. When a user asks something, the system searches your documents or database, puts the best-matching passages into the prompt, and the model writes an answer from them. The sequence is fixed by your developers and runs once per question. The model reads and writes text. It does not choose steps and it does not change anything in your systems.
Agentic AI completes tasks. The model gets a goal and a set of tools (search, a database query, an API call, a function that sends an email) and runs in a loop: pick a tool, read the result, decide what to do next, stop when the goal is met. The sequence is chosen by the model at run time, and it can differ for every request.
So the practical rule is short. If the job is "answer from our knowledge", you need RAG. If the job is "work out what needs doing and do it", you need an agent, and that agent will almost certainly use retrieval to look things up.
LLM vs RAG vs agent: three layers, not three options
It helps to see these as layers that stack, each adding one capability to the one below.
- An LLM alone answers from its training data and whatever is in the prompt. It knows nothing about your contracts, your product catalogue or yesterday's tickets, and it will sometimes fill that gap with a plausible guess.
- RAG fixes the knowledge gap. The model still makes one pass, but it reads the relevant passages first and can cite them. Our guide on what RAG is covers the mechanics.
- An agent fixes the control gap. The model is no longer limited to one pass over one prompt: it can search twice, check a record, run a calculation, then act.
RAG vs agentic AI side by side
| RAG pipeline | AI agent | |
|---|---|---|
| What it adds to a model | Knowledge: your documents and data at answer time | Control: the model chooses steps and uses tools |
| Who decides the steps | Developers, in code. Same path for every question | The model, at run time. Path varies per request |
| Model calls per request | Usually one (sometimes two or three with query rewriting or reranking) | One per step, so it grows with the task (for illustration, 5 to 20 on a multi-step task) |
| Output | An answer with sources | A completed task: records updated, messages sent, a report assembled |
| Touches your systems | Read-only | Read and, if you allow it, write |
| Response time | Predictable, typically seconds | Variable, from seconds to minutes |
| Cost per request | Low and stable | Higher and variable; grows with the number of steps |
| Worst-case failure | A wrong or unsupported answer | A wrong action, a loop that burns budget, or a wrong answer |
| How you test it | A set of questions with expected answers and sources | Task scenarios checked end to end, plus every tool call along the way |
| Best for | Questions answered from a body of knowledge | Multi-step tasks where the path is not known in advance |
How RAG works, and where it stops
A RAG system has two halves. Offline, your documents are split into passages, converted into embeddings (numeric representations of meaning) and stored in an index, often alongside a keyword index. Online, each question is used to search that index, the top passages are optionally reranked, and the best few are placed in the prompt with an instruction to answer only from them. The RAG architecture guide goes through each component.
The strength of this design is that it is boring. Every question takes the same path, so you can measure each stage separately: did the search return the right passage, and did the model use it correctly. When an answer is wrong, you can usually say which stage failed.
The same fixed path is the limit. A plain pipeline searches once, with the user's wording, and has to answer from whatever came back. It struggles in four situations:
- Multi-hop questions. "Which of our enterprise customers are on a contract that predates the new data-processing terms?" needs one lookup to find the date of the new terms and another to find contracts before it. A single search rarely returns both.
- Questions across sources. Half of the answer is in a PDF, the other half is a number in your billing database.
- Vague questions. The right move is to ask the user what they mean, and a pipeline has no step for that.
- Requests that are not questions. "Cancel my order" can be explained by RAG but not carried out.
You can extend a pipeline with query rewriting, routing between indexes or a second retrieval pass, and many teams should do that before reaching for an agent. Each extension is still a step a developer wrote. Once you find yourself adding branches for every new kind of question, you are hand-coding the decisions an agent would make.
How an agent works, and what it adds
An agent is a model running in a loop with tools. You describe each tool to the model: its name, what it does and what parameters it takes. On every turn the model either calls a tool or declares the task finished. Your code executes the call, appends the result to the conversation and asks the model again.
What this buys you is adaptation. The agent can notice that the first search returned nothing useful and try different terms. It can pull a customer record, see that the account is suspended, and take a different route from the one it would take for an active account.
What it costs you is predictability. Two runs of the same request may take different paths. The model can pick the wrong tool, pass wrong parameters, stop too early or keep going in circles.
How they combine: retrieval as a tool
In practice the two meet in the middle, in a pattern usually called agentic RAG. Your search index is wrapped as a tool, and the agent decides when to call it, what to search for and whether the results are enough. Compared with a pipeline, three things change:
- Search becomes optional. For "thanks, that helps" the agent does not search at all. A pipeline would retrieve passages anyway and pay for them.
- Search becomes repeatable. The agent can break a question into parts, search for each, and search again when the first results point somewhere new.
- Search becomes one source among several. The same agent can query a SQL database for numbers, call your order API for status and read policy documents through retrieval, then combine the three.
The retrieval layer itself does not get simpler. An agent searching a poorly chunked index with no access controls gets the same bad passages a pipeline would, only several times per task. The quality of the index, the permission checks and the evaluation of retrieval matter just as much. For the design of this combined pattern, see our guide to the RAG agent.
Cost, speed and risk: what changes when you add the loop
Cost per request
A RAG answer is one search plus one model call. Say the prompt carries 1,000 tokens of instructions and 3,000 tokens of retrieved passages: about 4,000 input tokens per question, every time.
An agent pays for a model call at every step, and each call re-sends the conversation so far, including earlier tool results. Take an eight-step task that starts with 2,000 tokens of instructions and tool descriptions, where each step adds 1,500 tokens of tool output. The input grows from 2,000 tokens on step one to 12,500 on step eight, for a total of 58,000 input tokens. That is roughly 14 times the RAG example, for one task. This is arithmetic to show the mechanism, not a benchmark, and prompt caching (where your model provider offers it) reduces the bill for the repeated part. The shape holds, though: agent cost grows faster than the number of steps.
Response time
Each step is a model call plus a tool call, run one after another. A RAG answer can start streaming to the user within a couple of seconds. A ten-step agent task takes as long as ten calls take, which is acceptable for back-office work and often not for a chat widget where a customer is waiting.
Reliability
Errors compound. If each step of a task goes right 95% of the time, a ten-step task finishes correctly about 60% of the time (0.95 to the power of 10). To get a ten-step task to 90%, each step has to be right about 99% of the time.
Security
Both designs read text the model did not write, and that text can contain instructions planted by someone else, known as prompt injection. In a RAG system the damage is limited to a misleading answer. In an agent with write tools, an injected instruction can try to trigger an action. The defence is structural: least-privilege tools, permission checks performed in your code and not left to the model, and human approval for anything irreversible.
A common misstep is building an agent for a job that is really question answering. The team pays agent prices in tokens, latency and testing for what one search and one model call would have done. Start at the lowest layer that passes your quality bar and move up only when your own failed test cases show why.
Built by Gilzor
Results we’ve shipped
Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.
A decision rule by task type
Go through these questions in order and stop at the first one that settles it.
- Is the output an answer or an action?If users need an answer drawn from your documents or data, start with RAG. If something has to change in a system, go to the next question.
- Can you write the steps down in advance?If every case follows the same sequence (extract these fields, check them against that table, file the result), build a fixed workflow: ordinary code that calls a model at specific steps, with retrieval where it needs facts. It is cheaper and easier to test than an agent.
- Does the path depend on what is found along the way?If the next step truly varies with the previous result, and the variations are too many to script, an agent earns its cost. Give it retrieval as a tool.
- What does a wrong action cost?If a mistake is cheap and reversible, the agent can act alone. If it is not (money, legal commitments, customer-facing messages), the agent prepares the action and a person approves it.
- Can the user wait?If the answer has to arrive in a few seconds, keep the live path to RAG or a short workflow and move agentic work to the background.
| Task | Fit | Why |
|---|---|---|
| Answering customer questions from help articles | RAG | One search usually finds the passage; speed and sources matter |
| Internal search over policies, contracts, wikis | RAG with permission filtering | Read-only; access control is the hard part |
| Extracting fields from invoices into an ERP | Fixed workflow | Same steps every time; no decisions to delegate |
| Research questions across documents and databases | Read-only agent with retrieval | Needs several searches and sources; nothing is changed |
| Resolving support tickets (refunds, address changes) | Agent with retrieval and narrow write tools | Path depends on account state and policy; actions need limits |
| Triage and routing of incoming requests | Workflow or a single classification call | One decision, not a sequence |
Three worked scenarios
1. A support assistant for a SaaS product
A B2B software company wants a chat assistant that answers "how do I" questions from 1,200 help articles and release notes. Every question is a lookup, and the answer must arrive in seconds with a link to its source.
Choice: RAG. An agent would add cost and delay without improving anything users notice. The work goes into the index (chunking, hybrid keyword and vector search, keeping it in sync with new releases), an honest "I could not find this" response, and a test set built from real tickets.
2. Refund handling for an online retailer
The same kind of assistant now has to process refund requests. The refund policy lives in documents, but the decision depends on live data: order date, delivery status, whether the item was already refunded, the customer's history.
Choice: an agent that uses RAG. Retrieval supplies the policy text. Tools supply the order record and issue the refund. The path differs by case, which is what the loop is for. The controls matter more than the model: the refund tool enforces an amount limit in code, anything above the limit goes to a person with the agent's reasoning attached, every tool call is logged, and the loop has a step cap.
3. An analyst assistant for a finance team
Finance staff ask questions such as "why did gross margin drop in the Midwest region last quarter, and does it relate to the supplier contract changes?" The numbers are in a data warehouse, the contract changes in PDFs, the commentary in meeting notes.
Choice: a read-only agent with retrieval. No single search answers this. The agent queries the warehouse, sees which product lines moved, searches the contracts for those suppliers and assembles an explanation with references. Because every tool is read-only, the worst case is a wrong analysis that a person reviews, so it can run without approvals.
Before you move from RAG to an agent
Is this task ready for an agent?
For budget ranges on the agent side, see our breakdown of AI agent development cost.
FAQ
What is the difference between RAG and agentic AI?
Is RAG a type of AI agent?
Do AI agents replace RAG?
What is the difference between an LLM, RAG and an agent?
Which is cheaper to build and run, RAG or an agent?
When should I use agentic RAG instead of plain RAG?
Is agentic AI riskier than RAG?
Where Gilzor fits
How far up the stack a task needs to go is worth settling before any code is written, and the answer is often "less than you planned". Gilzor builds custom RAG systems and chatbots, adds retrieval to existing products, and evaluates RAG systems that are already running. Our AI work includes a payroll platform that extracts data from PDFs and cut salary calculation errors by 17%, and image and text search with recommendations for the QLode fashion commerce product. If you are deciding between a pipeline, a workflow and an agent, a 2-week fixed-price Discovery Sprint ends with an architecture and an estimate you can act on. Start with our RAG development services, or describe the task and the data behind it.
No sales pitch
Get a straight answer for your project
Tell us what you’re building. We’ll reply with options, a rough cost and timeline. If we’re not the right fit, we’ll say so.
CTO of Gilzor. Responsible for architecture and the engineering standards our teams work by.
LinkedIn →Gilzor · AI/ML Solutions partner
Need a team for your AI product?
Services
AI/ML SolutionsComputer vision, NLP, predictive analytics, automation.→By company type
Selected projects
The team behind them
