RAG System Development with LangChain: What to Know Before You Commit

The short answer

RAG system development with LangChain means using LangChain's open-source components to wire a retrieval pipeline together: load documents, split them, embed them, store them, retrieve the relevant pieces for a question and hand them to a language model. LangChain gives you standard interfaces and a large catalog of integrations for each of those steps. LangGraph, the lower-level library from the same company, gives you control over multi-step and agentic flows. LangSmith, a separate commercial platform, gives you traces and evaluation.

None of them is the retrieval quality itself. Whether the right paragraph reaches the model depends on how your documents are cleaned and chunked, which embedding model and search method you use, how permissions are filtered and how you measure results. A LangChain RAG project and a plain-code RAG project fail for the same reasons. So the useful question is not "is LangChain good?" but "does it save us more work than it costs us in dependencies?" As a rule of thumb, for teams building more than a prototype the answer tends to be yes for integrations and agent control flow, and neutral for everything that decides answer quality.

What you actually get: LangChain, LangGraph, LangSmith

People say "LangChain" for three different things. Keep them apart, because they have different licenses, different costs and different lock-in.

Two relationships matter. First, according to the LangChain documentation, LangChain's own agents are built on top of LangGraph, so a team that starts with the high-level agent can drop down to a custom graph later without changing ecosystems. Second, LangSmith is optional. The libraries run without it, and you can send traces to another tool. The same company also publishes Deep Agents, which its documentation describes as a batteries-included harness built on LangChain agents, with context compression, a virtual filesystem and subagents; the current LangChain RAG tutorial uses it. Treat it as a fourth, optional layer.

How a LangChain RAG system is structured

The LangChain documentation splits a RAG application into two parts: indexing, which runs ahead of time, and retrieval plus generation, which runs on every question. LangChain has a component type for each stage.

In the documentation's own words, document loaders ingest data from external sources and return standardized Document objects, text splitters break large documents into chunks that can be retrieved individually and fit in a model's context window, vector stores store and search embeddings, and a retriever is an interface that returns documents for an unstructured query. The value of these interfaces is substitution. You can change the vector database or the embedding provider without rewriting the pipeline around it.

In outline, a first version looks like this:

1. Indexing job: for each source, load, split, embed, write to the vector store with metadata (source, date, access group).
2. Retrieval tool: take a query and the user's identity, search the store with a permission filter, return the top chunks with their sources.
3. Agent or chain: a model with a system prompt and the retrieval tool, returning an answer with citations.
4. Tracing and evaluation: every run recorded, scored against a fixed question set before each release.

Notice what the list assumes and the framework does not supply. Which chunk size suits your contracts versus your help articles. Whether you need keyword search next to vector search. How a user's permissions map to metadata filters. What the expected answers are. The LangChain tutorial uses a recursive character splitter with 1,000-character chunks and 200 characters of overlap. That is a starting value for a demo, not a recommendation for your corpus. Our RAG architecture guide covers these design choices in detail.

Three architectures, and what each costs you

The retrieval section of the LangChain documentation compares three ways to arrange the query lane. The table below follows that comparison.

The documentation's point on latency is the one to bring to a budget discussion: in 2-step RAG the maximum number of model calls is known and capped, so cost and response time per question are predictable. In agentic RAG the model chooses how many searches to run, so both vary by question. A sensible starting rule: start with 2-step, measure it against your evaluation set, and add agentic steps only for the question types where 2-step measurably fails. For a longer comparison, see RAG vs agentic AI.

When LangGraph is the right tool for agentic RAG

A LangGraph RAG agent is a retrieval flow written as a graph: nodes are steps, edges say which step comes next, and a shared state object carries the question, the retrieved documents and the draft answer between them. The agentic RAG tutorial in the LangGraph documentation shows the standard shape. A first node lets the model either answer directly or request a search. A retrieval node runs the search. A grading step checks whether the retrieved text is relevant. If it is, the graph generates the answer. If not, it rewrites the question and loops back. The documentation describes this retry loop as how the agent recovers from a weak first retrieval instead of stopping or inventing an answer.

You could write that loop in plain code. LangGraph earns its place when you need what its documentation lists as core capabilities:

• Durable execution. State is checkpointed, so a run can resume after a failure or a pause. This matters for long research-style tasks, not for a two-second FAQ answer.
• Human-in-the-loop. The graph can pause, let a person inspect or edit the state, then continue. Useful when an answer triggers an action or goes to a customer in a regulated setting.
• Memory. Short-term state within a conversation and longer-term memory across sessions.
• Streaming. Tokens and intermediate state updates are streamed as the graph runs, so users see progress during multi-step answers.

The rule of thumb: if your flow is "retrieve, then answer," LangGraph is overhead. If it has branches, retries, several retrieval sources, approval steps or tasks that outlive one request, an explicit graph is easier to reason about and test than a growing pile of conditionals. A graph also caps what the agent may do: you can limit the number of rewrite loops, which puts a ceiling on cost per question. More on design patterns in our RAG agent guide.

Where LangSmith fits

When a RAG answer is wrong, the first question is where it went wrong: the search returned the wrong chunks, the right chunks were ranked too low, or the model ignored good context. You can only tell from a trace that shows the query, the retrieved documents and the model call side by side. LangSmith's observability documentation describes exactly this: instrument the application, investigate traces, monitor performance in production. The platform also covers evaluation against datasets and managed deployment of agents.

Three things to settle before you adopt it:

• Data location. Traces contain user questions and the document text you retrieved. In the cloud version that content leaves your environment. According to the pricing page, self-hosted and hybrid deployment are Enterprise plan options. If your documents can't leave your network, check this before the first trace is sent.
• Billing model. The pricing page lists seats and traces as billed items, with agent deployment billed on resource usage and some other features metered separately. Trace volume grows with traffic and with the number of steps per agent run, so estimate it from expected usage.
• Alternatives. LangSmith is not required by the libraries. What is required is that some tool gives you traces and a repeatable evaluation run. Our RAG evaluation guide explains what to measure.

Built by Gilzor

Results we’ve shipped

70+products launched

98%delivered on time

85%clients come back

5.0 ★on Clutch

Aviation33%faster baggage weigh-inWeb and Mobile application for Airlines→E-commerce · AITaking an AI-built e-commerce SEO platform to production→Healthcare+30%lab test ordersMedical Lab Web and Native Mobile Applications→

Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.

See how we’d approach yours →

LangChain vs LlamaIndex vs plain code

LlamaIndex is the closest alternative. Its documentation describes the framework as an MIT-licensed, open-source toolkit for RAG and agents over your data, with data connectors, indexes, query and chat engines, agents and workflows. Its hosted document parsing and managed retrieval platform, LlamaParse, is a separate product. The third option is no framework at all: your code calling a model API and a vector database directly.

The honest summary: the two frameworks overlap heavily, and a competent team is likely to get similar retrieval quality from either, because quality comes from the data work. Pick the one your engineers know, check that it has the integrations you need, and don't mix both in one codebase without a reason. Choose plain code when the pipeline is simple and you value having few dependencies over development speed. Our comparison of RAG agent frameworks covers more options.

Lock-in and upgrade risk

Upgrade risk is now governed by a published release policy. Since LangChain 1.0, version numbers follow semantic versioning, and breaking changes are reserved for major releases. LangChain 1.0 and LangGraph 1.0 are designated long-term support releases: 1.0 stays in active status until 2.0 is released, then enters maintenance for at least one year. The legacy lines, LangChain 0.3 and LangGraph 0.4, are in maintenance mode until December 2026.

That last date matters if you are inheriting or buying a system. Older tutorials, code samples and existing codebases may still use the 0.x APIs. Ask any vendor which major version they build on, and ask the same about a codebase you take over. A system on the legacy line needs a migration budgeted soon.

Lock-in itself comes in three layers, and they are not equally sticky:

• Library code (moderate). Your pipeline imports framework classes. Limit this by wrapping retrieval and model calls in your own functions, so the framework sits behind an interface you own.
• Graph and state design (higher). An agent written as a LangGraph graph with checkpointed state is tied to that runtime. Moving it means rewriting control flow. Accept this only where you use the capabilities.
• Platform (highest if you use managed deployment). Traces, datasets and deployed agents on LangSmith. Keep your evaluation set in your own repository in a plain format so it can run against any stack.

What stays portable regardless: your cleaned documents, your chunking rules, your embeddings in a vector database you chose, your prompts and your evaluation set. Those are the real assets of a RAG system. Protect them and the framework becomes replaceable.

What production needs beyond the framework

Use this list when you review a proposal or a system that "uses LangChain." Every item is work the framework leaves to the team.

If a vendor's plan covers the framework setup in detail and these items in a sentence, the plan is for a demo. For the full build sequence, see our RAG development guide.

FAQ

Where Gilzor fits

Gilzor builds custom RAG systems, adds RAG to existing products and evaluates RAG systems that give answers nobody trusts yet. We treat LangChain and LangGraph as tools to choose per project, not as a default: a simple pipeline may need neither, and an agentic flow with approvals usually benefits from a graph. Our AI work includes a payroll platform that extracts data from PDFs and cut salary calculation errors by 17%, and AI search and recommendations for QLode, a fashion commerce product. If you want the framework question settled before you spend on a build, our 2-week fixed-price Discovery Sprint ends with an architecture, an evaluation plan and a scoped first version. The team works from Poland with partial overlap with the US working day.