What Is Agentic RAG?

Quick Definition#

Agentic RAG (Agentic Retrieval-Augmented Generation) is an AI design pattern where an agent dynamically controls the retrieval process — deciding when to retrieve, what to search for, how to evaluate retrieved results, and whether to search again with refined queries. Unlike traditional RAG, which executes a single fixed retrieval step, agentic RAG treats retrieval as a tool the agent calls multiple times in an adaptive reasoning loop.

If you are new to RAG, start with Retrieval-Augmented Generation (RAG) before reading this page. For the underlying agent capabilities, see AI Agent Planning and Tool Calling. Browse all AI agent terms in the AI Agent Glossary.

Standard RAG vs. Agentic RAG#

Standard RAG Pipeline#

User Query → Retrieve Top-K Documents → Generate Answer

Standard RAG is fast and predictable: embed the query, find similar documents, generate a response. It works well for straightforward question-answering when the relevant information is clearly related to the original query.

Limitations:

• One retrieval round, regardless of whether it found useful context
• The search query is always derived directly from the user's question
• No ability to discover intermediate facts and retrieve based on them
• Cannot combine information from multiple retrieval queries

Agentic RAG#

User Query → Agent Reasons → Decides to Retrieve → Evaluates Results
  → Decides: Sufficient? → Yes: Generate Answer
              No: Refine Query → Retrieve Again → ...

An agentic RAG agent treats retrieval as a tool call. It can:

• Decide whether retrieval is even needed for a given question
• Formulate multiple different search queries to find complementary information
• Evaluate whether retrieved documents are relevant before using them
• Retrieve from different data sources depending on what the question requires
• Stop when it has sufficient information or escalate to a human if it cannot find what it needs

Core Agentic RAG Patterns#

Self-RAG#

In Self-RAG, the agent evaluates its own retrieval decisions:

1. Generate an initial response without retrieval
2. Decide if retrieval would improve the response
3. If yes, retrieve and evaluate whether the results are relevant
4. Decide whether to use the retrieved information or ignore it
5. Generate a final response with explicit citations

This pattern reduces hallucination by making the agent actively verify whether its knowledge is sufficient.

Corrective RAG (CRAG)#

CRAG adds a correction loop: if the retrieved documents are evaluated as low-quality or irrelevant, the agent reformulates the query and tries again — potentially switching to web search if the knowledge base retrieval fails.

Multi-Query Retrieval#

The agent generates multiple distinct search queries for a single user question, retrieves documents for each query, deduplicates results, and synthesizes a comprehensive answer from the combined context. Useful when the user's question has multiple dimensions.

Hierarchical Retrieval#

The agent first retrieves at a high-level (document summaries or metadata), then decides which specific document sections to retrieve in full. This dramatically reduces context window usage on large knowledge bases while maintaining retrieval accuracy.

Adaptive Retrieval#

The agent decides which retrieval tool to use based on the question type:

• Semantic search for conceptual questions ("What is the company's approach to X?")
• Keyword search for exact-term queries ("Find all mentions of RFC 9999")
• SQL queries for structured data ("What is total revenue for Q3?")
• Web search for real-time information ("What is the current price of X?")

Implementation Example (LangGraph)#

A simple agentic RAG implementation using LangGraph:

from langgraph.graph import StateGraph, END
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.tools import tool
from typing import TypedDict, List

class AgentState(TypedDict):
    question: str
    retrieved_docs: List[str]
    answer: str
    retrieval_count: int

# Define retrieval tool
@tool
def retrieve_documents(query: str) -> str:
    """Search the knowledge base for relevant information."""
    results = vectorstore.similarity_search(query, k=4)
    return "\n\n".join([doc.page_content for doc in results])

# Agent node: reason and decide next action
def reasoning_node(state: AgentState):
    llm = ChatOpenAI(model="gpt-4o")
    tools = [retrieve_documents]
    # Agent decides whether to retrieve, what to search for, or generate final answer
    response = llm.bind_tools(tools).invoke([
        {"role": "user", "content": state["question"]},
    ])
    return {"answer": response.content}

# Build graph
builder = StateGraph(AgentState)
builder.add_node("reason", reasoning_node)
# ... add retrieval nodes, edges, and termination conditions
graph = builder.compile()

When to Use Agentic RAG#

Good fit scenarios#

• Complex multi-hop questions: "Which of our top 10 customers had contracts that referenced the product feature discontinued in Q2?"
• Uncertain knowledge scope: Questions where you don't know in advance whether the knowledge base will have sufficient context
• Multiple data sources: Questions requiring synthesis from a vector database, a SQL database, and a web API
• Research and analysis tasks: Where the agent needs to explore a topic iteratively, following threads as it discovers them

Stick with standard RAG when#

• Questions are simple, direct, and answerable from a single retrieval round
• Latency is critical and you cannot afford multiple LLM calls per query
• Cost per query must be minimized (high-volume, simple lookups)
• The knowledge base is small and well-indexed (retrieval is already highly accurate)

Agentic RAG vs. Standard RAG: Trade-offs#

Common Misconceptions#

Misconception: Agentic RAG is always better than standard RAG Agentic RAG is better for complex questions but adds unnecessary cost and latency for simple ones. Route queries to the appropriate pattern based on complexity.

Misconception: More retrieval rounds always improve quality Excessive retrieval introduces noise and can confuse the LLM with contradictory or tangential information. Setting a maximum retrieval count and quality thresholds is important.

Misconception: Agentic RAG eliminates hallucination Agentic RAG significantly reduces hallucination by grounding responses in retrieved context, but hallucination remains possible, especially when retrieved documents contain errors or when the agent misinterprets retrieval results.

Related Terms#

• Retrieval-Augmented Generation (RAG) — The foundational pattern agentic RAG extends
• Vector Database — The storage backend for semantic retrieval
• Tool Calling — How agents invoke retrieval as a tool
• AI Agent Planning — How agents decide which actions to take
• AI Agents — The broader capability that agentic RAG is built on
• Introduction to RAG for AI Agents — Foundation tutorial for RAG concepts and implementation
• Build AI Agent with LangChain — LangChain-based agent implementation including RAG patterns

Frequently Asked Questions#

What is the difference between RAG and agentic RAG?#

Standard RAG executes one fixed retrieval round. Agentic RAG treats retrieval as a tool an agent calls adaptively — deciding when to retrieve, what to search for, and whether to retrieve again based on the quality of initial results.

When should I use agentic RAG vs. standard RAG?#

Use standard RAG for simple, direct questions answerable in one retrieval round. Use agentic RAG for complex multi-hop questions, multi-source retrieval, or queries where the ideal search strategy is not obvious from the original question.

What are the performance trade-offs of agentic RAG?#

Agentic RAG produces higher-quality answers on complex questions but requires multiple LLM calls, adding latency (1-10 seconds vs. sub-second for standard RAG) and cost. This trade-off is acceptable for high-value queries but unsuitable for high-volume simple lookups.

What frameworks support agentic RAG?#

LangChain and LangGraph offer the most complete agentic RAG support with routing agents, Self-RAG, and Corrective RAG patterns. LlamaIndex provides strong agentic retrieval through its query engine abstractions. CrewAI and the OpenAI Agents SDK support RAG through custom tool definitions.