RAG Is Dead

RAG Is Dead




Or is it? We tested this controversial claim on real codebases

This article is based on a probe conducted by Genie, Emumba’s dedicated GenAI R&D group.
Vector-based retrieval-augmented generation has long been the default architecture for answering questions over large private datasets.
The standard process is familiar: split the source material into chunks, create embeddings, store them in a vector database, retrieve the most relevant chunks, and pass them to a language model.
More recently, two alternatives have led teams to question whether this retrieval layer is still necessary.
First, long-context models can accept very large inputs. If an entire codebase fits into the context window, the complete repository can be passed directly to the model.
Second, newer retrieval patterns are becoming more agentic.
Instead of receiving a fixed set of chunks, an agent can search, read, inspect, and reason over files during inference. This pattern is appearing in coding agents, document agents, deep research systems, and newer agentic retrieval frameworks.
We refer to this broader paradigm as agentic RAG.
Agentic RAG is not one specific architecture or implementation. It describes systems in which an agent actively decides how to find and inspect information, often through multiple search and tool-use steps, before producing an answer.
This approach can be attractive when:
  • The corpus is small enough to search directly
  • Files change frequently and repeated indexing is inconvenient
  • A team wants to avoid setting up embeddings and a vector database
  • The task benefits from iterative exploration rather than one-shot retrieval
  • The data already has useful structure, such as folders, filenames, code symbols, or document hierarchies
For this experiment, we tested one early implementation of the pattern: LlamaIndex’s fs-explorer. It allowed an LLM-driven agent to explore a repository using file-search and file-reading tools.
The results should therefore be understood as an evaluation of this particular implementation and setup, not as a general benchmark of all agentic RAG systems.
Newer agentic retrieval solutions are becoming better at planning searches, reasoning about where relevant information is likely to be, and avoiding unnecessary exploration. These improvements can reduce the number of tool calls, failed search paths, overall cost, and latency. We will examine those developments in a separate post.
With that context, we asked: if a model can either see the whole repository or search it directly, do we still need vector-based RAG?
We tested all three approaches on a real codebase.

The three approaches

The experiment compared:
  1. Vector-based RAG
  2. An agentic RAG implementation using LlamaIndex fs-explorer
  3. Direct prompting with the full repository
Each approach gives the model access to code in a different way.

Vector-based RAG

The vector RAG pipeline indexed the repository in advance.
Python files were divided using syntax-aware code chunking. Other text files used a standard text splitter. The chunks were embedded with OpenAI’s text-embedding-3-small model and stored in Chroma.
At query time, the pipeline combined:
  • Dense vector retrieval
  • BM25 lexical retrieval
  • Reciprocal-rank fusion
The top five chunks were passed to the answer-generation model.
This was an intentionally minimal RAG implementation. It did not use query expansion, repository filtering, reranking, or other advanced code-retrieval optimizations.

Agentic RAG using fs-explorer

The agentic RAG implementation worked directly over the raw repository.
At each stage, the model could choose to:
  • Search for text within files
  • Find files matching a name or pattern
  • Read a file
  • Traverse into a directory
  • Parse a relevant document
  • Stop searching and return an answer
The agent could inspect one result, decide what to search next, open another file, and continue until it believed it had enough evidence to answer.
Unlike one-shot vector retrieval, this approach allowed the model to adapt its search during inference. A search result could reveal a class name, function, file path, or related symbol that guided the next step.
However, this early implementation performed its exploration through sequential model and tool calls. That gave it more opportunities to find relevant evidence, but also increased token use and latency.

Direct prompting

For direct prompting, the repository was flattened into one large text input.
Build outputs, caches, virtual environments, binary files, and version-control metadata were removed. The remaining source files were concatenated and passed to the model with the question.
This removed the retrieval step entirely. The model received close to the full repository, provided the complete input still fit within its context window.

The experiment

The target repository contained:
  • Approximately 236,000 lines of code
  • 258 files
  • Around one million tokens
We created five manually written questions grounded in the repository. The questions covered implementation details across different files and services.
Each test item included:
  • The question
  • A ground-truth answer
  • The exact supporting code
  • The path of the source file
These were not simple questions that could be answered by matching a function name. The systems had to find the relevant code and reason about how it behaved.
All three approaches used Gemini 3 Flash with temperature set to zero for answer generation. Using the same model allowed us to focus the comparison on how each approach located and supplied context.
A GPT-4.1-based judge evaluated the answers against both the reference answer and the supporting code. The scoring was strict: an answer had to include all critical information to receive a correct score.
We evaluated the systems in two conditions:
  1. The target repository by itself
  2. The target repository alongside three unrelated repositories
The second condition kept the questions and answers unchanged while making the search space approximately four times larger.

Results with one repository

Approach
▲▼
Accuracy
▲▼
Average tokens per query
▲▼
Average latency
▲▼
Agentic RAG using fs-explorer
5/5
33,566
35.26 seconds
Vector-based RAG
3/5
4,216
10.36 seconds
Direct prompting
4/5
1,004,653
51.88 seconds
The three approaches occupied very different positions in the accuracy, token-use, and latency trade-off.

The tested agentic RAG implementation achieved the highest accuracy

The fs-explorer implementation was the only approach to answer all five questions correctly.
Because the agent could inspect the repository iteratively, it was not limited to a fixed set of chunks selected at the start of the query. It could search for a term, open a file, discover another relevant symbol, and continue following the code.
This worked well while the search space was limited to one repository.
However, it used roughly eight times as many tokens as vector RAG and was about 3.4 times slower. These costs reflect the behaviour of the tested implementation, which relied on several sequential tool calls to explore the repository.
They should not be treated as fixed characteristics of every agentic RAG system.

Vector RAG was fastest and cheapest, but missed context

Vector RAG used the fewest tokens and returned answers with the lowest latency.
Its two failures were retrieval failures. The correct code did not reach the model’s prompt in sufficient detail, so the answer-generation model did not have the evidence required to produce a complete answer.
Code is particularly sensitive to missing context. A retrieved chunk may contain the relevant function but omit a helper method, configuration value, caller, or conditional branch that changes its behaviour.
The result does not mean that vector RAG is inherently inaccurate for code. It shows that a basic retrieval configuration may not be enough.

Direct prompting offered high recall at a high cost

Direct prompting answered four of the five questions correctly.
Its main advantage was recall: nearly the entire repository was visible to the model. However, each query consumed approximately one million tokens and took almost 52 seconds.
It was already operating close to the usable context limit. This made it expensive and left little room for growth.
The fact that it still missed one question also shows that placing information inside the context does not guarantee that the model will identify and use every relevant detail correctly.

What happened when we added three more repositories?

The larger search space exposed a different limitation in each method.

Direct prompting stopped being viable

The four repositories could no longer fit into the model’s usable context window.
Direct prompting therefore dropped out of the comparison entirely. Its cost grows with the size of the corpus, and it has a hard limit once the combined input exceeds model capacity.
This makes it useful as a baseline for bounded datasets, but not as a general production strategy for growing codebases.

Vector RAG confused similar code across repositories

Vector RAG accuracy fell from 3/5 to 2/5.
The main failure was cross-repository confusion. Retrieval returned plausible-looking code from one of the unrelated repositories. The chunk was semantically similar to the query but came from the wrong project.
The answer-generation model then reasoned over misleading evidence.
This is a particularly important failure mode for enterprise code search. Different projects may use similar service names, class structures, helper functions, and architectural patterns. Semantic similarity alone may not distinguish between them.
Repository-level metadata and filtering are therefore essential when multiple codebases share one retrieval index.

The tested agentic RAG implementation became much slower

The fs-explorer implementation continued to recover most answers, but its average latency rose to approximately 229 seconds per query.
One query reached the 1,000-second timeout without producing an answer.
The problem was structural to this implementation. The agent had to explore a much larger directory tree through sequential tool calls. Each search result could lead to another file inspection and another model round trip.
This shows why agentic RAG benefits from search-space narrowing. Useful starting context might include:
  • A known repository
  • A relevant directory
  • An open file
  • A service or module name
  • Earlier conversational context
  • An initial retrieval stage that identifies the most relevant part of the corpus
It does not mean that agentic RAG as a paradigm cannot scale. More recent systems use better planning, tool selection, parallel exploration, indexing, caching, and stopping strategies to reduce unnecessary search steps.
The finding is narrower: an early agent that explores a large repository collection sequentially and from scratch can become slow and expensive.

What each approach is best suited for

Direct prompting: a useful baseline for bounded contexts

Direct prompting is simple and can provide strong recall when the entire corpus fits into the prompt.
It is useful for:
  • Small repositories
  • One-off analyses
  • Evaluation baselines
  • Checking whether retrieval is the source of an error
It becomes impractical as the corpus grows because token use increases with the full input size and eventually exceeds the context window.

Agentic RAG: useful for iterative and structured search

Agentic RAG is useful when the task benefits from exploring information over several steps rather than receiving one fixed set of chunks.
It is particularly attractive when:
  • Data changes frequently
  • Building and maintaining a separate index is undesirable
  • The corpus has navigable structure
  • The question requires following relationships across several files
  • The system can begin from a reasonably narrow search scope
Its performance depends heavily on the implementation. Important design choices include:
  • How the agent plans its search
  • Which tools it can use
  • Whether searches can run in parallel
  • How it uses filenames, directories, symbols, and metadata
  • How it remembers evidence already found
  • How it decides when to stop
  • Whether another retrieval method first narrows the search space
Our results show the strengths and limitations of the fs-explorer implementation tested in this probe. They do not define the limits of newer agentic RAG systems.

Vector-based RAG: the strongest scaling default in this experiment

Vector RAG remained the most practical option for token usage and latency.
Its weakness was not the idea of retrieval itself, but the quality of the retrieval design. For code Q&A, production pipelines may need:
  • Syntax-aware chunking
  • File-path and repository metadata
  • Repository-level filtering
  • Hybrid lexical and vector search
  • Reranking
  • Query expansion
  • Context expansion
  • Deduplication
  • Mechanisms for resolving references across chunks
In this experiment, the most important scaling problem was retrieving similar code from the wrong repository. This can be addressed through repository-aware indexing and filtering before semantic ranking is applied.

Is RAG dead?

Our results do not support that conclusion.
Long-context models can remove the retrieval step while the complete corpus fits. Once the input becomes too large, that approach stops working.
Agentic RAG does not eliminate retrieval either. It turns retrieval into an adaptive process in which an agent chooses what to search, inspect, and read during inference.
This can improve accuracy for tasks that require exploration, but the outcome depends on how effectively the agent plans and executes its search.
Vector RAG remains a practical default for larger datasets because it narrows the context efficiently before answer generation. However, its accuracy depends on careful retrieval engineering.
The strongest production design may combine these patterns rather than treating them as mutually exclusive.
For example, a system could:
  1. Use vector or lexical retrieval to identify a likely repository, directory, or set of files.
  2. Give that narrowed search space to an agent.
  3. Allow the agent to inspect the selected files, follow references, and collect the exact evidence needed for the answer.
This combines the efficiency of indexed retrieval with the adaptability of agentic exploration.

Key takeaway

No approach won across accuracy, cost, latency, and scalability.
Approach
▲▼
Primary strength
▲▼
Primary limitation observed
▲▼
Direct prompting
High recall while the corpus fits
Very high token cost and a hard context limit
Agentic RAG using fs-explorer
Strong exploratory accuracy in a small search space
Sequential exploration became slow as the corpus grew
Vector-based RAG
Best cost, latency, and overall scalability
Accuracy depended heavily on retrieval design
Vector-based RAG was the most practical option for cost and latency in this experiment.
However, agentic RAG is a broader and rapidly developing paradigm. The latency and cost observed here belong to one early implementation, not to every system that uses agentic retrieval.
The larger lesson is that retrieval is changing rather than disappearing.
Sometimes information is retrieved through a pre-built index. Sometimes an agent searches for it dynamically. Increasingly, production systems may use both.