RLMs Beat RAG on Accuracy. Is It Worth It?

RLMs Beat RAG on Accuracy. Is It Worth It?




Testing Recursive Language Models on code and document Q&A

This article is based on a probe conducted by Genie, Emumba’s dedicated GenAI R&D group.
Vector RAG follows a familiar process: divide a corpus into chunks, create embeddings, build an index, retrieve the top results, and ask an LLM to answer from that fixed context.
Recursive Language Models take a different approach.
Instead of receiving a final set of chunks before reasoning begins, an RLM can search and inspect the source context during inference. It can write code, locate relevant files, examine sections, and continue investigating when its first search is insufficient.
A useful way to summarize the difference is:
RAG retrieves before reasoning. RLM retrieves while reasoning.
This adaptive process appears well suited to large, heterogeneous corpora. It may also reduce the need for a vector database, embedding pipeline, and fixed chunking strategy.
But that flexibility requires more inference-time work.
We compared DSPy’s RLM implementation with a minimal vector RAG baseline to answer a practical question:
Does the accuracy improvement justify the additional cost and latency?

What is a Recursive Language Model?

In DSPy’s implementation, the complete context is not placed directly inside the orchestrator model’s prompt.
Instead, it is loaded as a variable inside a sandboxed Python environment. The main LLM receives metadata about that context and can write Python code to explore it.
A typical query follows this pattern:
  1. Examine the question and the available context structure.
  2. Search for filenames, keywords, symbols, or other clues.
  3. Slice and inspect promising passages.
  4. Follow related references where needed.
  5. Continue searching until enough evidence has been collected.
  6. Submit a final answer.
The model can also send selected snippets to a secondary LLM for semantic analysis. However, that capability was rarely used in our experiment.
This differs from conventional RAG, where the answering model normally receives one fixed top-k result set. If the correct evidence is absent from those chunks, the model has no opportunity to retrieve again.

Our experimental setup

We tested the two approaches on code and technical-document question answering.
Both used GPT-5 as the main answering model, and both were evaluated with the same lenient binary LLM judge.
Neither system received task-specific optimization.

Code repository Q&A

The code corpus contained five GitHub repositories:
  • 4,862 files
  • Approximately 64 million characters
  • Approximately 16.4 million tokens
The evaluation contained five questions targeting implementation details inside one repository. The questions covered PDF processing, retrieval logic, embedding configuration, page-number assignment, and web-search behaviour.
We tested RLM in two conditions:
  • With only the target repository
  • With a larger context assembled from five repositories

Technical document Q&A

The document corpus contained:
  • 261 Markdown files, converted from approximately 270 technical PDFs
  • Approximately 17 million characters
  • Approximately 3.4 million tokens
  • 138 evaluation questions
The source material included release notes, product specifications, regulatory documents, feature descriptions, and other technical reference content.

The vector RAG baseline

The baseline was intentionally minimal:
  • Recursive character chunking
  • 1,000-character chunks with 200-character overlap
  • OpenAI text-embedding-3-large
  • Chroma vector index
  • Top-five retrieval
  • One GPT-5 answer-generation call
It did not use hybrid keyword search, reranking, metadata filters, query expansion, or other production RAG optimizations.

The RLM configuration

DSPy RLM used GPT-5 to plan searches and generate Python code.
It had:
  • A maximum of 20 search and reasoning iterations
  • A maximum context budget of 1.5 million characters
  • GPT-5-nano available as a secondary model
  • No DSPy prompt optimizer or task-specific tuning
This context budget created an important asymmetry.
The RAG index covered the complete corpus. RLM could inspect only the portion that fit into its assembled context.
For the document experiment, only 43 of the 261 Markdown files were visible to RLM.

What the RLM actually did

The captured trajectories showed that RLM usually began by inspecting the size and structure of the context.
For code questions, it searched for exact symbols such as environment-variable names, functions, and classes. It then inspected the surrounding implementation and followed references when necessary.
For document questions, it used string searches, regular expressions, filename clues, and context slicing to locate likely evidence.
Although a secondary GPT-5-nano model was available, only 15 sub-model calls were recorded across all 138 document questions.
Most of the work was performed by GPT-5 itself through repeated planning, code generation, and inspection of the Python output.

The results

Workload
▲▼
Method
▲▼
Accuracy
▲▼
Average latency
▲▼
Average cost per query
▲▼
Code Q&A
RLM, one repository
5/5
38.1s
$0.065
Code Q&A
RLM, larger repository context
5/5
58.2s
$0.069
Code Q&A
Minimal vector RAG
3/5
9.5s
$0.007
Document Q&A
RLM
77.9%
77.4s
$0.078
Document Q&A
Minimal vector RAG
53%
7s
$0.0065
RLM achieved higher accuracy in both workloads.
For code Q&A, it answered all five questions correctly, compared with three for RAG.
For document Q&A, it improved accuracy from 53% to 77.9%, an increase of approximately 25 percentage points.
The improvement came with a steep trade-off:
  • Approximately 4–11 times more latency
  • Approximately 9–12 times more cost per query

The document results in more detail

The 138 document questions produced the following head-to-head breakdown:
Outcome
▲▼
Questions
▲▼
Both systems correct
68
RLM correct, RAG incorrect
41
RAG correct, RLM incorrect
6
Both systems incorrect
23
RLM answered 41 questions that the RAG baseline missed.
RAG answered only six questions that RLM missed.
This suggests that the difference was not caused by a few isolated examples. RLM recovered useful evidence across a substantial portion of the test set.

Why did RLM perform better?

It could recover after an unsuccessful search

RAG made one retrieval decision per query. The top five chunks were selected before GPT-5 began answering.
If those chunks did not contain the required evidence, the answering model could not search again.
RLM could inspect an initial result, decide that it was incomplete, and try another search strategy.

It could use exact structural signals

Embedding similarity is useful when a question and source passage are semantically related. However, exact technical facts are sometimes easier to locate through:
  • Symbol names
  • Environment variables
  • Model identifiers
  • Filenames
  • Version numbers
  • Regular expressions
RLM could choose these techniques dynamically.

It could follow evidence across files

Some code questions required more than locating one relevant function. The system had to identify a related configuration method, helper function, or class before producing a complete answer.
RLM could move through these steps iteratively.

It avoided the fixed top-five bottleneck

Two code failures illustrate the RAG baseline’s limitation.
For one question, retrieval missed the definition of MINIMUM_CHUNK_SIZE. The answer model correctly stated that the value could not be determined from its context.
For another, retrieval missed the function specifying BAAI/bge-small-en-v1.5 as the embedding model. The answer therefore claimed that the model was unspecified.
These were retrieval failures rather than reasoning failures. The evidence never reached GPT-5.

The trade-off: inference-time search is expensive

A typical RAG request required:
  1. One query embedding
  2. One vector search
  3. One GPT-5 answer call
A typical RLM request required several cycles of:
  1. GPT-5 planning
  2. Python code generation
  3. REPL execution
  4. Output inspection
  5. Further search and reasoning
The code experiment also showed how a larger search space affected performance.
Accuracy remained at 5/5 when the RLM context expanded from one repository to a larger multi-repository corpus. However, average latency increased from 38.1 to 58.2 seconds.
Cost increased only slightly, suggesting that most of the additional work involved more search iterations rather than heavy use of the secondary model.

Context assembly is still an engineering problem

RLM can remove the fixed vector-indexing pipeline, but it does not remove the need to decide how a corpus is presented to the system.
In this experiment, files were concatenated in alphabetical order and truncated when the context reached 1.5 million characters.
This meant that:
  • Some repositories were only partly visible.
  • One repository was excluded entirely.
  • Only 43 of 261 technical-document files were included.
Despite this restricted access, RLM still achieved higher accuracy than minimal RAG over the full corpus.
The correct interpretation is not that corpus coverage is unimportant. It is that, for this question set, adaptive navigation of the visible context was more effective than one-shot top-five retrieval across the full index.
A production RLM system would need a more deliberate context strategy, potentially involving manifests, relevance-based file selection, hierarchical loading, or dynamic access to additional files.

Important limitations

These results should be interpreted as a focused probe rather than a universal comparison.

The RAG baseline was deliberately simple

A production RAG system could improve through:
  • Hybrid lexical and vector retrieval
  • Reranking
  • Query expansion
  • Multi-query retrieval
  • Metadata filtering
  • Parent-child retrieval
  • Better chunking
The 53% and 60% baseline results are not the ceiling for vector RAG.

RLM was also not optimized

We did not apply DSPy optimizers, prompt tuning, or hyperparameter search.
Better context assembly and search instructions could potentially improve both accuracy and efficiency.

The code test was small

The code benchmark contained only five questions, all targeting one repository.
The 5/5 result is encouraging, but it should not be treated as a broad code-Q&A benchmark.

Corpus access was unequal

RAG indexed the complete corpus, while RLM saw a limited prefix.
This makes the experiment realistic in one sense—large contexts do require budgeting—but complicates a clean algorithm-to-algorithm comparison.

Evaluation used an LLM judge

Answers were scored through a lenient binary rubric. More detailed human evaluation could expose differences in completeness and precision that a correct-or-incorrect score does not capture.

When should teams use RAG or RLM?

The results suggest that the approaches serve different operating requirements.

Choose RAG when:

  • Queries are frequent
  • Low latency is important
  • Cost per request matters
  • The corpus can be indexed and retrieval can be tuned
  • Most questions can be answered through direct semantic or lexical matching

Choose RLM when:

  • Accuracy is more important than response time
  • Questions require investigation across files or sections
  • The workload is low-volume but high-value
  • Queries are difficult, unusual, or exploratory
  • Building and maintaining retrieval infrastructure is not worthwhile
Potential RLM use cases include code archaeology, compliance review, incident investigation, technical due diligence, and exploratory analysis over unfamiliar corpora.

Complementary, not competing

RLMs and RAG do not need to be treated as mutually exclusive architectures.
A practical system could use RAG as the fast and economical default, then route difficult or low-confidence questions to an RLM-style investigation process.
RAG can also be used to narrow the corpus before an RLM performs deeper exploration.
This would combine:
  • RAG’s low latency and broad corpus coverage
  • RLM’s ability to search repeatedly and follow evidence

Key takeaway

RLM substantially improved answer accuracy in both experiments.
It reached 100% on the five code questions and improved technical-document accuracy by approximately 25 percentage points.
But the improvement came at approximately an order of magnitude more cost and latency.
The choice is therefore not simply between an old and a new retrieval method.
RAG is the economical default for frequent production queries. RLM is a compelling option for difficult, high-value investigations where finding the right answer matters more than returning it quickly.