Where to Optimize in a RAG Pipeline?

Where to Optimize in a RAG Pipeline?




What improves accuracy the most in real systems—for our document Q&A use case

This article is based on a probe conducted by Genie, Emumba’s dedicated GenAI R&D group.
A RAG pipeline has several possible optimization points.
Documents can be parsed differently. Chunk sizes and boundaries can be changed. Retrieval can combine lexical and vector search. Results can be filtered, reranked, or expanded before they reach the answer-generation model.
When a RAG system produces inaccurate answers, teams therefore face a practical question: where should they optimize first?
In an earlier Genie probe, we compared a custom RAG pipeline with a managed RAG service for technical PDF question answering. The custom pipeline achieved materially higher answer accuracy.
That pipeline included several use-case-specific changes across parsing, chunking, retrieval, and reranking. In this probe, we studied key components separately to understand where the largest performance gains came from.

Our experimental setup

We ran controlled A/B experiments over:
  • 285 technical PDFs
  • 138 question-and-answer queries
The documents contained dense technical language, tables, hierarchical sections, similar product names, and specifications whose meaning depended on their surrounding context.
We compared:
  • Three chunking strategies
  • Four reranking strategies, alongside a no-reranking baseline
For each comparison, we changed one component while keeping the rest of the pipeline constant.
We evaluated performance at two levels.

Retrieval accuracy

Retrieval accuracy measured whether the expected ground-truth context appeared within the final top five chunks supplied to the model.
This tells us whether the retrieval pipeline found the evidence associated with the question.

Answer accuracy

Answer accuracy measured whether the final generated answer was correct, using an LLM-as-a-judge.
Retrieval and answer accuracy are related, but they are not identical. Another retrieved passage may contain enough information to answer even when the designated ground-truth chunk is absent. Conversely, the correct evidence may be present, but the model may still interpret it incorrectly.
Measuring both gives a clearer view of where the system is succeeding or failing.

Experiment 1: How much does chunking matter?

We compared three approaches.

Semantic chunking

Semantic chunking attempts to split the extracted text where its meaning changes. It uses embeddings to identify semantic boundaries instead of relying only on a fixed number of characters or tokens.

Recursive chunking

Recursive chunking splits text using a hierarchy of separators until chunks fall within a target size. It is simple, predictable, and widely used as a general-purpose strategy.
For the documented runs, the generic strategies used a target chunk size of 800 with an overlap of 150.

Custom layout- and section-aware chunking

The custom approach began with layout-aware OCR parsing.
Instead of treating the PDF as a flat stream of extracted text, it preserved document structure and created chunks around meaningful units such as:
  • Sections
  • Subsections
  • Tables
For example, one table or one subsection could become a complete chunk.
This changed both the quality of the extracted content and the boundaries of the information passed into the embedding and retrieval stages.

Chunking results

Chunking strategy
▲▼
Top-five retrieval accuracy
▲▼
Answer accuracy
▲▼
Semantic
~69%
~77%
Recursive
~62%
~78%
Custom layout- and section-aware
93%
90%
The custom approach produced the largest improvement in the experiment.
Compared with semantic chunking, it improved retrieval accuracy by approximately 24 percentage points. Compared with recursive chunking, the improvement was approximately 31 points.
The effect on final answers was also substantial. Answer accuracy improved by approximately 12–13 percentage points over the two generic strategies.
These results show that chunk construction is not simply a formatting decision. It determines what the embedding represents, what the retriever can find, and what context the answer-generation model receives.

Why layout-aware chunking performed better

Technical documentation is highly structured.
A specification may only make sense when read with:
  • Its section title
  • A product or model heading
  • A regional qualifier
  • A table header
  • A footnote
  • A neighbouring paragraph
For example, an “operating guidelines” section may mean something different depending on whether it appears beneath an “EU” heading or a “US” heading.
A generic text splitter may separate the guideline from that heading. The resulting chunk can still look meaningful in isolation, but it no longer carries the context required to interpret it correctly.
This creates problems throughout the pipeline.

The embedding represents incomplete meaning

An embedding is created for the content inside a chunk. If the chunk contains a value but not the heading that qualifies it, its vector representation may be ambiguous.

Retrieval returns plausible but incomplete evidence

The chunk may match the wording of a query while omitting the detail that distinguishes the correct answer from a similar one.

The answer-generation model must reason from missing context

Even a strong language model cannot reliably reconstruct a product name, regional constraint, or section relationship that was removed during parsing and chunking.
The custom pipeline preserved these structural relationships and produced more coherent units of information.
In this use case:
Better chunks led to better context, and better context led to better answers.

Experiment 2: How much does reranking add?

Initial retrieval scores do not always place the best evidence at the top.
A common approach is therefore to retrieve a broader candidate set, apply a more precise scoring or filtering method, and keep only the highest-ranked results.
In this experiment, we:
  1. Retrieved 30 candidate chunks.
  2. Applied the selected reranking strategy.
  3. Kept the final top five chunks.
We compared the following methods.
Reranking approach
▲▼
Mechanism
▲▼
Top-five retrieval accuracy
▲▼
None
Initial retrieval ranking
83%
FlagEmbedding
Cross-encoder scoring
~88%
ColBERT
Late-interaction scoring
~83%
LLM-based
Prompt-based scoring
~86%
Custom entity-aware method
Named entity recognition, filtering, and entity matching
~93%
The custom approach increased retrieval accuracy from 83% to approximately 93%—a 10-percentage-point improvement over the baseline.
It also outperformed the strongest off-the-shelf reranker tested by approximately five points.

Why entity-aware reranking helped

The technical corpus contained many closely related documents and passages.
Two chunks might be semantically similar while referring to:
  • Different products
  • Different product versions
  • Different components
  • Different standards
  • Different regional requirements
A general-purpose relevance model may rank both chunks highly because their overall meaning is similar.
Our custom method used named entity recognition to extract important entities from the query. It then filtered irrelevant candidates and promoted chunks containing the relevant entities.
This added a signal that was especially valuable for the dataset: not only whether the passage was generally related to the question, but whether it referred to the exact product or entity being asked about.
The result also demonstrates an important broader principle:
A simple use-case-specific signal can outperform a more sophisticated general-purpose method when that signal reflects the corpus’s actual failure modes.

Reranking is refinement, not recovery

Reranking improved the ordering of the retrieved candidates, but it had a hard limitation.
It could only operate on the information already present in the candidate set.
If the correct passage was not retrieved in the initial 30 chunks, no reranker could move it into the final five. It was not available to be rescored.
This is why strong initial retrieval remains critical.
The stages have different responsibilities:
  • Parsing and chunking determine the units of information available to search.
  • Initial retrieval determines which candidates enter the result set.
  • Reranking and filtering improve the ordering and precision of those candidates.
  • Answer generation reasons over the evidence that survives the earlier stages.
A downstream component cannot fully repair information lost upstream.

Where should teams optimize first?

There is no universal order that applies to every RAG system. Optimization should begin with evidence about the system’s actual failures.
However, our results suggest a practical sequence for complex structured documents.

1. Validate parsing and chunk quality

Before changing embedding models or adding sophisticated rerankers, inspect whether the documents are being represented correctly.
Check whether:
  • Reading order is preserved
  • Section headings remain attached to their content
  • Tables retain their headers
  • Important footnotes are included
  • Chunk boundaries follow meaningful document units
  • Product and version context remains visible
When the source representation is poor, every later stage inherits the problem.

2. Measure initial retrieval

Determine whether the required evidence appears anywhere in a reasonably broad candidate set.
If it does not, reranking is not the immediate solution. The team may need to improve:
  • Parsing
  • Chunking
  • Lexical retrieval
  • Vector retrieval
  • Metadata filtering
  • Query processing

3. Use reranking to improve precision

Once the initial candidate set has good recall, reranking can help ensure the strongest evidence appears within the limited context passed to the model.
Use-case-specific signals should be considered alongside generic rerankers. Product names, file paths, document types, dates, versions, and other domain entities may be critical.

4. Evaluate the complete pipeline

A retrieval gain does not automatically translate into an equal answer-accuracy gain.
Teams should measure both:
  • Whether the expected evidence was retrieved
  • Whether the final answer was correct
This makes it possible to distinguish retrieval failures from answer-generation failures.

Key takeaway

In this experiment, the highest-leverage optimization occurred early in the pipeline.
Layout-aware parsing and structurally aligned chunking improved top-five retrieval accuracy to 93% and answer accuracy to 90%. This was a larger gain than any of the reranking changes we tested.
Entity-aware filtering and reranking provided a further useful refinement, improving retrieval accuracy by 10 percentage points over the no-reranking baseline.
The results do not mean that layout-aware chunking will always be the most important RAG optimization. A simple prose corpus, source-code repository, message archive, and technical PDF collection have different structures and failure modes.
The broader lesson is that RAG optimization should not begin with a list of fashionable techniques.
It should begin by measuring each stage of the pipeline, identifying where relevant information is lost, and applying changes that match the structure of the data and the queries the system must answer.