Structured Context Extraction from PDFs: Analysing the State of the Art

Structured Context Extraction from PDFs: Analysing the State of the Art




This article is based on a probe conducted by Genie, Emumba’s dedicated GenAI R&D group.
Many enterprise Q&A systems depend on information stored in PDFs.
However, PDFs are designed to preserve how a document looks on a page—not to communicate its structure to a machine.
A person can look at a page and immediately recognize its title, columns, sections, tables, captions, images, and reading order. A language model or retrieval system does not receive those relationships automatically. They must first be reconstructed from visual and positional information.
This makes PDF parsing an important part of context engineering.
To understand the current state of PDF-to-Markdown extraction, we compared seven parsing approaches across several real-world document types. We evaluated not only whether they extracted the words, but whether they preserved the structure needed by downstream RAG pipelines, agentic retrieval systems, and document Q&A applications.

Why PDF structure matters

Extracting all the text from a PDF does not necessarily produce usable context.
Consider a two-column report. A parser may correctly identify every word but interleave the left and right columns. The resulting text is complete, yet its meaning is corrupted.
The same problem appears in other forms:
  • A subsection may be assigned the same heading level as its parent.
  • A table may be converted into a sequence of unrelated values.
  • A command-line example may be mixed into surrounding prose.
  • A figure caption may be separated from its image.
  • A regulatory requirement may lose the product or region heading that qualifies it.
These are not cosmetic formatting issues.
They determine whether later stages can correctly divide the document into chunks, retrieve the relevant information, and reason over the evidence.
Markdown is often used as an intermediate representation because it can preserve many of these relationships in a text-first format:
  • Heading levels represent document hierarchy.
  • Tables retain rows and columns.
  • Code fences distinguish executable or technical content from prose.
  • Image references preserve the existence and position of figures.
  • Ordered text approximates the intended reading sequence.
The quality of the PDF-to-Markdown conversion therefore places an upper bound on the quality of the context supplied to the rest of the system.

Our benchmark

We evaluated seven PDF parsing tools and pipelines:
  • AWS Textract
  • Azure Document Intelligence
  • LlamaParse
  • PaddleOCR v1.5
  • GLM-OCR
  • LandingAI
  • GenieParse, our internal parsing pipeline
The benchmark used five source PDFs containing a total of 11 pages.
Each document was chosen to expose a different set of parsing challenges.
Document category
▲▼
Main challenges
▲▼
Financial report
Multiple columns, sidebar callouts and chart images
Hardware quick-reference guide
Dense three-column layout, numerous images, tables and regulatory text
Installation guide
A borderless table and nested heading hierarchy
Network configuration guide
Multiple CLI code blocks
Academic paper
Two-column layout, complex tables, equations and figures
This was a focused stress test, not a universal leaderboard covering every possible document type or tool configuration.
Its purpose was to identify practical strengths and recurring failure modes across the types of documents commonly used in enterprise knowledge systems.

A note on GenieParse

GenieParse is a configurable parsing pipeline developed for our document Q&A use cases.
Rather than expecting one model to produce perfect Markdown directly, the pipeline combines a base parser with targeted processing for known failure modes.
For this experiment, it used Dolphin as the base OCR and document-parsing model. It then applied additional processing for:
  • Reconstructing heading hierarchy
  • Correcting table structure
  • Extracting and fencing code blocks
  • Preserving image references
  • Normalizing output for technical documents with complex layouts
LLM-based processing can be added where needed, but it is not required by default.
The purpose of this design is not to create a special rule for every document. It is to address consistent classes of errors that appear across the target corpus.
This makes GenieParse different from a single off-the-shelf parser. It is an engineering pipeline built around a base model.

How we evaluated the tools

We manually created and verified a Markdown reference for each source PDF.
The references preserved:
  • Document hierarchy
  • Reading order
  • Tables
  • Captions
  • Body text
  • Images
  • Code
  • Mathematical notation
The ground-truth files were reviewed before scoring. Incorrect text was fixed, visually derived chart data was not treated as an extractable table, and structures such as borderless tables were aligned with the source PDF.

The nine evaluation metrics

Each parser received scores from 0 to 100 across nine metrics.
Metric
▲▼
What it measured
▲▼
Headings extracted
Whether the correct heading text was present
Heading hierarchy
Whether each heading used the correct Markdown nesting level
Reading order
Whether content followed the intended sequence of the source layout
Captions
Whether figure and table captions were preserved correctly
Table accuracy
Whether rows, columns, headers and spans were structurally correct
Text accuracy
Character-level accuracy of prose and table-cell content
Images extracted
Whether images were represented or referenced in the output
Code-block accuracy
Whether code was correctly identified, fenced and indented
Math and special notation
Whether equations and special characters were preserved
A metric was excluded for documents where it did not apply. For example, code-block accuracy was not scored for a PDF containing no code.

LLM-led scoring with automated checks

The main evaluation used Claude-based agents to compare:
  1. The source-PDF description
  2. The manually verified Markdown
  3. The parser’s generated Markdown
  4. The scoring rubric
Automated scripts also produced structural counts and text-similarity signals, including heading counts, table counts, image references and body-text similarity.
These checks supported the evaluation, while the LLM handled judgments that required an understanding of document structure, such as:
  • Whether columns were placed in the correct reading order
  • Whether subsections were nested under the correct parent
  • Whether table rows and merged cells matched the source
  • Whether code and equations retained their intended representation

Aggregation and cross-validation

Metrics based on discrete items—such as headings, captions, tables and images—were weighted according to the number of applicable items in each document.
This prevented a PDF containing a single table from influencing the table score as much as one containing several tables.
Holistic metrics such as hierarchy, reading order, text, code and math used averages across the applicable documents.
As a bias check, the outputs were also evaluated directly against the source PDFs without using the prepared Markdown references. The overall ranking remained identical.

Overall results

Rank
▲▼
Tool
▲▼
Overall score
▲▼
Main strengths
▲▼
Main weaknesses
▲▼
1
GenieParse
88.6
Hierarchy, reading order, images and code
Missed a borderless table
2
LlamaParse
80.9
Text, captions and math
Weak image preservation
3
PaddleOCR v1.5
69.4
Tables and reading order
Weak hierarchy and code
4
LandingAI
65.7
Headings and captions
Weak hierarchy
5
Azure Document Intelligence
60.9
Reading order and tables
Weak hierarchy and code
6
GLM-OCR
60.9
Text accuracy
No Markdown heading hierarchy
7
AWS Textract
52.0
Tables and captions
No image references and weak code handling
No tool was consistently best across every category.
Some were strong at extracting text. Others performed well on conventional tables. Several found heading text reliably but failed to preserve the relationships between headings.
The highest overall score came from the pipeline that performed strongly across the largest number of structural and content categories.

What the category results reveal

Heading extraction is largely solved

GenieParse, LlamaParse and LandingAI extracted 100% of the reference headings. PaddleOCR also reached 97.9%.
Finding text that visually resembles a heading was therefore not the main difficulty.
The harder problem was reconstructing hierarchy.

Correct hierarchy remains a major gap

GenieParse achieved 89.8% for heading hierarchy.
The next-highest score was LlamaParse at 56.8%. Most other tools scored considerably lower, and GLM-OCR used no Markdown heading syntax in its output.
This distinction is important.
A parser may find every heading but flatten them all to the same level:
# Installation
# Hardware requirements
# Regional restrictions
The text is present, but the parent-child relationship has been lost. A downstream chunker can no longer reliably determine which requirements belong to which section.
Hierarchy was the most discriminating metric in the benchmark.

Complex layouts still disrupt reading order

GenieParse achieved 97% for reading order, followed by LlamaParse at 94% and GLM-OCR at 93.6%.
Several tools performed reasonably well overall, but complex multi-column and callout-heavy layouts still caused mistakes.
Reading-order errors can be particularly damaging because they join sentences or sections that were never intended to be read together.

Standard tables are handled better than irregular ones

PaddleOCR achieved the strongest table-structure score at 87.5%, followed closely by AWS Textract at 86.7% and LlamaParse at 85%.
Most tools handled ordinary bordered tables reasonably well.
The harder cases involved:
  • Borderless tables
  • Merged rows or columns
  • Row spans
  • Tables embedded within complex page layouts
GenieParse’s main failure was a borderless documentation-links table, reducing its table score to 71.7%.
This is a useful reminder that the strongest overall pipeline still had a clear area for improvement.

Text accuracy is comparatively mature

LlamaParse led text accuracy with 88.6%, followed by GLM-OCR at 87% and GenieParse at 85%.
Text extraction was generally stronger than structural preservation across the benchmark.
However, accurate text does not compensate for incorrect hierarchy or reading order. A parser can extract nearly every character while still producing poor context for Q&A.

Image preservation remains inconsistent

GenieParse achieved 96.2% for image extraction.
PaddleOCR also performed strongly at 88.6%, while several tools detected images but represented them through HTML or proprietary tags rather than standard Markdown.
LlamaParse scored 38.8%. It sometimes converted charts into data-like text rather than preserving them as figures. AWS Textract returned no image references in the accepted formats.
For multimodal documents, silently dropping an image can remove a significant part of the source evidence.

Code blocks sharply separate the tools

GenieParse scored 95% for code blocks. LlamaParse scored 75%, and GLM-OCR scored 65%.
PaddleOCR, Azure Document Intelligence and AWS Textract generally returned CLI commands as ordinary text without code fencing or reliable indentation.
For technical documentation, code formatting is structural information. It tells the downstream system where a command begins, which lines belong together, and which content is explanatory prose.

Math remains a specialized challenge

The research paper also tested mathematical notation.
LlamaParse performed best at 90%, followed by GLM-OCR at 78% and GenieParse at 70%.
This category was not shown in the carousel’s simplified summary, but it reinforces the broader result: different tools specialize in different content types.
A parser chosen for research literature may therefore require a different priority order from one chosen for installation guides or CLI documentation.

The main challenge is consistency

The benchmark did not identify one universally difficult element.
Heading text was usually extracted correctly. Many standard tables were reconstructed well. Text accuracy was generally reasonable.
The difficult part was preserving all relevant element types together within one coherent representation.
A useful technical document may contain:
  • A nested heading hierarchy
  • A two-column layout
  • A borderless table
  • An image with a caption
  • A command-line block
  • Version-specific prose
A parser that handles five of these six elements may still create a critical downstream failure if the missing element contains the answer to a user’s question.
This is why overall consistency matters more than winning one category.

Why targeted fixes helped

GenieParse did not lead every metric.
It ranked below other tools on table structure, text accuracy, captions and math. Its overall advantage came from combining strong results across hierarchy, reading order, images and code while remaining reasonably accurate elsewhere.
The pipeline approach allowed repeated failure modes to be addressed directly.
For example:
  • If the base parser finds headings but assigns incorrect levels, hierarchy can be reconstructed.
  • If CLI commands appear as prose, they can be detected and fenced.
  • If image references use inconsistent forms, they can be normalized.
  • If table output follows a known malformed pattern, targeted corrections can be applied.
This does not mean that every parser should become a large collection of brittle document-specific rules.
It means that parsing should be treated as an observable and testable engineering layer. When a failure appears repeatedly across the target corpus, a focused correction may provide more value than replacing the entire parser.
The missed borderless table also shows that this process is iterative. A custom pipeline is not automatically complete merely because it outperforms the tested alternatives.

How should teams choose a PDF parser?

The overall benchmark score is useful, but it should not be the only selection criterion.
The right choice depends on the documents and the downstream task.

Start with representative documents

A parser should be evaluated on the actual types of PDFs the system will process.
A benchmark dominated by financial reports may tell little about performance on equipment manuals, scanned forms or academic papers.
Include the difficult cases, not only clean documents.

Define which elements matter

Different applications need different forms of fidelity.
For example:
  • A compliance assistant may depend on heading hierarchy and regional qualifiers.
  • A financial analysis system may prioritize tables and charts.
  • A developer assistant may require accurate code blocks.
  • A research assistant may need equations and citations.
  • A multimodal Q&A system may need reliable image extraction.
The evaluation rubric should reflect these priorities.

Measure structure separately from text

A single text-similarity score can hide serious parsing errors.
At minimum, teams should evaluate:
  • Reading order
  • Heading hierarchy
  • Table structure
  • Images
  • Code or formulas where applicable
  • Raw text accuracy
This reveals whether the content is merely present or actually usable.

Decide whether one parser is enough

An off-the-shelf parser may be sufficient when the documents are consistent and its failure modes do not affect the application.
For more complex corpora, a pipeline may be more appropriate. It can use a strong base parser and add targeted processing for the structures that matter most.
In some systems, different parsers may even be selected for different document categories.

Key takeaway

There is no single PDF parser that gets everything right.
In this focused benchmark, the tools specialized:
  • LlamaParse led text and math.
  • PaddleOCR and AWS Textract performed strongly on tables.
  • Several tools extracted heading text reliably.
  • GenieParse performed best overall because it preserved hierarchy, reading order, images and code more consistently across the document set.
The clearest pattern was that a pipeline with targeted fixes on top of a capable base parser outperformed the unmodified tools on average.
This reinforces a broader lesson from our earlier probes:
Generic, one-size-fits-all components often fall short when real documents contain varied and interacting structures.
Document parsing is not merely a preprocessing utility. It is an engineering layer that determines the context available to retrieval and answer-generation systems.
Designing and evaluating it accordingly can directly improve downstream accuracy.