run-llama/ParseBench

Evaluation Methodology

ParseBench treats document parsing as an end-to-end agent task rather than isolated character recognition. The framework tests models across three tiers of complexity: structured extraction (tables/forms), layout preservation (headers, columns, reading order), and semantic coherence (contextual understanding across pages). Crucially, it measures functional correctness—feeding parsed output into downstream retrieval systems to test if the right chunks surface.

Data Pipeline

The benchmark uses an adversarial document corpus spanning scanned PDFs, degraded images, and complex layouts (multi-column, marginalia, mixed handwriting). Notably, it includes "parser traps"—documents designed to fool specific strategies like tables formatted as images or text-as-path SVG embeddings. All evaluations run in containerized environments with deterministic seeds to ensure reproducibility.

Beyond Character-Level Accuracy

Most document benchmarks optimize for CER (Character Error Rate). ParseBench introduces downstream task validation—if you feed the parsed output into a retrieval system, does it retrieve the right chunks? This aligns evaluation with actual RAG failure modes where perfect OCR produces unusable text blocks.

Agent-Centric Evaluation

Rather than static metrics, ParseBench evaluates parsing as a dynamic agent workflow. Models must handle multi-step decisions: choosing between OCR engines, deciding when to use vision vs. text extraction, and repairing inconsistencies. This mirrors how LlamaIndex agents consume parsers in production, testing adaptability rather than static capability.

Robustness Partitioning

The benchmark explicitly distinguishes between "in-distribution" capabilities (clean academic PDFs) and "out-of-distribution" robustness (scanned receipts, coffee-stained pages, mobile phone captures). This prevents overfitting to datasets like DocVQA and exposes how parsers degrade under real-world noise.

Early Capability Patterns

Initial evaluations reveal a stark capability chasm between traditional OCR pipelines and multimodal LLMs. Vision-language models (GPT-4V, Claude 3 Opus) demonstrate superior semantic table understanding but struggle with long-document coherence—they hallucinate hierarchical relationships when documents exceed 20 pages, flattening nested sections into linear text.

Surprising Finding: Models optimizing for character-level accuracy often score poorly on Agent Success Rate—the metric that matters for RAG. A "perfect" OCR producing unstructured paragraphs destroys retrieval precision compared to a "messy" parser preserving hierarchical headings.

LlamaIndex Native Integration

ParseBench isn't an academic exercise—it's tightly coupled with the LlamaIndex ecosystem, providing evaluators for SimpleDirectoryReader, Unstructured.io integrations, and MarkdownElementNodeParser. This positions it as a diagnostic tool for production pipelines rather than a research curiosity.

Community Momentum

Despite being days old, the repository shows immediate product-market fit with developers frustrated by "it works on my PDF" debugging. The 321% weekly velocity suggests pent-up demand for standardized document evaluation, particularly among enterprises building compliance and financial document pipelines.

Research Direction

The benchmark is shifting focus from recognition to understanding. Roadmap items include evaluating multimodal reasoning (parsing charts as queryable data structures) and hierarchical RAG (maintaining parent-child relationships in nested legal contracts). Expect submissions from closed-source providers (Anthropic, OpenAI) to establish baseline comparisons within weeks.