ExtractBench

Overview

ExtractBench is a document extraction benchmark released by LlamaIndex on Kaggle for evaluating schema-guided extraction on difficult, real-world enterprise documents. Rather than focusing on clean PDFs or narrow datasets, it tests systems across noisy scans, handwriting, rotated pages, long record lists, and complex tables. The benchmark is designed to measure whether extraction systems can reliably pull structured fields from messy business documents while also grounding outputs with citations.

For AI Product Managers, ExtractBench matters because it exposes a common failure mode in enterprise AI: systems that look accurate on simple samples but break on long, noisy, or operationally critical documents. Its deterministic evaluation approach, including citation correctness and word-level box matching, makes it useful for comparing tools, validating vendor claims, and setting realistic quality bars for production document workflows.

Key Developments

• 2026-08-12: LlamaIndex announced ExtractBench as a deterministic benchmark with zero LLM judges, testing 14 systems across 370 enterprise documents, 4,869 pages, and 67 document types. It highlighted that beyond 50 pages, commercial VLMs dropped below 35% recall while often maintaining high precision, silently missing many table rows.
• 2026-08-13: LlamaIndex said it had released ExtractBench the previous day, benchmarking 14 systems across 370 enterprise documents. It reported that frontier VLMs scored just 8.9%–35.8% F1 on the longest documents as recall collapsed, while Agentic Plus achieved 96.1% F1 on long-list tasks and held performance steady as document length increased.
• 2026-08-14: LlamaIndex announced Agentic Plus (Extract Tier) results from ExtractBench, stating it was the only system among 14 tested without a clear blind spot. It scored 95.9%, 93.9%, and 93.8% across rotated, scanned, and handwritten documents, showing far less variance than competing systems.
• 2026-08-18: LlamaIndex recapped that ExtractBench requires both extracted values and citations to be correct, and evaluates word-level boxes at IoU 0.5. It reported LlamaExtract Agentic Plus leading with 84.9% page-level and 46.4% word-level results, including 87.1% on long documents where other systems scored zero.
• 2026-09-03: LlamaIndex released ExtractBench on Kaggle for testing schema-guided document extraction across long record lists, noisy scans, handwriting, and complex tables. The benchmark was described as covering 370 enterprise documents across 8 business domains and 67 document types, comparing LlamaParse, Codex, Claude Code, and other systems.

Relevance to AI PMs

1. Improves vendor and model evaluation: ExtractBench gives AI PMs a more realistic way to compare extraction platforms and multimodal models than clean-demo benchmarks. It is especially useful when selecting tools for invoices, claims, onboarding forms, financial statements, or other enterprise document pipelines.

2. Helps define production-ready quality metrics: Because the benchmark checks both extracted values and supporting citations, PMs can use its framing to build stronger acceptance criteria for products where auditability, traceability, and human review matter.

3. Reveals failure modes that affect roadmap and UX: The benchmark shows how performance can collapse on long documents, handwritten content, scanned pages, and dense tables. PMs can use these insights to prioritize fallback workflows, escalation paths, confidence thresholds, and domain-specific testing before launch.

Related

• llamaindex: Creator and publisher of ExtractBench, and the main source of benchmark announcements and results.
• llamaparse: One of the document processing/extraction systems compared in benchmark-related coverage.
• commercial-vlms: A key comparison category in ExtractBench, especially around recall drop-offs on long documents.
• agentic-plus: LlamaIndex's agentic extraction approach that reportedly performed strongly on difficult and long-document tasks in ExtractBench.
• llamaextract-agentic-plus: The specific extraction tier highlighted as a leading performer in benchmark recaps.
• codex: Included among systems referenced in ExtractBench comparisons.
• claude-code: Also referenced as a system compared in benchmark coverage.