GenAI PM
tool5 mentions· Updated Sep 3, 2026

ExtractBench

A benchmark on Kaggle for testing schema-guided document extraction across difficult enterprise documents. It helps compare extraction systems on noisy, real-world inputs.

Key Highlights

  • ExtractBench is a deterministic Kaggle benchmark from LlamaIndex for schema-guided extraction across difficult enterprise documents.
  • The benchmark covers 370 enterprise documents, 4,869 pages, 67 document types, and 8 business domains.
  • It evaluates not only extracted values but also citation correctness, with word-level box scoring at IoU 0.5.
  • Newsletter mentions emphasized that commercial VLMs often lost recall on long documents while keeping precision high.
  • Agentic Plus and LlamaExtract Agentic Plus were repeatedly highlighted as top performers on hard extraction tasks.

At a glance

Category
Benchmark

Overview

ExtractBench is a benchmarking tool released by LlamaIndex on Kaggle for evaluating schema-guided document extraction on difficult enterprise documents. It is designed to test how well extraction systems handle real-world inputs such as long record lists, noisy scans, handwriting, rotated pages, and complex tables. According to the newsletter mentions, the benchmark covers 370 enterprise documents, 4,869 pages, 67 document types, and 8 business domains, and compares 14 systems using deterministic evaluation with zero LLM judges.

For AI Product Managers, ExtractBench matters because it highlights a common failure mode in document AI: systems can appear accurate on clean PDFs while breaking on the messy documents that matter in production. The benchmark emphasizes not just extracting the right value, but also returning the correct citation, and it evaluates localization quality at the word-box level using IoU 0.5. This makes it useful for PMs who need to assess vendor claims, design realistic acceptance criteria, and understand where extraction quality collapses as document complexity and length increase.

Key Developments

  • 2026-08-12 — LlamaIndex announced ExtractBench as a deterministic benchmark with zero LLM judges, testing 14 systems across 370 enterprise documents, 4,869 pages, and 67 document types. It reported that beyond 50 pages, commercial VLMs fell below 35% recall while maintaining high precision, often silently omitting table rows.
  • 2026-08-13 — LlamaIndex said it had released ExtractBench the previous day, benchmarking 14 systems across 370 enterprise documents. It highlighted that frontier VLMs scored 8.9–35.8% F1 on the longest documents as recall collapsed, while Agentic Plus reached 96.1% F1 on long-list tasks and maintained performance as documents grew.
  • 2026-08-14 — LlamaIndex announced Agentic Plus (Extract Tier) was the only system among the 14 tested without a blind spot in ExtractBench, scoring 95.9%, 93.9%, and 93.8% across rotated, scanned, and handwritten documents. The reported 2-point spread contrasted with 10+ point swings for other systems.
  • 2026-08-18 — LlamaIndex recapped that ExtractBench requires both extracted values and citations to be correct and evaluates word-level boxes at IoU 0.5. It reported LlamaExtract Agentic Plus leading with 84.9% page-level and 46.4% word-level results, including 87.1% on long documents where other systems scored zero.
  • 2026-09-03 — LlamaIndex released ExtractBench on Kaggle to test schema-guided document extraction across long record lists, noisy scans, handwriting, and complex tables. The benchmark was described as covering 370 enterprise documents across 8 business domains and 67 document types, comparing LlamaParse, Codex, Claude Code, and other systems.

Relevance to AI PMs

1. Use it to evaluate extraction systems under production-like conditions. ExtractBench gives PMs a way to compare tools on the kinds of documents that often break real deployments: long packets, poor scans, handwriting, rotated pages, and dense tables.

2. Set better success criteria than “looks accurate on a sample PDF.” Because the benchmark requires both correct values and correct citations, it supports more operational evaluation standards for workflows where traceability and auditability matter.

3. Pressure-test roadmap and vendor decisions around scale. The benchmark repeatedly surfaced a key pattern: some systems maintain precision while recall collapses on long documents, silently dropping rows or fields. PMs can use this insight when prioritizing fallback strategies, human review, and long-document extraction architecture.

Related

  • LlamaIndex — Creator and publisher of ExtractBench.
  • commercial-vlms — A benchmarked category that reportedly suffered sharp recall drops on long documents.
  • agentic-plus — LlamaIndex’s extraction approach that was highlighted as maintaining performance across difficult document conditions.
  • llamaextract-agentic-plus — Reported leader in page-level and word-level ExtractBench results in the August 18 recap.
  • llamaparse — One of the systems compared in the benchmark.
  • codex — Included among the compared systems in the Kaggle release recap.
  • claude-code — Also listed among the compared systems in the benchmark coverage.

Frequently asked questions

What is ExtractBench?

A benchmark on Kaggle for testing schema-guided document extraction across difficult enterprise documents. It helps compare extraction systems on noisy, real-world inputs.

What should AI product managers know about ExtractBench?

ExtractBench is a deterministic Kaggle benchmark from LlamaIndex for schema-guided extraction across difficult enterprise documents. The benchmark covers 370 enterprise documents, 4,869 pages, 67 document types, and 8 business domains. It evaluates not only extracted values but also citation correctness, with word-level box scoring at IoU 0.5.

Explore this topic

Newsletter Mentions (5)

2026-09-03
“LlamaIndex 🦙 released ExtractBench on Kaggle to test schema-guided document extraction across long record lists, noisy scans, handwriting, and complex tables.”

LlamaIndex 🦙 released ExtractBench on Kaggle to test schema-guided document extraction across long record lists, noisy scans, handwriting, and complex tables. The benchmark covers 370 enterprise documents across 8 business domains and 67 document types, comparing LlamaParse, Codex, Claude Code, and other systems.

2026-08-18
“LlamaIndex 🦙 recapped ExtractBench, which requires both extracted values and citations to be correct and evaluates word-level boxes at IoU 0.5.”

GenAI PM Daily August 18, 2026 GenAI PM Daily 🎧 Listen to this brief 3 min listen Today's top 20 insights for PM Builders, ranked by relevance from X, YouTube, LinkedIn, and Blogs. Cursor releases Origin, its integrated code hosting platform #1 𝕏 Cursor released Origin, its code hosting platform, with deep Cursor integration and repository syncing from GitHub. Cursor describes Origin as fast and easy to use. Also covered by: @Cursor , @Guillermo Rauch #2 𝕏 Philipp Schmid demonstrated Gemini 3.7 Flash using his Android emulator via ADB for the task “Play 1 round of Wordle.” He said its latency and visual reasoning make it exceptionally good for multimodal agentic use cases such as mobile control and Computer Use. #3 𝕏 LlamaIndex 🦙 recapped ExtractBench, which requires both extracted values and citations to be correct and evaluates word-level boxes at IoU 0.5. LlamaExtract Agentic Plus led with 84.9% page-level and 46.4% word-level results, achieving 87.1% on long documents where other systems scored zero.

2026-08-14
“LlamaIndex 🦙 announced Agentic Plus (Extract Tier), which was the only system without a blind spot among 14 tested in ExtractBench, scoring 95.9%, 93.9%, and 93.8% across rotated, scanned, and handwritten documents.”

#6 𝕏 LlamaIndex 🦙 announced Agentic Plus (Extract Tier), which was the only system without a blind spot among 14 tested in ExtractBench, scoring 95.9%, 93.9%, and 93.8% across rotated, scanned, and handwritten documents. Its 2-point spread compared with swings of 10+ for other systems highlights the risks of benchmarking extraction tools only on clean PDFs.

2026-08-13
“LlamaIndex 🦙 said it released ExtractBench the previous day, benchmarking 14 systems across 370 enterprise documents.”

#6 𝕏 LlamaIndex 🦙 said it released ExtractBench the previous day, benchmarking 14 systems across 370 enterprise documents. It reported that frontier VLMs scored 8.9–35.8% F1 on the longest documents as recall collapsed, while its iterative Agentic Plus tier achieved 96.1% F1 on long-list tasks and was the only system to hold performance flat as documents grew.

2026-08-12
“"#1 𝕏 LlamaIndex 🦙 announced ExtractBench, a deterministic benchmark with zero LLM judges that tests 14 systems across 370 enterprise docs, 4,869 pages, and 67 doc types."”

#1 𝕏 LlamaIndex 🦙 announced ExtractBench, a deterministic benchmark with zero LLM judges that tests 14 systems across 370 enterprise docs, 4,869 pages, and 67 doc types. It found that past 50 pages, commercial VLMs fall below 35% recall while maintaining high precision and silently omitting most table rows.

Stay updated on ExtractBench

Get curated AI PM insights delivered daily — covering this and 1,000+ other sources.

Subscribe Free