Technical report · Public benchmarks

Measured, or not run at all.

A public ledger of benchmark executions for the OCRs.ai document pipeline. Every number is traceable to a committed artifact; every gap is labelled with its status. Nothing on this page is projected, estimated or inferred.

Harness: ocrs-bench v0.1.0
Artifact schema: ocrs-bench/1
Committed artifacts: 0 under benchmarks/published/
Measured runs: 0
not_run targets: 9
Cloud cost axis: unavailable

State of the evidence

No measured public results are published here — yet.

The three vendor datasets are not vendored into this repository, and cloud credentials were not exercised for publication. So every public-suite target is not_run, and this page shows exactly that instead of a score. When a run executes, its deterministic summary.json is committed under benchmarks/published/, and every value on this page updates from that artifact — nothing else.

Real ParseBench / OmniDocBench / olmOCR-bench executions
Vendor datasets are not vendored into this repository and are never auto-downloaded.
not_run
balanced / agentic cloud runs
Not-run without OCRS_API_KEY — every sample carries not_run: missing_credentials.
not_run
Cloud monetary cost
The API exposes none. ocr_duration_ms and page counts are captured instead; cost_usd stays null, never $0.
unavailable
Raster golden cases without Tesseract
not_run: missing_engine on machines without the binary, while born-digital PDF cases still execute.
not_run

§ 01

Protocol

The harness is offline-first and never fabricates results. Six rules are enforced in code across every run, on every tier.

01Three statuses, never conflated
Every unit of work is exactly one of ok, failed or not_run. Missing credentials, missing datasets and missing engines are not_run with a reason — never a silent pass, never a red failure.
02Results are never fabricated
A run that did not execute produces no score. Aggregates report not_run counts separately so a run can never look like a pass when nothing actually ran.
03No automatic downloads
Vendor datasets are obtained manually from each vendor. The harness only validates whatever is already on disk under --dataset-root.
04Deterministic artifacts
Identical inputs map to an identical run_id (sha256 of benchmark + runner + config + sample ids, no timestamps) and byte-identical JSON/CSV except the explicit provenance timestamps.
05Full provenance per run
Every summary embeds UTC timestamp, git commit + dirty flag + branch, package versions, platform and the Tesseract binary version. A score is only meaningful together with its summary.json.
06Cost reported only when real
Monetary cost appears only when an API actually exposes it. Cloud runs carry cost_usd: null with cost_available: false — a missing cost is never rendered as $0.

§ 02

Tiers under test

The harness evaluates the three pipeline tiers exactly as the API exposes them — same cascade, same QC controls, same runners.

Fast

fast

Local runner · SDK pipeline · offline

cost:measured$0.0000 measured

Tesseract-first OCR cascade; Document AI is invoked only on escalated pages
QC off unless explicitly requested
Born-digital PDFs parse natively (pypdfium2) with no OCR engine at all
Raster samples need Tesseract/PaddleOCR, else per-sample not_run: missing_engine

Balanced

balanced

Cloud runner · requires OCRS_API_KEY

cost:unavailable

The same tier-aware cascade as Fast, with the QC reconcile applied on escalated pages
Default tier for API jobs
Without OCRS_API_KEY every sample is not_run: missing_credentials

Agentic

agentic

Cloud runner · requires OCRS_API_KEY

cost:unavailable

Document AI on every page, with the Tesseract cross-check
QC forced on — the pre-tier pipeline, unchanged
Without OCRS_API_KEY every sample is not_run: missing_credentials

Cost semantics are asymmetric by design: a local Fast run records 0.0 as a measured value, while cloud runs keep cost_usd: null with cost_available: false, because the API exposes no monetary cost. A missing cost is never rendered as $0 anywhere on this page.

§ 03

Accuracy versus cost

The plot every vendor page draws — here with an empty coordinate system until a measured run exists. The axes are the promise; the points will be the evidence.

Fig. 1 — accuracy (primary score) versus measured cost per run. Runs appear here only when a summary.json with a measured cost and accuracy is committed. The ledger table below is the complete text equivalent.

measured — plotted
measured, cost unavailable — excluded from the plot, listed in the ledger
not_run — no coordinates exist

§ 04

The run ledger

Every target this page tracks: status, accuracy, latency, cost and provenance. The complete text equivalent of Fig. 1 — including everything the figure honestly refuses to draw.

10 targets · 0 measured · 9 not_run · 1 unavailable

Run ledger: status, accuracy, latency, cost and provenance for every benchmark target. This table is the complete text equivalent of the accuracy-versus-cost figure.
Run target	Status	Accuracy	Latency p50 / p95	Cost	Run · provenance
olmOCR-Benchrunner: fast	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root).	—	—	—	—
olmOCR-Benchrunner: balanced	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
olmOCR-Benchrunner: agentic	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
OmniDocBenchrunner: fast	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root).	—	—	—	—
OmniDocBenchrunner: balanced	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
OmniDocBenchrunner: agentic	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
ParseBenchrunner: fast	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root).	—	—	—	—
ParseBenchrunner: balanced	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
ParseBenchrunner: agentic	not_runVendor dataset is not vendored into this repository and is never auto-downloaded (obtain it from the vendor, place it under --dataset-root). Cloud tier: without OCRS_API_KEY every sample is not_run: missing_credentials.	—	—	—	—
Golden setrunner: fast	unavailableAlways runnable offline, but no golden run is committed to this page yet. Raster cases not_run (missing_engine) on machines without Tesseract while born-digital PDF cases still execute.	—	—	—	—

— no measured value exists for this cell. unavailable means the value cannot exist — the API exposes no monetary cost — and it is never rendered as $0. Run ids are the harness's deterministic sha256 of benchmark + runner + config + sample ids.

§ 05

Suites and their provenance

Each adapter documents what is verified from the vendor's own paper and README, what is implemented locally, and what is explicitly left to the vendor's evaluator. Verified means verified — not approximated.

olmOCR-Bench

AllenAI

github.com/allenai/olmocr arXiv:2502.18443

Verified & implemented locally

Binary pass/fail unit semantics: presence, absence, order, baseline
The paper's documented text-normalization steps
Per-category pass rate, macro-averaged overall

Vendor-only / not re-implemented

Table and formula render-based matching (KaTeX layout, table cell neighborhoods) — counted as unit_tests_vendor_only and excluded from local scoring
Groups with zero locally-applicable tests are excluded from the macro-average and listed in groups_without_local_tests
Full scoring needs pip install 'olmocr[bench]' and the vendor's own unit files

OmniDocBench

OpenDataLab

github.com/opendatalab/OmniDocBench arXiv:2412.07626

Verified & implemented locally

Ground-truth page JSON schema (layout_dets + page_info) is validated locally
Markdown predictions are emitted in the vendor's documented shape
A documented vendor config YAML (end2end_eval) is generated for the official evaluator

Vendor-only / not re-implemented

Vendor metrics (Edit_dist, TEDS, CDM and friends) are not re-implemented — run the generated config inside OmniDocBench's own environment
The evaluator's external toolchain (TeX Live, ImageMagick, Ghostscript for CDM) is vendor-side

ParseBench

LlamaIndex / run-llama

github.com/run-llama/ParseBench huggingface.co/datasets/llamaindex/ParseBench

Verified & implemented locally

The five documented dimensions (table, chart, text_content, text_formatting, layout) and their official metric names
The vendor output layout output/<pipeline>/<dimension>/*.result.json

Vendor-only / not re-implemented

Every vendor metric implementation (GTRM, ChartDataPointMatch, Content Faithfulness, Semantic Formatting, Element Pass Rate) requires the vendor's parse-bench package and dataset
A local reference_similarity proxy is computed only when a fixture supplies reference markdown — clearly labelled, never presented as a ParseBench score

Golden set

OCRs.ai (in-repo)

always runnable · offline · no vendor data

Fixtures are generated from strings authored in this repository — no third-party documents, no secrets, no network
Deterministic PDF 1.4 writer and Pillow-rendered PNG fixtures
Regression gates: min_pass_rate, min_similarity_mean, max_latency_ms_p95 (breach exits 1)
Cases that could not run count as cases_not_run and only fail the --strict gate

§ 06

Run book — reproduce any of it

Verbatim harness commands. Obtain the datasets from the vendors listed above; the harness never downloads anything for you, and exit codes are documented: 0 ok, 1 golden regression, 2 usage error, 3 strict not-run gate.

Run from the repository root · any Python ≥ 3.10 with the SDK · Tesseract optional (raster inputs only)

Check what the harness can see

Reports dataset availability without downloading anything. Exits 0 even when unavailable.

PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench status \  --benchmark olmocr --dataset-root /path/to/olmocr-bench-dataset

Run a public benchmark locally (Fast tier)

Free and offline for born-digital PDFs; raster inputs additionally need Tesseract or PaddleOCR.

PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench run \  --benchmark olmocr --dataset-root /path/to/olmocr-bench-dataset \  --runner fast --output benchmark-results

Run a cloud tier

Requires OCRS_API_KEY; without it the run is recorded not_run rather than failing.

export OCRS_API_KEY=ocr_...PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench run \  --benchmark omnidocbench --dataset-root /path/to/OmniDocBench \  --runner balanced --output benchmark-results

Run the golden regression set

Always runnable: generated fixtures authored in this repository, no vendor data, no network.

PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench golden \  --runner fast --output benchmark-results/golden --strict

Merge runs into a Pareto CSV

One row per run — accuracy, cost, latency, sample counts and provenance — ready to chart.

PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench pareto \  a/summary.json b/summary.json --out pareto.csv

Print the provenance block

Quote this when re-running: git commit, package versions, machine, Tesseract binary.

PYTHONPATH=benchmarks sdk/.venv/bin/python -m ocrs_bench env

Publish a run to this page

Run any command above, then commit the written benchmark-results/<run_id>/summary.json under benchmarks/published/<run_id>/. The page validates each artifact against the ocrs-bench/1 contract at build time and renders only what passes — run id, git commit, counts, latency, cost status and the primary score, exactly as the harness wrote them.

API documentation

§ 07

Colophon

What this page is made of, and what it deliberately is not.

Data sources

Harness documentation — benchmarks/README.md at ocrs-bench v0.1.0 — for the protocol, tier semantics and known not-run state.
Adapter vendor blocks — benchmarks/ocrs_bench/adapters/*.py — for what is verified versus vendor-only per suite.
Committed artifacts — benchmarks/published/*/summary.json (0 currently) — the only source of measured values. Rendered at build time; no client-side fetching, no third-party scripts on this page.
OCRs.ai list prices — this site's own pricing and comparison data, quoted below for orientation only.

List-price context — not measurements

OCRs.ai

$0.012/page with QC · $0.008 without (list price, pay-as-you-go)pricing →

Google Document AI

Enterprise / cloud billing (typically higher per page)comparison →

AWS Textract

AWS metered (forms/tables add-ons)comparison →

Azure Document Intelligence

Azure metered / commitment tierscomparison →

Quoted from the site's published pricing and comparison pages for orientation. These are list prices, not benchmark measurements, and none of them is plotted or scored anywhere on this page.