Extract 8 fields from 24 invoices in 4 formats (EU numbers, credit notes, MM/DD dates)
Exact prompt
# Task: extract structured data from mixed-format invoices `invoices/` has 24 files (txt, html, eml, kv). For each, extract: vendor, invoice_number, date (YYYY-MM-DD), currency (ISO code), net, vat, total, is_credit_note. Rules: credit notes have NEGATIVE net/vat/total. European number formats (1.234,56) must be parsed correctly. Ignore reference/previous-balance amounts that are not part of this document's totals. If a currency symbol is used, map € EUR, £ GBP, $ USD. Write `extracted.json`: {"inv01": {...}, ..., "inv24": {...}} (key = filename stem). No network.
Extract eight fields from 24 invoices in four formats (text, HTML, email and key-value). Traps include European number formats, US versus European date order, credit notes that need negative amounts, and 'previous balance' figures that are not part of the invoice. The bar is near-perfect field accuracy. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. 24 invoices × 8 fields compared to the key: numeric fields within 0.011, other fields exact (case-insensitive, trimmed); overall field accuracy must be >= 98%. 2. Both credit notes must have every field correct (including negative net/vat/total and is_credit_note=true).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 35s).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.05 | 27s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.01 | 59s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 35s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.17 | 26s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.08 | 20s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.