Answer 6 multi-hop questions from ~150KB of meeting notes with corrections and decoys
Exact prompt
# Task: answer questions from a year of meeting notes `meeting_notes.txt` holds ~400 meeting entries (≈220 KB). Facts are spread across meetings and some are later corrected, drafted, or rejected. Answer from the notes only. Write `answers.json`: {"q1": {"answer": "...", "meeting": <meeting number that is the authoritative source>}, ...} - q1: What capex was approved for the warehouse-robotics initiative? - q2: Who approved it? - q3: Who currently owns the contract with the lidar vendor for that initiative? - q4: Who chairs the security standup now? - q5: Which firmware version fixed the root cause of the lidar incident? - q6: When and where is the robotics lab moving (month and building)? No network. Put only the current answer in "answer"; any context about superseded values must go in parentheses (parenthetical text is ignored by grading).
Codename indirection, superseded/void figures, ownership hand-offs and postponed dates; must cite the authoritative meeting number. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 6 answers must contain the required token(s) for the authoritative (latest/approved) fact, case-insensitive. 2. Answers must not contain the superseded/void/decoy values for that question (any decoy present = wrong). 3. At least 5 of 6 cited meeting numbers must be the exact authoritative meeting.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 42s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.04 | 27s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 42s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.14 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.08 | 22s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.