Answer 8 questions from 12 conflicting dated sources with correct citations
Exact prompt
# Task: answer questions from conflicting internal sources (as of 2026-10-15) `sources/` contains 12 dated documents about Voltaris's Kestrel Valley plant. They conflict. Prefer approved/official and newer documents; corrections override what they correct; drafts, rumors and unverified posts are not facts. Answer in `answers.json`: {"q1": {"answer": "...", "sources": ["<filename>", ...]}, ...}. Put the single most authoritative file FIRST in sources. - q1: Current planned annual capacity in GWh (number only) - q2: Current planned investment in EUR billion (number only) - q3: Current planned production start (e.g. "Q1 2028") - q4: Planned headcount at full ramp (integer) - q5: Primary cathode supplier (company name) - q6: Separator supplier (company name) - q7: Is the CEO leaving before launch, according to the best evidence? ("yes"/"no") - q8: What must be operational before production start under the water permit? (short phrase) Also write `brief.md` (≤200 words) summarising the current plan and flagging which sources were superseded or unreliable. No network.
Answer eight questions about a company's planned battery plant using twelve dated internal documents that contradict each other. Corrections override what they correct, approved minutes beat drafts, and rumors are not facts. Each answer must cite the single most authoritative file first. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. At least 7 of 8 answers correct (normalised: the expected value must appear in the given answer, ignoring case/punctuation/commas). 2. For at least 7 of 8 questions the FIRST file in 'sources' must be the authoritative source file. 3. brief.md must exist, be 1–200 words, and must not state the rumoured/draft figures (the unapproved capacity values or the rumoured headcount) unless explicitly labelled as rumour/draft/unverified/rejected/not approved.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0050, 16s).
All runs (5)
60% pass rate across 5 eligible runs: 3 pass, 0 partial, 2 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.04 | 40s | 0 | Fully autonomous | fail | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0072 | 40s | 0 | Fully autonomous | fail | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0050 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.15 | 24s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking off | $0.05 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.