Answer 8 business questions against a SQLite shop database
Exact prompt
# Task: answer business questions from a SQLite database `shop.db` is a SQLite database (use python's sqlite3; inspect the schema yourself). Only orders with status 'shipped' count as sales. Revenue = qty * unit_price from order_items (NOT list_price). Write `answer.json` with keys q1..q8: - q1: total shipped revenue (round 2) - q2: customer country with the highest shipped revenue (2-letter code) - q3: product category with highest shipped revenue NET of returns (subtract returned qty * that line's unit_price) - q4: number of customers with zero shipped orders - q5: returned units / shipped units, over shipped orders only (round 4) - q6: month (YYYY-MM) with highest shipped revenue - q7: number of distinct discontinued products that appear in shipped orders dated on/after 2026-07-01 - q8: average shipped order value (round 2) Also write `queries.sql` with the SQL you used. No network.
Answer eight business questions from a small SQLite shop database. Traps include using the order line's unit price instead of the list price, counting only shipped orders, netting out returns per line, and finding customers with no shipped orders. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 8 answers (q1–q8) must be correct: numeric answers within 0.01 (q5 within 0.0005); text answers exact match, case-insensitive. 2. Pass requires 8/8.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0057, 22s).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.03 | 39s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0057 | 28s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0057 | 22s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.15 | 35s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.06 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.