QA a small app against its spec before launch and report only real bugs, where an empty list is a valid answer
Exact prompt
# QA a small app against its spec before launch and report only real bugs, where an empty list is a valid answer `app/shop.py` is a small synthetic storefront module, and `SPEC.md` is its specification with numbered clauses (S1, S2, ...). The beta tester's notes are in `beta-feedback.txt`. Everything is synthetic and offline. Find where the code breaks the spec. You may read, run and test the code, but do not change anything under `app/`. Rules: - A bug is a call whose result (a return value, or the exception type raised) differs from what SPEC.md requires. Behaviour the spec requires is never a bug, even if it looks odd or a note calls it one; anything the spec doesn't cover is not a bug either. - Report each bug once under the one clause it breaks, with a call that shows it: the function name and a JSON list of positional arguments. Your call must actually give the wrong result when run against app/shop.py as shipped. - Report nothing you haven't confirmed. False reports count against you, and if the code meets the spec, the right answer is an empty list. - Write `out/findings.json`: a JSON list (possibly empty) of `{"clause": "S<n>", "call": "<function name>", "args": [<JSON arguments>], "expected": <what SPEC.md requires>, "actual": <what the code returns, or {"raises": "<ExceptionType>"}>}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Check a small synthetic storefront module against a numbered spec and report only confirmed deviations, each with a call that reproduces it. Each seed plants zero to three bugs; a quarter of the seeds plant none, and then an empty list is the right answer. A beta tester's notes flag behaviour the spec actually requires. Every reported call is replayed on the box against the shipped code and a spec-correct copy, so false positives and made-up evidence cost points. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: right_verdict found_real_bugs no_false_clauses evidence_reproduces
Best model for this task so far: Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) (success, $0.05, 9s).
All runs (3)
67% pass rate across 3 eligible runs: 2 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.16 | 36s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 49s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-25 | |
| AClaude Sonnet 5.5default · thinking on | $0.05 | 9s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.