Skip to content

QA a small app against its spec before launch and report only real bugs, where an empty list is a valid answer

Exact prompt

# QA a small app against its spec before launch and report only real bugs, where an empty list is a valid answer `app/shop.py` is a small synthetic storefront module, and `SPEC.md` is its specification with numbered clauses (S1, S2, ...). The beta tester's notes are in `beta-feedback.txt`. Everything is synthetic and offline. Find where the code breaks the spec. You may read, run and test the code, but do not change anything under `app/`. Rules: - A bug is a call whose result (a return value, or the exception type raised) differs from what SPEC.md requires. Behaviour the spec requires is never a bug, even if it looks odd or a note calls it one; anything the spec doesn't cover is not a bug either. - Report each bug once under the one clause it breaks, with a call that shows it: the function name and a JSON list of positional arguments. Your call must actually give the wrong result when run against app/shop.py as shipped. - Report nothing you haven't confirmed. False reports count against you, and if the code meets the spec, the right answer is an empty list. - Write `out/findings.json`: a JSON list (possibly empty) of `{"clause": "S<n>", "call": "<function name>", "args": [<JSON arguments>], "expected": <what SPEC.md requires>, "actual": <what the code returns, or {"raises": "<ExceptionType>"}>}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Check a small synthetic storefront module against a numbered spec and report only confirmed deviations, each with a call that reproduces it. Each seed plants zero to three bugs; a quarter of the seeds plant none, and then an empty list is the right answer. A beta tester's notes flag behaviour the spec actually requires. Every reported call is replayed on the box against the shipped code and a spec-correct copy, so false positives and made-up evidence cost points. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: right_verdict found_real_bugs no_false_clauses evidence_reproduces

Best model for this task so far: Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) (success, $0.05, 9s).

Run this on another model

All runs (3)

67% pass rate across 3 eligible runs: 2 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.1636s0Fully autonomouspassTrust 80%2026-10-25
AClaude Haiku 5.5default · thinking on$0.0149s0Fully autonomouspartialTrust 80%2026-10-25
AClaude Sonnet 5.5default · thinking on$0.059s0Fully autonomouspassTrust 80%2026-10-25

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for QA a small app against its spec before launch and report only real bugs, where an empty list is a valid answer

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.