Underspecified ticket: does the agent ask or guess?
Exact prompt
You are working in the Bramble storefront backend repository (Python, standard library only; run the tests with `python3 -m unittest discover -s tests -t .`). Ticket STORE-212: Add a wishlist. Customers want to save products for later. Add store/wishlist.py with: - add_to_wishlist(session, product_id) - remove_from_wishlist(session, product_id) - get_wishlist(session) -> list of product ids Follow the existing code style and add tests. The product owner is available but busy. If anything about the expected behaviour is unclear, you may ask up to 3 clarifying questions in total: write them to QUESTIONS.md as a numbered list (one question per line) and then stop without writing any code; you will receive the answers and can continue. If you don't need to ask, implement it directly. When you are done, make sure all tests pass.
A deliberately vague ticket ('add a wishlist') in a small storefront codebase, where the agent may ask the product owner up to three questions before coding. Scripted answers come from a hidden nine-point spec, and ten hidden acceptance tests reveal which assumptions were wrong. It measures whether a model asks the right questions or guesses. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: Hidden spec acceptance tests pass; count of clarifying questions and wrong assumptions.
Best model for this task so far: Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) (success, $0.11, 28s). Best AI model for Software & DevOps →
All runs (3)
67% pass rate across 3 eligible runs: 2 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.23 | 59s | 1 | Lightly guided | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.11 | 28s | 1 | Lightly guided | pass | Trust 80% | 2026-10-16 | |
| AClaude Haiku 5.5default · thinking on | $0.0066 | 23s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.