Write a pytest suite for a money module, graded by killing 14 mutants
Exact prompt
# Task: write a strong test suite (graded by mutation testing) `money.py` contains three small functions with their contracts in docstrings. Do NOT modify `money.py`. Write `tests/test_money.py` (pytest) that thoroughly tests the documented behaviour, including edge cases. Your suite is graded by running it against many subtly-broken copies of `money.py` ("mutants"): it must pass on the real module and fail on as many mutants as possible. Run with `python -m pytest -q`. No network, no new dependencies.
Must pass on the real code and kill ≥13/14 mutants (boundary, sign, rounding, regex-width, symbol). Rewards reading docstring contracts precisely. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. money.py must be byte-identical to the original. 2. The agent's test suite must pass on the original module. 3. It must fail on (kill) at least 13 of the 14 mutants (each mutant run separately with pytest -x, 120 s timeout).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 1m).
All runs (4)
75% pass rate across 4 eligible runs: 3 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.07 | 1m | 0 | Fully autonomous | partial | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.37 | 1.4h | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.10 | 35s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.