Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Write a pytest suite for a money module, graded by killing 14 mutants

AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.37
Wall time: 1.4h
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: write a strong test suite (graded by mutation testing) `money.py` contains three small functions with their contracts in docstrings. Do NOT modify `money.py`. Write `tests/test_money.py` (pytest) that thoroughly tests the documented behaviour, including edge cases. Your suite is graded by running it against many subtly-broken copies of `money.py` ("mutants"): it must pass on the real module and fail on as many mutants as possible. Run with `python -m pytest -q`. No network, no new dependencies.

Must pass on the real code and kill ≥13/14 mutants (boundary, sign, rounding, regex-width, symbol). Rewards reading docstring contracts precisely. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. money.py must be byte-identical to the original. 2. The agent's test suite must pass on the original module. 3. It must fail on (kill) at least 13 of the 14 mutants (each mutant run separately with pytest -x, 120 s timeout).

Setup notes: Use Case Scout batch 4 case d08-tests-mutation-graded (offline fixture). Prepared with batch4/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-opus-5.5: $0.365121. Score: 2/2 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.37
Wall time: 1.4h
Human step-ins: 0
Turns: 12

Model usage
Model	Input tok	Output tok	Cost
AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on	248,703	6,266	$0.37

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. money.py must be byte-identical to the original. 2. The agent's test suite must pass on the original module. 3. It must fail on (kill) at least 13 of the 14 mutants (each mutant run separately with pytest -x, 120 s timeout).
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 128951,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.36512100000000003,
  "usage": {
   "input_tokens": 22,
   "cache_creation_input_tokens": 24356,
   "cache_read_input_tokens": 224325,
   "output_tokens": 6266,
   "output_tokens_details": {
    "thinking_tokens": 280
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 24356,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 720,
     "cache_read_input_tokens": 28911,
     "cache_creation_input_tokens": 188,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 188
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 22,
    "outputTokens": 6266,
    "cacheReadInputTokens": 224325,
    "cacheCreationInputTokens": 24356,
    "webSearchRequests": 0,
    "costUSD": 0.36512100000000003,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 280,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 12,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `tests/test_money.py` (174 test cases) and left `money.py` untouched. I couldn't run `python -m pytest -q` as asked: there is no `python` command here (only `python3`), and pytest isn't installed. Instead I ran the suite with a throwaway stand-in for pytest that I wrote, which handles `parametrize` and `raises`. All 174 cases pass on the real module, but a real pytest run is still unconfirmed.\n\n**What's covered:**\n- **`split_amount`:** exact expected lists for positive, negative and zero totals, plus edge cases. A sweep of totals \u00d7 part counts checks that:\n  - there are exactly n parts and they add up to the total;\n  - the parts differ by at most 1 and the extra cents go to the earlier parts;\n  - negative totals give parts that are all \u2264 0 and mirror the positive split;\n  - an invalid n raises `ValueError` with the right message.\n- **`parse_money`:** all the examples from the docstring, plus:\n  - one- and two-digit decimals with either `.` or `,` as the separator;\n  - three or more trailing digits, which are treated as a whole-number part;\n  - leading or trailing separators (`.5`, `5.`) and the \u20ac/$/\u00a3 symbols;\n  - whitespace inside and around the amount, including inside the brackets;\n  - double negatives like `(-3)` and a minus sign after the currency symbol;\n  - invalid inputs that must raise `ValueError` (empty, `-`, `()`, unbalanced brackets, letters, `-(3)`, `--3`).\n- **`format_money`:** defaults, zero, sub-1\u20ac amounts, the 999,99 \u2192 1.000,00 boundary, negatives, every keyword argument on its own, and a round trip through `parse_money`.\n\n**Mutation check:** I hand-made 14 broken copies of `money.py` and ran the suite against each. 12 were caught. The other two behave exactly like the original (`total_cents < 0` \u2192 `<= 0`, and the `\"0\"` fallback \u2192 `\"00\"`), so no test can catch them.",
  "ttft_ms": 1775,
  "type": "result",
  "duration_ms": 130406,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 802,
  "time_to_request_ms": 232,
  "first_content_frame_ms": 802,
  "queued_turn_count": 0,
  "result_index": 0
 }
]