Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Write a pytest suite for a money module, graded by killing 14 mutants

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.02
Wall time: 1m
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: write a strong test suite (graded by mutation testing) `money.py` contains three small functions with their contracts in docstrings. Do NOT modify `money.py`. Write `tests/test_money.py` (pytest) that thoroughly tests the documented behaviour, including edge cases. Your suite is graded by running it against many subtly-broken copies of `money.py` ("mutants"): it must pass on the real module and fail on as many mutants as possible. Run with `python -m pytest -q`. No network, no new dependencies.

Must pass on the real code and kill ≥13/14 mutants (boundary, sign, rounding, regex-width, symbol). Rewards reading docstring contracts precisely. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. money.py must be byte-identical to the original. 2. The agent's test suite must pass on the original module. 3. It must fail on (kill) at least 13 of the 14 mutants (each mutant run separately with pytest -x, 120 s timeout).

Setup notes: Use Case Scout batch 4 case d08-tests-mutation-graded (offline fixture). Prepared with batch4/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.015292. Score: 2/2 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.02
Wall time: 1m
Human step-ins: 0
Turns: 12

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	342,613	14,966	$0.02

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. money.py must be byte-identical to the original. 2. The agent's test suite must pass on the original module. 3. It must fail on (kill) at least 13 of the 14 mutants (each mutant run separately with pytest -x, 120 s timeout).
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 72675,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.01529212,
  "usage": {
   "input_tokens": 24,
   "cache_creation_input_tokens": 23057,
   "cache_read_input_tokens": 319532,
   "output_tokens": 14966,
   "output_tokens_details": {
    "thinking_tokens": 7597
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 23057,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 627,
     "cache_read_input_tokens": 34809,
     "cache_creation_input_tokens": 787,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 787
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 24,
    "outputTokens": 14966,
    "cacheReadInputTokens": 319532,
    "cacheCreationInputTokens": 23057,
    "webSearchRequests": 0,
    "costUSD": 0.01529212,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 7597,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 12,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `tests/test_money.py` with 50 tests covering `split_amount`, `parse_money` and `format_money`. `money.py` is unchanged.\n\n**Caveat: I did not run it under pytest.** `python` isn't on this machine and pytest isn't installed for `python3`. I ran the tests through a small stand-in for `pytest.raises` in `/tmp`, outside the repo. All 50 pass on the real module. Please run `python -m pytest -q` in an environment with pytest to confirm.\n\n**What the suite covers**\n- **`split_amount`:** remainder going to the earliest parts, zero and one-part splits, more parts than cents, negative totals split by magnitude, exact large values, `int` return types, an exhaustive invariant check (sum, sign, spread of at most one cent, non-increasing magnitudes), and `ValueError` for `n < 1`.\n- **`parse_money`:** both separator styles, the \"last separator with 1\u20132 digits is decimal\" rule (including the 3-digit and 4-digit tails), currency symbols, whitespace, `-` and `(\u2026)` negatives, negative zero, and rejection of malformed input.\n- **`format_money`:** default output, zero-padded cents, negatives, group boundaries, custom symbol/decimal/thousands passed positionally and by keyword, empty and multi-character symbols.\n- **Round trips:** `parse_money(format_money(x)) == x` over an exhaustive range and with custom separators.\n\n**Mutation sanity check:** I wrote 15 mutants of `money.py` by hand, covering the split remainder, sign, `n` bound, parse padding, decimal width, parentheses, symbols, empty input, format padding, separator defaults, zero sign and `abs`. The suite killed 14. The survivor only changes the undocumented double negative `(-4.00)`, so I didn't add a test for it.",
  "ttft_ms": 1950,
  "type": "result",
  "duration_ms": 74260,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1440,
  "time_to_request_ms": 234,
  "first_content_frame_ms": 1441,
  "queued_turn_count": 0,
  "result_index": 0
 }
]