Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Stress-test a small multiplayer game server, then write a minimal crash reproducer and a fix that holds up under hidden stress tests
AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-25 · by @bobsiopener
- Estimated cost
- $0.47
- Wall time
- 4m
- Human step-ins
- 0
- Outcome
- Partial
Compare with other models on this task →
Summary
Exact prompt
# Stress-test a small multiplayer game server, find the bug that crashes it, write a minimal reproducer and fix it `game/` is a small synthetic multiplayer city-builder server that runs in-process, with no network and no ports; `SPEC.md` says how it must behave. Players report that the server sometimes crashes during busy sessions. Everything is synthetic and offline; don't open any network port. Rules: - You may edit files under `game/` (that is the fix) as well as writing under `out/`; this overrides the general rule below. - Find the crash: a sequence of client messages (several clients may send in the same tick) that makes `Server.process` raise. There is one such bug. - Reproducer: write `out/repro.py`, at most 60 lines, that drives the server through `game.server` or `game.sim` (it is run from the folder root with the folder on PYTHONPATH, as `python3 out/repro.py`). On the code as shipped, it must crash with the server's own uncaught exception, raised inside `game/`. Once the bug is fixed, it must run cleanly and exit 0. - Fix: edit the code under `game/` so that the server never raises and follows SPEC.md, including on your reproducer's input. A rejected request must change nothing, so swallowing exceptions is not a fix if it leaves half-done changes. Keep the public API unchanged. - No regression: behaviour on normal sessions must stay exactly as it was (same replies and the same `world.snapshot()` after every tick). - Write `out/bug.json`: `{"exception": "<exception type raised>", "crash_module": "<path of the game/ file where it is raised, e.g. game/x.py>", "trigger": "<one sentence>"}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
A small synthetic in-process multiplayer city-builder server (no network) has one seeded crash bug; the bug type and the module it lives in vary by seed. The agent must find a message sequence that crashes it, write a reproducer of at most 60 lines, and fix the code. Scoring runs the reproducer against the shipped and fixed code, replays hidden stress and fuzz sessions against the spec, and replays normal sessions to catch regressions. Swallowing exceptions is not a fix, because a rejected request must change nothing. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: repro_crashes_original repro_clean_once_fixed fix_survives_stress no_regression bug_report
Setup notes: Batch 37 case ty01-game-server-crash-repro-fix (offline, synthetic inputs, seed 2053929010). Autonomous case: no stand-in user and no step-ins expected; any file the solver wrote to ask/ is counted as a measured intervention (0) and blocks a pass. Prompt = TASK.md verbatim. Hidden key kept outside the solver's work dir. Scored with the case's scorer (python3 score.py WORKDIR KEYDIR, absolute paths); pass = every check true and no ask/ files. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.61 (8 cores). Time-zone results depend on the system time-zone database: tzdata 2026c-0+deb13u1 (Python zoneinfo). Scored only inside a bwrap sandbox (bubblewrap 0.12.0): no network, read-only system, case dir and key read-only, only a throwaway copy of the workdir writable. Claude Code's own reported cost for claude-opus-5.5: $0.47412. Score: 4/5. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.
Estimated cost List-price estimate; not actually billed.
Metrics
- Estimated cost
- $0.47
- Wall time
- 4m
- Human step-ins
- 0
- Turns
- 17
| Model | Input tok | Output tok | Cost |
|---|---|---|---|
| AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on | 496,080 | 9,619 | $0.47 |
Reproduce this run
- Use the use case prompt on the use case page.
- Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via
Claude Code 2.1.299 (headless -p), cloud. - It counts as a success only if it meets every success criterion: repro_crashes_original repro_clean_once_fixed fix_survives_stress no_regression bug_report
- Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.
How runs are scored: Methodology.
Was this useful?
Automated checks
Trust 80% comes from these checks (how Trust % works).
- Run date is not in the future : passed
- Model is in the dated pricing table : passed
- Cost recomputed from tokens matches : passed
- Cached tokens are consistent : passed
- Output throughput is plausible : passed
- Autonomy matches step-ins : passed
- Step-ins do not exceed turns : passed
- Retries agree with first-try result : passed
- Evidence log SHA-256 matches : passed
- Evidence log is substantial : passed
Evidence log
[
{
"duration_api_ms": 115479,
"stop_reason": "end_turn",
"session_id": "[REDACTED]",
"total_cost_usd": 0.47412040000000005,
"usage": {
"input_tokens": 34,
"cache_creation_input_tokens": 23384,
"cache_read_input_tokens": 472662,
"output_tokens": 9619,
"output_tokens_details": {
"thinking_tokens": 3211
},
"server_tool_use": {
"web_search_requests": 0,
"web_fetch_requests": 0
},
"service_tier": "standard",
"cache_creation": {
"ephemeral_1h_input_tokens": 23384,
"ephemeral_5m_input_tokens": 0
},
"inference_geo": "global",
"iterations": [
{
"input_tokens": 2,
"output_tokens": 943,
"cache_read_input_tokens": 35079,
"cache_creation_input_tokens": 451,
"cache_creation": {
"ephemeral_5m_input_tokens": 0,
"ephemeral_1h_input_tokens": 451
},
"type": "message"
}
],
"speed": "standard",
"fallback_credit": null
},
"modelUsage": {
"claude-opus-5-5": {
"inputTokens": 34,
"outputTokens": 9619,
"cacheReadInputTokens": 472662,
"cacheCreationInputTokens": 23384,
"webSearchRequests": 0,
"costUSD": 0.47412040000000005,
"contextWindow": 1000000,
"maxOutputTokens": 128000,
"thinkingTokens": 3211,
"canonicalModel": "claude-opus-5-5",
"provider": "firstParty",
"costBasis": "list"
}
},
"permission_denials": [],
"terminal_reason": "completed",
"fast_mode_state": "off",
"fast_mode_disabled_reason": "sdk_opt_in_required",
"subagent_stats": {
"spawned": 0,
"requested": {
"background": 0,
"foreground": 0,
"unset": 0
},
"started_in_background": 0,
"max_depth": 0,
"spawned_by_subagents": 0,
"completed": 0,
"failed": 0,
"killed": {
"parent": 0,
"user": 0,
"system": 0
},
"refused": {
"depth_limit": 0,
"concurrency_limit": 0,
"budget": 0
},
"by_type": {}
},
"safety_stops": 0,
"is_error": false,
"num_turns": 17,
"subtype": "success",
"api_error_status": null,
"result": "I found the crash and fixed it with a one-line change, and wrote the reproducer and `out/bug.json`.\n\n**The bug.** `accept` in `game/handlers/market.py` looks offers up in `world.tick_offers`, a copy of the offers taken at the start of each tick. An offer accepted or withdrawn earlier in the same tick still looks open there. So when two clients accept the same offer in one tick:\n- The second accept passes every check and swaps goods a second time.\n- Then `del world.offers[oid]` raises `KeyError`, and the half-done swap is left in place.\n\nThe same stale copy also crashes the server when an offer's maker leaves earlier in the tick (`world.players[maker]` \u2192 `KeyError`).\n\n**The fix** (`game/handlers/market.py`, one line). `accept` now reads the live offer table: `o = world.offers.get(oid) if oid in world.tick_offers else None`. An offer that has already been accepted or withdrawn, or whose maker has left, is rejected with `no_such_offer` before anything changes, as SPEC requires.\n\nOne judgement call: I kept the rule that an offer must have existed at the start of the tick. The shipped code already rejects accepting an offer made earlier in the same tick, and keeping that keeps normal sessions exactly as they were. SPEC's \"still open at that moment in the tick\" could be read as allowing those accepts. If so, the fix would be to drop the `in world.tick_offers` condition, but that would change replies the shipped code gives today.\n\n**Outputs:**\n- `out/repro.py` (19 lines): three players join, `ana` makes an offer, then `bob` and `cat` both accept it in the same tick. On the shipped code it dies with `KeyError: 1` at `game/handlers/market.py:48`. With the fix it prints `ok` and exits 0, after checking that `cat`'s accept is rejected and `cat`'s goods are unchanged.\n- `out/bug.json`: `KeyError`, raised in `game/handlers/market.py`, with a one-sentence trigger.\n\n**Testing.** I compared the shipped and fixed code on 3,000 random sessions of 30 ticks each, with several clients per tick:\n- The shipped code crashed in 398 sessions; the fixed code never raised.\n- No rejected message ever changed the world snapshot.\n- Replies and snapshots matched on every session up to its first double-accept. The only differences were accepts of an offer closed earlier in the same tick: the shipped code crashed or returned the stale `maker_cannot_afford`, and the fix returns `no_such_offer`.\n\nMy first few fuzz runs reported zero crashes because the shipped copy wasn't actually being loaded. The figures above are from the corrected run.",
"ttft_ms": 2689,
"type": "result",
"duration_ms": 249071,
"uuid": "[REDACTED]",
"ttft_stream_ms": 1900,
"time_to_request_ms": 165,
"first_content_frame_ms": 1900,
"queued_turn_count": 0,
"result_index": 0
}
]