Stress-test a small multiplayer game server, then write a minimal crash reproducer and a fix that holds up under hidden stress tests
Exact prompt
# Stress-test a small multiplayer game server, find the bug that crashes it, write a minimal reproducer and fix it `game/` is a small synthetic multiplayer city-builder server that runs in-process, with no network and no ports; `SPEC.md` says how it must behave. Players report that the server sometimes crashes during busy sessions. Everything is synthetic and offline; don't open any network port. Rules: - You may edit files under `game/` (that is the fix) as well as writing under `out/`; this overrides the general rule below. - Find the crash: a sequence of client messages (several clients may send in the same tick) that makes `Server.process` raise. There is one such bug. - Reproducer: write `out/repro.py`, at most 60 lines, that drives the server through `game.server` or `game.sim` (it is run from the folder root with the folder on PYTHONPATH, as `python3 out/repro.py`). On the code as shipped, it must crash with the server's own uncaught exception, raised inside `game/`. Once the bug is fixed, it must run cleanly and exit 0. - Fix: edit the code under `game/` so that the server never raises and follows SPEC.md, including on your reproducer's input. A rejected request must change nothing, so swallowing exceptions is not a fix if it leaves half-done changes. Keep the public API unchanged. - No regression: behaviour on normal sessions must stay exactly as it was (same replies and the same `world.snapshot()` after every tick). - Write `out/bug.json`: `{"exception": "<exception type raised>", "crash_module": "<path of the game/ file where it is raised, e.g. game/x.py>", "trigger": "<one sentence>"}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
A small synthetic in-process multiplayer city-builder server (no network) has one seeded crash bug; the bug type and the module it lives in vary by seed. The agent must find a message sequence that crashes it, write a reproducer of at most 60 lines, and fix the code. Scoring runs the reproducer against the shipped and fixed code, replays hidden stress and fuzz sessions against the spec, and replays normal sessions to catch regressions. Swallowing exceptions is not a fix, because a rejected request must change nothing. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: repro_crashes_original repro_clean_once_fixed fix_survives_stress no_regression bug_report
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 1m).
All runs (3)
67% pass rate across 3 eligible runs: 2 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.47 | 4m | 0 | Fully autonomous | partial | Trust 80% | 2026-10-25 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 | |
| AClaude Sonnet 5.5default · thinking on | $0.08 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.