Make a stdlib Python service pass its healthcheck (4 layered config bugs)
Exact prompt
# Task: get the service healthy This directory has a small Python service (`svc/app.py`), its config, data, `.env`, `run.sh` (starts it) and `healthcheck.sh`. Right now `./run.sh && ./healthcheck.sh` fails. Make it pass. Rules: do NOT edit `healthcheck.sh` or the `/health` handler in `app.py`; don't hard-code the health response; the service must still read config from `config.toml`, the token from `.env`, and items from `data/items.json`. Fix root causes, not symptoms. When done, stop the service (`kill $(cat .pid)`) and write `FIXES.md` listing each problem you found (one line each). Python stdlib only, no network.
Bugs surface one at a time (env typo, TOML port string, relative data path, JSON trailing comma). Scorer verifies the health handler/healthcheck are untouched and that health reflects live data, so hard-coding fails. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. healthcheck.sh must be byte-identical to the original. 2. The /health handler block in app.py must be unchanged (text compare). 3. After ./run.sh, ./healthcheck.sh must exit 0 (service on 127.0.0.1:18080 returns status ok with the original item count). 4. Anti-hardcoding: the scorer appends a 4th item to data/items.json, restarts the service, and /health must report 4 items (data then restored). 5. FIXES.md must list at least 4 problems (lines starting with -, * or a digit).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0061, 32s).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.05 | 48s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0085 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0061 | 32s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.14 | 31s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking off | $0.05 | 14s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.