Reach a goal stack configuration using a simulator CLI with minimal moves
Exact prompt
# Task: plan with a tool `wh.py` is a warehouse stacking simulator (read its docstring: `python wh.py status`). Reach the goal configuration in `goal.json` using ONLY `python wh.py move SRC DST` calls. Every call (including `status` and rejected moves) is logged in `calls.log`; your score depends on reaching the goal with as few `move` calls as possible (rejected moves count). Do NOT edit `wh.py`, `state.json`, `goal.json`, or `calls.log` by hand — the grader replays `calls.log` from the initial state. Plan before moving. No network.
Grader replays calls.log; optimal is 13 moves (BFS), pass ≤15 with zero rejected moves. Rewards planning before acting over trial-and-error. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. wh.py and goal.json must be unmodified. 2. Replaying calls.log from the original initial state must reach the goal configuration. 3. Zero rejected (illegal) moves in the log. 4. Number of move calls <= optimal (computed by BFS in the scorer) + 2.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0046, 21s).
All runs (5)
80% pass rate across 5 eligible runs: 4 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.04 | 30s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0076 | 54s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Haiku 5.5default · thinking on | $0.0046 | 21s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.11 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 36s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.