Fix a spreadsheet formula engine (precedence, error values, static cycle detection, 5,000-cell chains without recursion errors)
Exact prompt
# Task: fix the spreadsheet formula engine `sheet.py` exposes `evaluate(grid)`: it takes a sheet as a list of rows of cell strings (as read from a CSV file) and returns the computed values in the same shape. The current version gets precedence wrong, crashes on reference cycles and long dependency chains, and does not produce the error values users expect. Make it implement `SHEET_RULES.md` exactly (rewrite freely). Keep the function name and signature. Python standard library only. Do not use `eval`/`exec` on cell text. No network. `python3 sheet.py in.csv out.csv` should keep working as a command line wrapper (read CSV, evaluate, write CSV). The grader runs a hidden unittest suite that calls `evaluate` on many small grids and a few large ones and compares every output cell string exactly.
The starting engine rewrites cells into Python eval (wrong precedence for unary minus, no error values, RecursionError on cycles and long chains). Correctness needs a real parser, error propagation order, aggregate ignore-rules, a static cycle graph and iterative evaluation. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 12 hidden unittest cases pass (cell kinds and output shape, references incl. AA/$, precedence and comparisons, values in arithmetic, aggregates, IF/ABS, error values and propagation order, cycles, number formatting, 5,000-cell chains both directions, 10,000-cell grid under 5 s, CLI wrapper).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.08, 7m).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.09 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.08 | 7m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Opus 5.5default · thinking on | $0.61 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Sonnet 5.5default · thinking on | $0.19 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.