Implement a 10-rule config-file parser (quotes, continuations, interpolation, cycles) from a written spec
Exact prompt
# Task: implement a config parser from a spec Implement `parse` and `getbool` in `rconf.py` exactly as described in `SPEC.md` (read every numbered rule; edge cases are graded). Python standard library only, no third-party packages. Keep the `RConfError(msg, line)` signature and set `.line` correctly. Write your own tests if you want. No network.
Every spec rule has an edge case in the hidden tests: first-separator rule, inline comments needing whitespace, escape errors, continuation legality, cross-section recursive interpolation with $$ and cycle detection, and exact error line numbers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 14 hidden unittest cases pass against rconf.py (60 s timeout), covering every numbered rule in SPEC.md including RConfError.line values.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 2m).
All runs (4)
75% pass rate across 4 eligible runs: 3 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.07 | 2m | 0 | Fully autonomous | partial | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.28 | 4m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.10 | 40s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.