Write a lexer, parser and evaluator for a small closure-based language from a 1-page spec
Exact prompt
# Task: write an interpreter for the Tiny language Implement `run(source)` and `TinyError` in `tiny.py` exactly as specified in `SPEC.md` (lexer, parser, evaluator). Python standard library only. Programs in the grader are small but exercise every rule, including error messages and their line numbers. Deep recursion up to the specified limit must work without crashing Python. No network.
Precedence incl. prefix not below comparisons, non-associative comparisons, truncating division with dividend-signed modulo, short-circuit operand values, block scoping vs. assignment, closures by reference, recursion depth limit without crashing Python, and exact error kinds with line numbers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 29 hidden unittest cases pass (120 s cap): 12 programs with exact printed output and 17 error programs raising TinyError with the right message and 'line N: ' prefix.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.04, 3m).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.09 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.04 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.52 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.16 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.