Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Write a lexer, parser and evaluator for a small closure-based language from a 1-page spec

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-17 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.04
Wall time
3m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Task: write an interpreter for the Tiny language Implement `run(source)` and `TinyError` in `tiny.py` exactly as specified in `SPEC.md` (lexer, parser, evaluator). Python standard library only. Programs in the grader are small but exercise every rule, including error messages and their line numbers. Deep recursion up to the specified limit must work without crashing Python. No network.

Precedence incl. prefix not below comparisons, non-associative comparisons, truncating division with dividend-signed modulo, short-circuit operand values, block scoping vs. assignment, closures by reference, recursion depth limit without crashing Python, and exact error kinds with line numbers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. All 29 hidden unittest cases pass (120 s cap): 12 programs with exact printed output and 17 error programs raising TinyError with the right message and 'line N: ' prefix.

Setup notes: Use Case Scout batch 6 case f08-tiny-interpreter (offline fixture). Prepared with batch6/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.038081. Score: 29/29. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.04
Wall time
3m
Human step-ins
0
Turns
9
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on440,17946,728$0.04

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. All 29 hidden unittest cases pass (120 s cap): 12 programs with exact printed output and 17 error programs raising TinyError with the right message and 'line N: ' prefix.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 193073,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.03808136999999999,
  "usage": {
   "input_tokens": 18,
   "cache_creation_input_tokens": 54284,
   "cache_read_input_tokens": 385877,
   "output_tokens": 46728,
   "output_tokens_details": {
    "thinking_tokens": 34339
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 54284,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 1016,
     "cache_read_input_tokens": 65899,
     "cache_creation_input_tokens": 924,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 924
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 18,
    "outputTokens": 46728,
    "cacheReadInputTokens": 385877,
    "cacheCreationInputTokens": 54284,
    "webSearchRequests": 0,
    "costUSD": 0.03808136999999999,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 34339,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 9,
  "subtype": "success",
  "api_error_status": null,
  "result": "`tiny.py` now implements `run(source)` and `TinyError` per `SPEC.md`, using only the standard library. I checked it with a scratch test script outside the work directory. It covers arithmetic, scoping, closures, recursion, and the error messages and line numbers. The only three failures were wrong expectations in my script, and the interpreter's behaviour is correct for them.\n\n- **Call depth:** exactly 200 nested calls succeed. The 201st fails with `line N: stack overflow` at its own call line. Infinite recursion gives the same error without crashing Python.\n- **Deep nesting:** 5000-deep parenthesised expressions parse and run, and pathological nesting that exhausts the recursion limit is reported as a Tiny syntax or stack error rather than a Python traceback.\n\nThe spec left some cases open, so I chose these:\n\n- **Newlines in brackets:** a newline is ignored only when the innermost open bracket is `(`. Inside a `{ }` body nested in a call, newlines still end statements.\n- **Function scope:** parameters and the function body share one scope, so `fn f(a) { let a = 1 }` gives `already declared`.\n- **Duplicate parameters:** `fn f(a, a)` is reported as `already declared` when the function is called, not when it is defined.\n- **`return outside function`:** raised when the statement executes, not at parse time.\n- **Check order:** on a call, the type check runs first, then arity, then stack depth.\n- **`let` and assignment:** the right-hand side is evaluated before the already-declared or undefined-variable check.\n- **Lexer errors:** unknown escapes, unterminated strings, and stray characters like `!` are `line N: syntax error`.\n- **Global side effects:** `run` temporarily raises the Python recursion limit to 50000 and restores it afterwards. Importing the module also lifts Python's 4300-digit limit on int\u2194str conversion, which is process-wide, so that every Tiny integer prints in full.\n\nNothing is committed; the directory is not a git repository.",
  "ttft_ms": 1819,
  "type": "result",
  "duration_ms": 193685,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1638,
  "time_to_request_ms": 243,
  "first_content_frame_ms": 1639,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly