Skip to content

Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Write a lexer, parser and evaluator for a small closure-based language from a 1-page spec

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-17 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.16
Wall time
1m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Task: write an interpreter for the Tiny language Implement `run(source)` and `TinyError` in `tiny.py` exactly as specified in `SPEC.md` (lexer, parser, evaluator). Python standard library only. Programs in the grader are small but exercise every rule, including error messages and their line numbers. Deep recursion up to the specified limit must work without crashing Python. No network.

Precedence incl. prefix not below comparisons, non-associative comparisons, truncating division with dividend-signed modulo, short-circuit operand values, block scoping vs. assignment, closures by reference, recursion depth limit without crashing Python, and exact error kinds with line numbers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. All 29 hidden unittest cases pass (120 s cap): 12 programs with exact printed output and 17 error programs raising TinyError with the right message and 'line N: ' prefix.

Setup notes: Use Case Scout batch 6 case f08-tiny-interpreter (offline fixture). Prepared with batch6/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-sonnet-5.5: $0.163019. Score: 29/29. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.16
Wall time
1m
Human step-ins
0
Turns
6
Model usage
ModelInput tokOutput tokCost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on113,5609,025$0.16

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. All 29 hidden unittest cases pass (120 s cap): 12 programs with exact printed output and 17 error programs raising TinyError with the right message and 'line N: ' prefix.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 61986,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.1630188,
  "usage": {
   "input_tokens": 10,
   "cache_creation_input_tokens": 15742,
   "cache_read_input_tokens": 97808,
   "output_tokens": 9025,
   "output_tokens_details": {
    "thinking_tokens": 2052
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 15742,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 596,
     "cache_read_input_tokens": 27821,
     "cache_creation_input_tokens": 460,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 460
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 10,
    "outputTokens": 9025,
    "cacheReadInputTokens": 97808,
    "cacheCreationInputTokens": 15742,
    "webSearchRequests": 0,
    "costUSD": 0.1630188,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 2052,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 6,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote the interpreter in `tiny.py` (`run` and `TinyError`, standard library only). I ran a handful of programs by hand. I did not write a test suite or run anything beyond those.\n\nWhat those checks showed:\n- **Error messages and line numbers:** these matched the spec for redeclaration, a chained comparison, an undefined variable, division by zero and `return` outside a function.\n- **Call-depth limit:** a recursive function nested 200 calls deep ran, and the call that would be the 201st failed with `stack overflow` at that call's line. It runs the interpreter in a thread with a large stack, so Python itself doesn't crash.\n- **Arithmetic and printing:** `-7 % 3` and `-7 / 2` came out as the spec says, and `print` of a function and of a string repeat worked.\n\nThe spec doesn't settle a few things, so I picked:\n- **Function body scope:** a function's body shares the scope that holds its parameters, so `let` of a parameter name inside the body is an `already declared` error.\n- **`let` evaluation order:** the right-hand side of a `let` is evaluated before the redeclaration check.\n- **Call error line:** for a call, errors are reported at the line of the opening `(`.\n- **`else` on its own line:** `else` must follow the `}` on the same line, so an `else` that starts its own line is a syntax error.\n- **Leniency:** block statements (`if`, `while`, `fn`) don't need a terminator after their closing `}`. A second statement on the same line right after a `}` is accepted rather than rejected.",
  "ttft_ms": 4402,
  "type": "result",
  "duration_ms": 62421,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 4042,
  "time_to_request_ms": 245,
  "first_content_frame_ms": 4042,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly