Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Model ladder: fix 10 real merged bugs with tests hidden

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.0084
Wall time: 52s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt (this run)

You are working in a checkout of the pillarjs/path-to-regexp repository. A user filed this bug report: --- stringify() merges literal text into a parameter name when an empty text token is in between For custom `TokenData` like `new TokenData([param('foo'), text(''), text('bar')])`, `stringify()` emits `:foobar` instead of `:"foo"bar`. Parsing/compiling the serialized path then has a parameter named `foobar` and no `foo`, so the round trip breaks (compile with `{ foo: 'x' }` fails). The same happens for wildcards (`*foo` followed by an empty text token and then identifier text). Without the empty text token, stringify quotes the name correctly. --- Fix the bug in the library code. Dependencies are already installed (node_modules). Keep the change minimal and in the style of the codebase; you may add or update tests. Make sure the existing test suite still passes. You have no internet access.

Five real bug fixes merged into popular open-source libraries after the models' training cutoff (humanize, marshmallow, tenacity, nanoid, path-to-regexp). Each agent starts from the commit before the fix with only a symptom-style bug report, and the fix PR's own regression tests are applied afterwards to check the result. The size of the agent's source change is compared with the human fix. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/pillarjs/path-to-regexp/pull/456, https://github.com/pillarjs/path-to-regexp/commit/7dd6f8f7bd63ae8bffca0968144508228dc61729.

Success criteria: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.

Setup notes: Real merged bug fix: pillarjs/path-to-regexp https://github.com/pillarjs/path-to-regexp/pull/456 (merged after the models' Jun 2026 cutoff). Agent got a history-free snapshot of the parent commit 8877f41873 with deps installed, and a symptom-only bug report written by the runner from the PR description (no fix hints). The PR's test files were overlaid only after the agent finished. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Diff-size flag (not scored): agent changed 10 source lines vs 27 in the human fix (allowance 54; within allowance). Claude Code's own reported cost for claude-haiku-5.5: $0.008395. Score: 2/2. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.0084
Wall time: 52s
Human step-ins: 0
Turns: 13

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	257,168	4,373	$0.0084

Reproduce this run

Use the exact prompt shown above for this run (the use case page has the shared prompt).
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 24595,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.008395250000000002,
  "usage": {
   "input_tokens": 20,
   "cache_creation_input_tokens": 19133,
   "cache_read_input_tokens": 238015,
   "output_tokens": 4373,
   "output_tokens_details": {
    "thinking_tokens": 853
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 19133,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 409,
     "cache_read_input_tokens": 30928,
     "cache_creation_input_tokens": 744,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 744
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 20,
    "outputTokens": 4373,
    "cacheReadInputTokens": 238015,
    "cacheCreationInputTokens": 19133,
    "webSearchRequests": 0,
    "costUSD": 0.008395250000000002,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 853,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 13,
  "subtype": "success",
  "api_error_status": null,
  "result": "`stringify()` now quotes a parameter or wildcard name when the next text is only reached after one or more empty text tokens. `npm test` passes: 486 tests, typecheck, and the size check.\n\n**The fix** is in `src/index.ts`. `stringifyName` only looked at the token immediately after a param or wildcard. An empty text token has no first character, so the name was left unquoted and `:foo` ran into `bar`. It now skips empty text tokens before checking whether the next character could continue an identifier. I also changed its signature to take `tokens` and `index` so it can look ahead.\n\n**Tests:** I added two cases to `src/cases.spec.ts`, one each for the param and wildcard forms in the report. They expect `:\"foo\"bar` and `*\"foo\"bar`.\n\nRunning `npx vitest run` directly shows a \"No test suite found\" failure for `src/cases.spec.ts`. It fails the same way on the untouched code, and `npm test` runs cleanly, so I left it alone. I haven't committed anything.",
  "ttft_ms": 1830,
  "type": "result",
  "duration_ms": 51202,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1324,
  "time_to_request_ms": 255,
  "first_content_frame_ms": 1324,
  "queued_turn_count": 0,
  "result_index": 0
 }
]