Skip to content

Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Model ladder: fix 10 real merged bugs with tests hidden

AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.35
Wall time
4m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt (this run)

You are working in a checkout of the ai/nanoid repository. A user filed this bug report: --- customAlphabet()/customRandom() return a constant string for a non-integer size Since 5.1.8, calling a generator with a fractional size gives broken ids on power-of-two alphabets, e.g. `customAlphabet('0123456789abcdef')(16.5)` returns `"00000000000000000"` (the first alphabet character repeated) in the browser build, and the node build is also wrong. I'd expect a fractional size to behave like its integer part (16 random characters here), in both the node and browser builds, and for `customRandom()` too. --- Fix the bug in the library code. Dependencies are already installed (node_modules). Keep the change minimal and in the style of the codebase; you may add or update tests. Make sure the existing test suite still passes. You have no internet access.

Five real bug fixes merged into popular open-source libraries after the models' training cutoff (humanize, marshmallow, tenacity, nanoid, path-to-regexp). Each agent starts from the commit before the fix with only a symptom-style bug report, and the fix PR's own regression tests are applied afterwards to check the result. The size of the agent's source change is compared with the human fix. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/ai/nanoid/pull/616, https://github.com/ai/nanoid/commit/0b2cec58ffd5ddc45978bf75a3bab46f53c76128.

Success criteria: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.

Setup notes: Real merged bug fix: ai/nanoid https://github.com/ai/nanoid/pull/616 (merged after the models' Jun 2026 cutoff). Agent got a history-free snapshot of the parent commit ccb0e6ee55 with deps installed, and a symptom-only bug report written by the runner from the PR description (no fix hints). The PR's test files were overlaid only after the agent finished. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Diff-size flag (not scored): agent changed 12 source lines vs 6 in the human fix (allowance 20; within allowance). Claude Code's own reported cost for claude-opus-5.5: $0.345573. Score: 2/2. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.35
Wall time
4m
Human step-ins
0
Turns
19
Model usage
ModelInput tokOutput tokCost
AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on382,9055,514$0.35

Reproduce this run

  1. Use the exact prompt shown above for this run (the use case page has the shared prompt).
  2. Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 58225,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.34557260000000006,
  "usage": {
   "input_tokens": 28,
   "cache_creation_input_tokens": 20334,
   "cache_read_input_tokens": 362543,
   "output_tokens": 5514,
   "output_tokens_details": {
    "thinking_tokens": 1496
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 20334,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 705,
     "cache_read_input_tokens": 32106,
     "cache_creation_input_tokens": 374,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 374
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 28,
    "outputTokens": 5514,
    "cacheReadInputTokens": 362543,
    "cacheCreationInputTokens": 20334,
    "webSearchRequests": 0,
    "costUSD": 0.34557260000000006,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 1496,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 19,
  "subtype": "success",
  "api_error_status": null,
  "result": "Fixed: a fractional size now acts like its integer part in both the node and browser builds, for both `customAlphabet()` and `customRandom()`. `customAlphabet('0123456789abcdef')(16.5)` now returns 16 random characters, such as `26ec0dbdd7efae32`.\n\n**Cause:** the functions returned by `customRandom()` used `size` directly as a loop counter. With 16.5, the counter took values like 15.5, 14.5\u2026, so `bytes[15.5]` was `undefined`. `undefined & mask` is `0`, which gave the first alphabet character every time. Also, `id.length >= 16.5` only stopped at 17 characters. The node build's `customAlphabet()` fast path already did `size |= 0`. But node's `customRandom()` didn't, and the browser's `customAlphabet()` goes through `customRandom()`.\n\n**Fix:**\n- **`index.js` and `index.browser.js`:** both functions returned by `customRandom()` now start with `size |= 0`. That's the same idiom the codebase already uses in `random()` and in node's `customAlphabet()`.\n- **`index.browser.js`:** I removed the `size | 0` that `customAlphabet` applied to its default size. `customRandom()` now truncates every size, so it did nothing, and dropping it offsets the bytes I added to the browser bundle.\n\n**Tests:** I added \"truncates fractional size\" tests in `test/index.test.js` for `customAlphabet` (power-of-two and other alphabets) and for `customRandom`. Both run against the node and browser builds. They fail without the fix and pass with it, and the full suite passes (87/87 with `bnt`).\n\nI couldn't run `size-limit` or `oxlint`: this sandbox has Node 20, and both tools fail to start under it. The browser `customAlphabet` bundle (limit 219 B) changes by a few bytes, so run `pnpm test` on a supported Node version (22 or later) to confirm the size check and lint still pass.",
  "ttft_ms": 2087,
  "type": "result",
  "duration_ms": 220848,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1382,
  "time_to_request_ms": 229,
  "first_content_frame_ms": 1382,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly