Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Model ladder: fix 10 real merged bugs with tests hidden

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.01
Wall time
45s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt (this run)

You are working in a checkout of the jd/tenacity repository. A user filed this bug report: --- Custom wait strategy is silently skipped when the strategy object is falsy I pass a callable wait strategy object to `Retrying(wait=...)`. The object happens to evaluate as false (in my case it subclasses `list` and is empty; an object with a `__bool__` returning False behaves the same). Tenacity never calls it and retries with no delay at all. My strategy returns 1 second, so I expect `sleep(1)` between attempts. Only 'no wait' values like `None` or `0` should disable waiting; any real strategy should be called. Please make sure the async retrying path behaves the same. --- Fix the bug in the library code. Dependencies are already installed (Python virtualenv in .venv; run tests with .venv/bin/pytest). Keep the change minimal and in the style of the codebase; you may add or update tests. Make sure the existing test suite still passes. You have no internet access.

Five real bug fixes merged into popular open-source libraries after the models' training cutoff (humanize, marshmallow, tenacity, nanoid, path-to-regexp). Each agent starts from the commit before the fix with only a symptom-style bug report, and the fix PR's own regression tests are applied afterwards to check the result. The size of the agent's source change is compared with the human fix. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/jd/tenacity/pull/741, https://github.com/jd/tenacity/commit/b6201eceedfc719e996dcbc3f6c17f21011fb236.

Success criteria: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.

Setup notes: Real merged bug fix: jd/tenacity https://github.com/jd/tenacity/pull/741 (merged after the models' Jun 2026 cutoff). Agent got a history-free snapshot of the parent commit 4a74682e01 with deps installed, and a symptom-only bug report written by the runner from the PR description (no fix hints). The PR's test files were overlaid only after the agent finished. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Diff-size flag (not scored): agent changed 12 source lines vs 31 in the human fix (allowance 62; within allowance). Claude Code's own reported cost for claude-haiku-5.5: $0.010122. Score: 2/2. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.01
Wall time
45s
Human step-ins
0
Turns
15
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on322,4896,140$0.01

Reproduce this run

  1. Use the exact prompt shown above for this run (the use case page has the shared prompt).
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 33382,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.010121559999999998,
  "usage": {
   "input_tokens": 24,
   "cache_creation_input_tokens": 20129,
   "cache_read_input_tokens": 302336,
   "output_tokens": 6140,
   "output_tokens_details": {
    "thinking_tokens": 1258
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 20129,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 570,
     "cache_read_input_tokens": 31872,
     "cache_creation_input_tokens": 796,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 796
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 24,
    "outputTokens": 6140,
    "cacheReadInputTokens": 302336,
    "cacheCreationInputTokens": 20129,
    "webSearchRequests": 0,
    "costUSD": 0.010121559999999998,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 1258,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 15,
  "subtype": "success",
  "api_error_status": null,
  "result": "Fixed. A custom wait strategy that evaluates as false is now called on both the sync and async paths, and its return value is slept on. The full suite passes: 185 passed, 1 skipped.\n\n**Root cause:** `_run_wait` in `tenacity/__init__.py` and `AsyncRetrying._run_wait` in `tenacity/asyncio/__init__.py` used `if not self.wait:` to mean \"no wait\". Any falsy strategy, such as an empty list subclass or an object whose `__bool__` returns False, was therefore skipped.\n\n**Change:** Both checks now only treat `None` and numeric zero as \"no wait\":\n\n```python\nif self.wait is None or (isinstance(self.wait, (int, float)) and not self.wait):\n    retry_state.upcoming_sleep = 0.0\n```\n\nNumeric zero still covers the `sum([])` case from the existing comment.\n\n**Tests added:**\n- `tests/test_tenacity.py::test_falsy_wait_strategy_is_called` checks an empty list subclass and an object with `__bool__` returning False, asserting `sleep(1.0)` is called.\n- `tests/test_asyncio.py::test_falsy_wait_strategy_is_called` checks the same for `AsyncRetrying`.\n\nWithout the library fix, the new tests fail for both sync cases and the async case. With the fix, they pass.\n\nI did not run ruff or mypy: the virtualenv has neither installed, so the style and type checks are unverified. The new lines are within the 88-character limit.",
  "ttft_ms": 1938,
  "type": "result",
  "duration_ms": 44247,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1400,
  "time_to_request_ms": 260,
  "first_content_frame_ms": 1401,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly