Skip to content

Model ladder: fix 10 real merged bugs with tests hidden

Exact prompt

You are working in a checkout of the python-humanize/humanize repository. A user filed this bug report: --- intcomma() raises OverflowError for very large integers `humanize.intcomma(10**400 + 123)` raises `OverflowError: int too large to convert to float` instead of returning the number with thousands separators. Negative huge integers (`-(10**400 + 123)`) fail the same way. Python ints are arbitrary precision, so I expect the full digits, correctly grouped (ending in `,123`), just like for small integers. --- Fix the bug in the library code. Dependencies are already installed (Python virtualenv in .venv; run tests with .venv/bin/pytest). Keep the change minimal and in the style of the codebase; you may add or update tests. Make sure the existing test suite still passes. You have no internet access.

Five real bug fixes merged into popular open-source libraries after the models' training cutoff (humanize, marshmallow, tenacity, nanoid, path-to-regexp). Each agent starts from the commit before the fix with only a symptom-style bug report, and the fix PR's own regression tests are applied afterwards to check the result. The size of the agent's source change is compared with the human fix. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/python-humanize/humanize/pull/392, https://github.com/python-humanize/humanize/commit/984526c02448110a8392dde3f5e26096dfd7c729.

Success criteria: Hidden PR tests pass; existing suite stays green; diff under 2x the human diff size.

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0058, 19s). Best AI model for Model ladder: fix 10 real merged bugs with tests hidden →

Run this on another model

All runs (15)

100% pass rate across 15 eligible runs: 15 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.2050s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5default · thinking on$0.0749s0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.008452s0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5default · thinking on$0.354m0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.008937s0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5default · thinking on$0.2745s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5default · thinking on$0.073m0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.0145s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5default · thinking on$0.0515s0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5default · thinking on$0.1633s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5default · thinking off$0.0410s0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.005819s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5default · thinking on$0.0936s0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5default · thinking on$0.2547s0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.009036s0Fully autonomouspassTrust 80%2026-10-16

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Model ladder: fix 10 real merged bugs with tests hidden

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.