Bisect a 42-commit local git history to the first bad commit, then fix forward with a regression test
Exact prompt
# Task: find and fix a regression in git history `repo/` is a git repository (branch `main`). Since some point in its history, `parse_duration("1h30m")` returns 3600 instead of 5400 - multi-part durations only count their first part. Nobody noticed because the tests only cover single units. 1. Find the **first** commit that introduced this regression. Write its full 40-character SHA to `FIRST_BAD.txt` (in this folder, next to `repo/`, not inside it), optionally followed by a one-line explanation. 2. Fix the bug with **new commit(s) on top of `main`** - do not rewrite, rebase, amend or reset existing history. Apart from the regression itself, `parse_duration` must behave exactly as at the current tip of `main` (accepted formats, rejected inputs, exact error messages, and the `timeutil.legacy` import path). The fix commit message must contain `Fixes <sha>` with at least the first 7 characters of the first bad commit. 3. Add a regression test to `repo/tests/` (plain `test_*` functions, runnable without pytest) that fails on the buggy code and passes after your fix. Keep the existing test. The grader runs every `test_*` function in `repo/tests/*.py`. When done, `main` must be checked out (finish any `git bisect` with `git bisect reset`) and the working tree clean (everything committed; untracked `__pycache__`/`*.pyc` files are ignored by the grader). Git is configured locally in the repo. No network.
Requires identifying the exact introducing commit (a 'perf' refactor, not the later commit that also touched the file), fixing without rewriting history, keeping later intentional changes (day unit, new error message, legacy import) and writing a test that provably fails on the bad commit. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. first_bad_correct: FIRST_BAD.txt contains the full SHA of the commit that introduced the regression. 2. history_preserved: the original tip is an ancestor of main and main is checked out. 3. fix_committed_with_reference: at least one new commit, and a new commit message contains 'Fixes <first 7+ chars of the bad SHA>'. 4. working_tree_clean: git status --porcelain shows nothing except untracked __pycache__/*.pyc files. 5. behaviour_correct: hidden checks at HEAD pass (multi-part and day-unit durations, legacy import path, exact 'empty duration string' error, invalid inputs rejected). 6. regression_test_catches_bug: the repo's tests/ (>= 1 test function) all pass at HEAD, and at least one fails when run against the first bad commit's code.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 44s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.06 | 53s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 44s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.17 | 29s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.06 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.