Make a unified-diff patch applier correct (offset search, no-newline markers, create/delete, atomic failure)
Exact prompt
# Task: make the patch applier correct `patcher.py` provides `apply_patch(diff_text, root)` which applies a unified diff to the files under directory `root`. It works for the simplest diffs but breaks on real ones and can leave a half-patched tree behind. Fix it (rewrite freely) so it follows `PATCH_RULES.md` exactly. Keep the names `apply_patch` and `PatchError`. Python standard library only; do not shell out to `patch` or `git`. No network. The grader runs a hidden unittest suite that builds small directory trees in temporary folders, calls `apply_patch`, and checks the return value, the exact bytes of every file afterwards, and that `PatchError` is raised (with the tree unchanged) where the rules say so.
The starting applier strips paths wrongly, requires explicit counts, ignores offsets and newline markers, rewrites files hunk by hunk (half-patched trees on failure) and accepts malformed hunks. Tests check exact bytes and an unchanged tree after every failure. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 13 hidden unittest cases pass (git headers and tab timestamps, omitted counts, offset search with carry, +1 before -1, +/-50 limit, no match before previous hunk, no fuzz, no-newline markers, CRLF content and blank context lines, create/delete, failing create/delete/modify, atomic multi-file failure, malformed diffs and unsafe paths).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.03, 3m).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.08 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.03 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Opus 5.5default · thinking on | $0.42 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Sonnet 5.5default · thinking on | $0.14 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.