Implement npm-style semver range matching (caret/tilde/x/hyphen, prerelease rule) from a written spec
Exact prompt
# Task: implement semver range matching Implement `compare`, `satisfies` and `max_satisfying` in `semrange.py` following `RANGES.md` exactly (it is a subset of npm's node-semver semantics; prerelease handling is graded). Python standard library only; no third-party packages, no network. Add tests if you want.
Desugaring partial versions with implicit -0 upper bounds, caret on 0.x/0.0.x, hyphen ranges with partial ends, and the same-tuple prerelease rule are classic places where implementations diverge from npm. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 49 hidden unittest cases pass (60 s cap): 46 satisfies() vectors, compare() precedence incl. prerelease ordering and build metadata, ValueError on invalid versions/ranges, max_satisfying() incl. skipping invalid strings and returning the original string.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.03, 2m).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.07 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.03 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.33 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.14 | 52s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.