Ship settings format v3 that deployed v2 readers can still read (forward/backward compatibility, atomic save, idempotent migration)
Exact prompt
# Task: settings format v3 without breaking deployed readers Upgrade `settings.py` to the v3 format in `FORMAT_V3.md`: `load(path)` reads v1, v2, v3 and newer-compatible files; `save(path, settings)` writes v3; add `migrate_dir(dir)` and an `UnsupportedVersion` exception class in `settings.py`. Files written by the new code must still be readable by the already-deployed `legacy_reader.py` (do not edit it). Standard library only. No network.
Must satisfy an unchangeable legacy reader (min_reader_version, legacy-safe theme value), preserve unknown keys at three levels, honour future versions, validate strictly (bool is not int), save atomically, and migrate idempotently with .bak copies. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. All 9 hidden unittest cases pass: load v1/v2 with defaults, load v3 (theme_mode), exact v3 on-disk shape readable by the deployed legacy reader, round-trips, unknown keys preserved at top/appearance/notifications, future version handling and UnsupportedVersion, validation errors leave the existing file untouched with no temp files left, migrate_dir counts/.bak/idempotence, legacy_reader.py unchanged.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 2m).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.08 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.43 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.12 | 43s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.