Refactor a legacy freight tariff into structured output while preserving its quirks for old orders
Exact prompt
# Refactor a legacy freight tariff without changing old invoices `legacy/tariff.py` is a 2019 script that prints one total per order from a pipe-separated file (`python legacy/tariff.py orders.txt`). Old invoices depend on its **exact** output, quirks included. Write `solve.py` (Python 3 standard library only): `python solve.py input.json answer.json`, where `input.json` has `orders` (each `id`, `date`, `zone`, `kg`, `declared`, `flags`; `orders.txt` holds the same orders in the legacy format). **The `legacy/` directory is deleted before scoring**, so `solve.py` must not import or run it. Fresh order sets are generated. ## Required behaviour 1. Orders dated **before 2027-01-01**: reproduce the legacy script exactly. That covers its `total` string and its `ERROR zone` cases, and every intermediate value as the legacy code computes it: billable kg, the rate it uses, base (after the express doubling), fuel, insurance, discount and the remote charge. Floating-point and formatting quirks included. 2. Orders dated **on or after 2027-01-01** follow the same legacy calculation, except: a. the zone rate is always the order's own zone rate (A, B or C; zone is case-insensitive), with no weight-based switch; any other zone is an error; b. flag `H` (hazmat) adds a flat 15.00 after everything else (not discounted). Before 2027 `H` is ignored, like any unknown flag. 3. Output per order, in input order: `{"id", "billable_kg", "rate", "base", "fuel", "insurance", "discount", "remote", "hazmat", "total"}` with numbers as JSON numbers and `total` as the string the legacy code would print (`"%.2f"`). An error order is `{"id", "error": "zone"}`. `hazmat` is 0 for orders without it, and `remote` is 9.99 or 0. ## answer.json `{"quotes": [...]}` ## Scoring Hidden checks (partial credit; all must pass; values must be equal exactly): `legacy_totals_preserved`, `legacy_breakdown` (pre-2027 orders), `new_rules_totals`, `new_rules_breakdown` (2027+ orders) on the given orders; `fresh_run_1`, `fresh_run_2` (all fields, all orders). 30-second limit per run.
A 'clean' rewrite with exact decimal rounding, or one that fixes the zone bug everywhere, breaks old totals; the legacy file is removed before scoring, so it must be reimplemented. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. legacy_totals_preserved 2. legacy_breakdown 3. new_rules_totals 4. new_rules_breakdown 5. fresh_run_1 6. fresh_run_2
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 50s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.05 | 56s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 50s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Opus 5.5default · thinking on | $0.21 | 42s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 22s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.