Refactor nested legacy pricing code + add a bulk-discount rule with exact behaviour preservation
Exact prompt
# Task: refactor the legacy pricing engine and add one rule `pricing.py` has a deeply nested `price(order)` function. 1. Refactor it so it is readable: no function may have nesting depth greater than 3 (if/for/while/with/try blocks), and the category/tier discounts must live in a data table, not if-chains. Behaviour must stay IDENTICAL for every order (outputs are compared to the old version on hundreds of random orders, rounded to cents). 2. Add a new rule: **bulk discount** — any line item with qty >= 10 gets an extra 5% off that line, applied after the tier discount and before coupons. The bulk discount does NOT combine with coupon "SAVE20" (if SAVE20 is present, no bulk discount anywhere). Keep the signature `price(order) -> float`. Add tests in `test_pricing.py`. No network, no new dependencies. Order shape: {"customer":{"tier":"gold|silver|none"},"region":"EU|US|ROW","coupon":null|"SAVE10"|"SAVE20"|"FREESHIP","items":[{"category":"books|electronics|toys|garden","unit":float,"qty":int}]}
Refactor a deeply nested legacy pricing function into readable code with a discount table, then add a bulk-discount rule. Behaviour must stay identical to the cent on 800 random orders, including several odd regional tax and shipping quirks that a careless refactor would 'fix'. Nesting depth and the data-table requirement are checked from the code's syntax tree. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. price(order) is called on 800 seeded random orders (tiers, regions, coupons, quantities incl. >=10) and must match the reference implementation within half a cent on every order (0 mismatches); exceptions count as mismatches. 2. Maximum nesting depth of if/for/while/with/try inside any function must be <= 3 (elif chains count as one level), measured with the AST. 3. pricing.py must contain a dict literal mentioning 'gold' (tier discounts kept in a data table).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 1m).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.08 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.02 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.28 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.11 | 49s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.