Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Refactor nested legacy pricing code + add a bulk-discount rule with exact behaviour preservation

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.02
Wall time: 1m
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: refactor the legacy pricing engine and add one rule `pricing.py` has a deeply nested `price(order)` function. 1. Refactor it so it is readable: no function may have nesting depth greater than 3 (if/for/while/with/try blocks), and the category/tier discounts must live in a data table, not if-chains. Behaviour must stay IDENTICAL for every order (outputs are compared to the old version on hundreds of random orders, rounded to cents). 2. Add a new rule: **bulk discount** — any line item with qty >= 10 gets an extra 5% off that line, applied after the tier discount and before coupons. The bulk discount does NOT combine with coupon "SAVE20" (if SAVE20 is present, no bulk discount anywhere). Keep the signature `price(order) -> float`. Add tests in `test_pricing.py`. No network, no new dependencies. Order shape: {"customer":{"tier":"gold|silver|none"},"region":"EU|US|ROW","coupon":null|"SAVE10"|"SAVE20"|"FREESHIP","items":[{"category":"books|electronics|toys|garden","unit":float,"qty":int}]}

Refactor a deeply nested legacy pricing function into readable code with a discount table, then add a bulk-discount rule. Behaviour must stay identical to the cent on 800 random orders, including several odd regional tax and shipping quirks that a careless refactor would 'fix'. Nesting depth and the data-table requirement are checked from the code's syntax tree. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. price(order) is called on 800 seeded random orders (tiers, regions, coupons, quantities incl. >=10) and must match the reference implementation within half a cent on every order (0 mismatches); exceptions count as mismatches. 2. Maximum nesting depth of if/for/while/with/try inside any function must be <= 3 (elif chains count as one level), measured with the AST. 3. pricing.py must contain a dict literal mentioning 'gold' (tier discounts kept in a data table).

Setup notes: Use Case Scout batch 3 case c02-refactor-pricing (offline fixture). Prepared with batch3/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.016899. Score: 1/1 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.02
Wall time: 1m
Human step-ins: 0
Turns: 9

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	279,757	18,178	$0.02

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. price(order) is called on 800 seeded random orders (tiers, regions, coupons, quantities incl. >=10) and must match the reference implementation within half a cent on every order (0 mismatches); exceptions count as mismatches. 2. Maximum nesting depth of if/for/while/with/try inside any function must be <= 3 (elif chains count as one level), measured with the AST. 3. pricing.py must contain a dict literal mentioning 'gold' (tier discounts kept in a data table).
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 80118,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.01689868,
  "usage": {
   "input_tokens": 18,
   "cache_creation_input_tokens": 26371,
   "cache_read_input_tokens": 253368,
   "output_tokens": 18178,
   "output_tokens_details": {
    "thinking_tokens": 11205
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 26371,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 960,
     "cache_read_input_tokens": 37972,
     "cache_creation_input_tokens": 938,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 938
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 18,
    "outputTokens": 18178,
    "cacheReadInputTokens": 253368,
    "cacheCreationInputTokens": 26371,
    "webSearchRequests": 0,
    "costUSD": 0.01689868,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 11205,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 9,
  "subtype": "success",
  "api_error_status": null,
  "result": "The refactor is done, the bulk rule is in, and all 30 tests pass.\n\n**Refactor (`pricing.py`)**\n- Category/tier discounts now live in `CATEGORY_TIER_RATES`, with `DEFAULT_TIER_RATES` for toys, garden and unknown categories. The old if-chains are gone.\n- `price()` is split into `_tier_rate`, `_line_total`, `_apply_coupon` and `_shipping`. The deepest function is `_shipping` at nesting depth 3, which I checked with `ast`.\n- The arithmetic keeps the legacy operation order, including the US tax on `(total - ship)`, so results should match the old code bit for bit.\n- I checked equivalence in a scratch directory outside the deliverable, against a copy of the original. Across 20,000 random orders where the new rule can't fire (all qty < 10, or SAVE20 present), there were 0 mismatches.\n\n**Bulk rule**\n- Each line with `qty >= 10` gets `\u00d7 0.95` after its tier rate and before coupons.\n- It is switched off for every line when `coupon == \"SAVE20\"`.\n- Orders with qty \u2265 10 and no SAVE20 change, as intended: 1,595 of about 2,000 sampled orders with a qty \u2265 10 line changed.\n\n**Tests (`test_pricing.py`)**\n- 30 `unittest` tests, run with `python3 -m unittest`, no new dependencies. They cover tier rates, bulk thresholds and ordering, bulk with SAVE20/SAVE10/FREESHIP, coupon thresholds, EU/US/ROW shipping and tax, and the empty order.\n- The expected values are hand-calculated. Two of my first guesses were wrong (one arithmetic slip, one misleading name), and I corrected them before the final run.\n\n**Small differences to be aware of**\n- `price()` now always returns a `float`, as the signature says. Legacy could return an `int` for int-only inputs or an empty ROW order (e.g. `15` instead of `15.0`). The values are equal.\n- `customer.tier` is read once at the top. Legacy only read it inside the loop or for some US orders, so a malformed order missing `customer` could raise now where it previously didn't. Valid orders are unaffected.\n- I used an explicit loop for the line sum rather than `sum()`, because Python 3.12+ uses compensated summation for floats, which can differ in the last bit.\n\nThe directory isn't a git repo, so I didn't commit anything.",
  "ttft_ms": 1478,
  "type": "result",
  "duration_ms": 81129,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1284,
  "time_to_request_ms": 250,
  "first_content_frame_ms": 1284,
  "queued_turn_count": 0,
  "result_index": 0
 }
]