Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Refactor a legacy freight tariff into structured output while preserving its quirks for old orders

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.01
Wall time
50s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Refactor a legacy freight tariff without changing old invoices `legacy/tariff.py` is a 2019 script that prints one total per order from a pipe-separated file (`python legacy/tariff.py orders.txt`). Old invoices depend on its **exact** output, quirks included. Write `solve.py` (Python 3 standard library only): `python solve.py input.json answer.json`, where `input.json` has `orders` (each `id`, `date`, `zone`, `kg`, `declared`, `flags`; `orders.txt` holds the same orders in the legacy format). **The `legacy/` directory is deleted before scoring**, so `solve.py` must not import or run it. Fresh order sets are generated. ## Required behaviour 1. Orders dated **before 2027-01-01**: reproduce the legacy script exactly. That covers its `total` string and its `ERROR zone` cases, and every intermediate value as the legacy code computes it: billable kg, the rate it uses, base (after the express doubling), fuel, insurance, discount and the remote charge. Floating-point and formatting quirks included. 2. Orders dated **on or after 2027-01-01** follow the same legacy calculation, except: a. the zone rate is always the order's own zone rate (A, B or C; zone is case-insensitive), with no weight-based switch; any other zone is an error; b. flag `H` (hazmat) adds a flat 15.00 after everything else (not discounted). Before 2027 `H` is ignored, like any unknown flag. 3. Output per order, in input order: `{"id", "billable_kg", "rate", "base", "fuel", "insurance", "discount", "remote", "hazmat", "total"}` with numbers as JSON numbers and `total` as the string the legacy code would print (`"%.2f"`). An error order is `{"id", "error": "zone"}`. `hazmat` is 0 for orders without it, and `remote` is 9.99 or 0. ## answer.json `{"quotes": [...]}` ## Scoring Hidden checks (partial credit; all must pass; values must be equal exactly): `legacy_totals_preserved`, `legacy_breakdown` (pre-2027 orders), `new_rules_totals`, `new_rules_breakdown` (2027+ orders) on the given orders; `fresh_run_1`, `fresh_run_2` (all fields, all orders). 30-second limit per run.

A 'clean' rewrite with exact decimal rounding, or one that fixes the zone bug everywhere, breaks old totals; the legacy file is removed before scoring, so it must be reimplemented. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. legacy_totals_preserved 2. legacy_breakdown 3. new_rules_totals 4. new_rules_breakdown 5. fresh_run_1 6. fresh_run_2

Setup notes: Use Case Scout batch 12 case m09-legacy-freight-refactor (offline, synthetic fixture). Prepared with batch12/prepare.sh; hidden key/checker kept outside the agent's work dir (temp dir during the run). Prompt = TASK.md verbatim. Scored with the case's scorer: `python score.py WORKDIR KEYDIR` (python3, run from the case dir). Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.29 (8 cores). Claude Code's own reported cost for claude-haiku-5.5: $0.011608. Score: 6/6. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.01
Wall time
50s
Human step-ins
0
Turns
6
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on176,29410,514$0.01

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. legacy_totals_preserved 2. legacy_breakdown 3. new_rules_totals 4. new_rules_breakdown 5. fresh_run_1 6. fresh_run_2
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 48531,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.011607810000000001,
  "usage": {
   "input_tokens": 12,
   "cache_creation_input_tokens": 24141,
   "cache_read_input_tokens": 152141,
   "output_tokens": 10514,
   "output_tokens_details": {
    "thinking_tokens": 7307
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 24141,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 670,
     "cache_read_input_tokens": 35648,
     "cache_creation_input_tokens": 1032,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 1032
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 12,
    "outputTokens": 10514,
    "cacheReadInputTokens": 152141,
    "cacheCreationInputTokens": 24141,
    "webSearchRequests": 0,
    "costUSD": 0.011607810000000001,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 7307,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 6,
  "subtype": "success",
  "api_error_status": null,
  "result": "`solve.py` is written at `work/solve.py`. It uses only the standard library and runs `python solve.py input.json answer.json`. On the given orders it writes `answer.json` in about a second, well under the 30-second limit. I did not run the hidden checks, so the fresh-run checks are untested.\n\n**Verification:** For the 55 orders dated before 2027, the `total` strings match the legacy script's output exactly (0 mismatches). The intermediate values come from the same formulas in the same order of operations, so they should match too, but only the totals were compared against the legacy run. The 2027+ rules were spot-checked by hand on a few orders.\n\n**Legacy quirks kept as-is** (since old invoices depend on them):\n- Zone `Z` with more than 30 billable kg is billed at rate B, not treated as an error.\n- Only the last `D` flag sets the discount. Discounts don't stack.\n- Flags are case-sensitive, so `x` doesn't trigger the express doubling.\n- Insurance has a 0.50 floor for declared values over 100.\n- Remote and discount use the legacy order of operations.\n\n**2027+ rules:** the zone rate applies with no weight switch, and zone `Z` is an error. Hazmat `H` adds 15.00 after remote, and it is not discounted. Before 2027, `H` is ignored.\n\n**Choices I made where the spec was open:**\n- Numeric fields that are zero are output as `0.0`, not `0`. Numerically they are equal.\n- Zone is uppercased but not stripped, so `\" B\"` is an error, as it is in the legacy script.\n- Dates are parsed as ISO `YYYY-MM-DD`.",
  "ttft_ms": 1506,
  "type": "result",
  "duration_ms": 49029,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1401,
  "time_to_request_ms": 263,
  "first_content_frame_ms": 1402,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly