Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Sailing regatta series scoring with codes, dead heats, redress, discards and tie-breaks

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.01
Wall time
1m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Rescored on 2026-10-20. Previous score: 90.0 / 100, outcome: success. See the previous run.

Summary

Exact prompt

# Sailing regatta series scoring (low-point system) `input.json` (synthetic) has `entries` (`sail`), `races` (each with `finishes` in finishing order), `discards` ({"min races": discards}), and `penalty_pct`. Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json` writes `answer.json` and `results.html`. Fresh series are generated. ## Rules (N = number of entries) 1. A finish item is `{"sail": s}` or `{"sail": [s1, s2, ...]}` (a dead heat), with optional `code` (case-insensitive) and `points`. Sails not in `entries` are ignored. A boat already seen in that race is ignored (only its first item counts). 2. Codes DNC, DNS, OCS, DNF, RET, DSQ, DNE, UFD, BFD score **N + 1** and do **not** take a finishing place (boats behind move up). 3. Other boats take places 1, 2, 3, ... in item order; a dead heat of k boats takes the next k places and each scores their average (e.g. 2.5). 4. ZFP and SCP: place points + round_half_up(N * penalty_pct / 100), capped at N + 1. DPI: place points + `points`, capped at N + 1. 5. RDG with `points`: that score. RDG without `points`: the average of that boat's scores in all other races, excluding its other averaged-RDG races, rounded half-up to 1 decimal; N + 1 if there are none. 6. An entry missing from a race's items scores N + 1 with code DNC. A boat's code for a race is the upper-cased code (`""` if none). 7. Discards: d = the value for the largest key k with races sailed >= k (0 if none). Exclude the boat's d worst scores, never a DNE score; among equal scores exclude the later race first. `total` = sum of all scores; `net` = total - excluded. 8. Ranking: lower net first. Ties: (A8.1) compare each boat's non-excluded scores sorted best to worst, first difference wins; (A8.2) then compare scores in the last race, then the race before, and so on (excluded scores included). Still tied: entry order. ## Visual output: results.html A static, self-contained page (no `<script>`, `<img>`, `<link>`, `<iframe>`/`<object>`/`<embed>`, `on*` handlers, external URLs, CSS url(http)/@import or fetches; renders headless) with `<table id="results">` and one `<tr id="row-<sail>">` per boat, **in rank order**. Layout and styling are not graded. ## answer.json `{"boats": {sail: {"points": [per race], "codes": [per race], "discarded": [0-based race indices, ascending], "total", "net", "rank"}}, "order": [sails in rank order], "discards_applied": d}` ## Scoring Hidden checks (partial credit; all must pass; numbers within 0.001): `codes_and_dnc`, `race_points_incl_ties_penalties`, `discards_and_totals`, `ranking_with_tiebreaks`, `results_html_static` (exists, static rules, ids, row order) on the given series; `fresh_run_1`, `fresh_run_2`. 30-second limit per run.

Each rule is small, but the tests are layered: one place-counting slip changes every later boat's points, discards and the final order. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Oct 20, 2026: Scorer mode: file-based. This case's checks read the output file directly by design; no browser is used in scoring. Measured data unchanged.

Success criteria: 1. codes_and_dnc 2. race_points_incl_ties_penalties 3. discards_and_totals 4. ranking_with_tiebreaks 5. results_html_static 6. fresh_run_1 7. fresh_run_2

Setup notes: Oct 20, 2026: Scorer mode: file-based. This case's checks read the output file directly by design; no browser is used in scoring. Measured data unchanged. Use Case Scout batch 12 case m06-regatta-scoring (offline, synthetic fixture). Prepared with batch12/prepare.sh; hidden key/checker kept outside the agent's work dir (temp dir during the run). Prompt = TASK.md verbatim. Scored with the case's scorer: `python score.py WORKDIR KEYDIR` (python3, run from the case dir). Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.23 (8 cores). results.html rendered offline with google-chrome headless (CDP pipe, network blocked): PNG 1280x800 at ~3 s after load; MP4 ~6 s captured at 1280x720, H.264 1280x720. Claude Code's own reported cost for claude-haiku-5.5: $0.01488. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Output

Metrics

Estimated cost
$0.01
Wall time
1m
Human step-ins
0
Turns
7
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on208,61415,442$0.01

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. codes_and_dnc 2. race_points_incl_ties_penalties 3. discards_and_totals 4. ranking_with_tiebreaks 5. results_html_static 6. fresh_run_1 7. fresh_run_2
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 71433,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.014880259999999998,
  "usage": {
   "input_tokens": 14,
   "cache_creation_input_tokens": 26694,
   "cache_read_input_tokens": 181906,
   "output_tokens": 15442,
   "output_tokens_details": {
    "thinking_tokens": 9677
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 26694,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 536,
     "cache_read_input_tokens": 35843,
     "cache_creation_input_tokens": 3390,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 3390
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 14,
    "outputTokens": 15442,
    "cacheReadInputTokens": 181906,
    "cacheCreationInputTokens": 26694,
    "webSearchRequests": 0,
    "costUSD": 0.014880259999999998,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 9677,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 7,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `solve.py` in the work directory. It runs on the given series and writes `answer.json` and `results.html`. I checked the given series by hand against the rules, and it passed 60 random series without crashing. I did not run the hidden checks, so I can't confirm those.\n\nThe given series ranks GBR309 first (net 7.5) and FRA668 last (net 51). NZL128 and USA79 tie on net 51.5 and are separated by the first-difference rule. Race points, discards, and the penalty and RDG cases match my hand calculations.\n\nChoices I made where the spec was open:\n- **Races sailed for discards:** I used the number of races in the series, so one discard count `d` applies to every boat and matches the top-level `discards_applied` field. Per-boat sailed counts would have made that field meaningless.\n- **RDG takes a place:** I read rule 3 literally, so an RDG boat consumes a finishing place and boats behind it move down one place. Its score is `points` or the averaged value.\n- **DPI without `points`:** treated as 0.\n- **Unknown codes** (e.g. `XYZ`): scored as a normal finisher.\n- **Sail matching:** exact first, then case-insensitive and trimmed.\n- **results.html location:** written next to `answer.json`, which is the working directory in the normal run.\n- **Ties** that survive A8.1 and A8.2 fall back to entry order.",
  "ttft_ms": 1608,
  "type": "result",
  "duration_ms": 74399,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1406,
  "time_to_request_ms": 266,
  "first_content_frame_ms": 1406,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly