Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Sailing regatta series scoring with codes, dead heats, redress, discards and tie-breaks

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first trySupersededCompare

Estimated cost: $0.08
Wall time: 37s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Superseded. This run was rescored on 2026-10-20 and replaced by a newer run. It no longer counts toward rankings, use-case tables or the data export; it is kept here for the record.

Exact prompt

# Sailing regatta series scoring (low-point system) `input.json` (synthetic) has `entries` (`sail`), `races` (each with `finishes` in finishing order), `discards` ({"min races": discards}), and `penalty_pct`. Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json` writes `answer.json` and `results.html`. Fresh series are generated. ## Rules (N = number of entries) 1. A finish item is `{"sail": s}` or `{"sail": [s1, s2, ...]}` (a dead heat), with optional `code` (case-insensitive) and `points`. Sails not in `entries` are ignored. A boat already seen in that race is ignored (only its first item counts). 2. Codes DNC, DNS, OCS, DNF, RET, DSQ, DNE, UFD, BFD score **N + 1** and do **not** take a finishing place (boats behind move up). 3. Other boats take places 1, 2, 3, ... in item order; a dead heat of k boats takes the next k places and each scores their average (e.g. 2.5). 4. ZFP and SCP: place points + round_half_up(N * penalty_pct / 100), capped at N + 1. DPI: place points + `points`, capped at N + 1. 5. RDG with `points`: that score. RDG without `points`: the average of that boat's scores in all other races, excluding its other averaged-RDG races, rounded half-up to 1 decimal; N + 1 if there are none. 6. An entry missing from a race's items scores N + 1 with code DNC. A boat's code for a race is the upper-cased code (`""` if none). 7. Discards: d = the value for the largest key k with races sailed >= k (0 if none). Exclude the boat's d worst scores, never a DNE score; among equal scores exclude the later race first. `total` = sum of all scores; `net` = total - excluded. 8. Ranking: lower net first. Ties: (A8.1) compare each boat's non-excluded scores sorted best to worst, first difference wins; (A8.2) then compare scores in the last race, then the race before, and so on (excluded scores included). Still tied: entry order. ## Visual output: results.html A static, self-contained page (no `<script>`, `<img>`, `<link>`, `<iframe>`/`<object>`/`<embed>`, `on*` handlers, external URLs, CSS url(http)/@import or fetches; renders headless) with `<table id="results">` and one `<tr id="row-<sail>">` per boat, **in rank order**. Layout and styling are not graded. ## answer.json `{"boats": {sail: {"points": [per race], "codes": [per race], "discarded": [0-based race indices, ascending], "total", "net", "rank"}}, "order": [sails in rank order], "discards_applied": d}` ## Scoring Hidden checks (partial credit; all must pass; numbers within 0.001): `codes_and_dnc`, `race_points_incl_ties_penalties`, `discards_and_totals`, `ranking_with_tiebreaks`, `results_html_static` (exists, static rules, ids, row order) on the given series; `fresh_run_1`, `fresh_run_2`. 30-second limit per run.

Each rule is small, but the tests are layered: one place-counting slip changes every later boat's points, discards and the final order. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. codes_and_dnc 2. race_points_incl_ties_penalties 3. discards_and_totals 4. ranking_with_tiebreaks 5. results_html_static 6. fresh_run_1 7. fresh_run_2

Setup notes: Use Case Scout batch 12 case m06-regatta-scoring (offline, synthetic fixture). Prepared with batch12/prepare.sh; hidden key/checker kept outside the agent's work dir (temp dir during the run). Prompt = TASK.md verbatim. Scored with the case's scorer: `python score.py WORKDIR KEYDIR` (python3, run from the case dir). Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.32 (8 cores). results.html rendered offline with google-chrome headless (CDP pipe, network blocked): PNG 1280x800 at ~3 s after load; MP4 ~6 s captured at 1280x720, H.264 1280x720. Claude Code's own reported cost for claude-sonnet-5.5: $0.075389. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Output

Metrics

Estimated cost: $0.08
Wall time: 37s
Human step-ins: 0
Turns: 3

Model usage
Model	Input tok	Output tok	Cost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on	60,964	3,041	$0.08

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. codes_and_dnc 2. race_points_incl_ties_penalties 3. discards_and_totals 4. ranking_with_tiebreaks 5. results_html_static 6. fresh_run_1 7. fresh_run_2
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 35583,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.0753891,
  "usage": {
   "input_tokens": 6,
   "cache_creation_input_tokens": 9967,
   "cache_read_input_tokens": 50991,
   "output_tokens": 3041,
   "output_tokens_details": {
    "thinking_tokens": 674
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 9967,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 339,
     "cache_read_input_tokens": 19514,
     "cache_creation_input_tokens": 2992,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 2992
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 6,
    "outputTokens": 3041,
    "cacheReadInputTokens": 50991,
    "cacheCreationInputTokens": 9967,
    "webSearchRequests": 0,
    "costUSD": 0.0753891,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 674,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 3,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `solve.py` in the working directory. It ran on the given `input.json` and wrote `answer.json` and `results.html`, but I haven't checked its output against the hidden checks. I only looked at the start of `answer.json`: GBR309 ranks first with a net of 7.5, and FRA91 is second with 11.5. I didn't open `results.html`, so I haven't confirmed one row per boat or the static-page rules. The only check was that `grep` found `row-` on the page's single line.\n\nTwo rules were ambiguous, and I took the literal reading of each:\n- **RDG and places:** RDG isn't in the list of codes that skip a place, so an RDG finisher still takes a finishing place and boats behind it don't move up.\n- **Dead heat with a code:** a code on a dead-heat item applies to every boat in it.\n\nI rounded with exact fractions rather than floats, so half-up rounding is exact. `results.html` has no scripts, external links or styling.",
  "ttft_ms": 10485,
  "type": "result",
  "duration_ms": 35987,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 9797,
  "time_to_request_ms": 284,
  "first_content_frame_ms": 9798,
  "queued_turn_count": 0,
  "result_index": 0
 }
]