Skip to content

Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Settle a wedding caterer's final headcount from changing replies, merged duplicate guests and birthdays on the day

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-22 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.06
Wall time
19s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Work out the wedding caterer's final headcount and bill Inputs: `guests.csv` (guest_id, name, date_of_birth, merged_into), `rsvps.csv` (every reply received: guest_id, replied_at, attending, plus_one), `menu.json` (event date, prices, the child age limit and the service charge) and `planner-notes.txt`. Every amount is in pounds. The files are the record; where notes disagree with them, the files win. Rules: - A guest record with merged_into set is a duplicate: its replies count as replies of the guest it names, and it is never a guest in its own right. - Each guest's latest reply by actual time counts (replied_at carries a UTC offset; compare the moments, not the text). - A guest attends if their latest reply says attending yes; a guest with no reply does not attend. If that reply says plus_one yes, they bring one extra adult. - A guest is a child if they are under 12 on the event_date, worked out from the full date of birth (a guest whose 12th birthday is on the event_date is an adult). - Food cost = adults x adult_price + children x child_price (plus-ones count as adults). - Service charge = service_charge_pct of the food cost, rounded half-up to the penny. Write `out/catering.json`: `{"attending": [<sorted guest_ids attending>], "adults": <count incl. plus-ones>, "children": <count>, "food": <food cost>, "service_charge": <amount>, "total": <food + service charge>}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Turn a guest list and every RSVP received into a final headcount and catering bill. Guests changed their minds from phones in different time zones, some guests appear twice and were merged, children are priced by age on the wedding day (one turns 12 that day), and plus-ones count as adults. A planner's note offers a stale headcount and a wrong age rule. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: latest_reply children adults money

Setup notes: Batch 27 case c02-wedding-catering-headcount (offline, synthetic inputs, seed 1499421488). Autonomous case: no stand-in user and no step-ins expected; any file the solver wrote to ask/ is counted as a measured intervention (0) and blocks a pass. Prompt = TASK.md verbatim. Hidden key kept outside the solver's work dir. Scored with the case's scorer (python3 score.py WORKDIR KEYDIR, absolute paths); pass = every check true and no ask/ files. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.24 (8 cores). Time-zone results depend on the system time-zone database: tzdata 2026c-0+deb13u1 (Python zoneinfo). Claude Code's own reported cost for claude-sonnet-5.5: $0.056565. Score: 4/4. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.06
Wall time
19s
Human step-ins
0
Turns
4
Model usage
ModelInput tokOutput tokCost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on79,8571,330$0.06

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: latest_reply children adults money
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 18383,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.056564699999999996,
  "usage": {
   "input_tokens": 8,
   "cache_creation_input_tokens": 9042,
   "cache_read_input_tokens": 70807,
   "output_tokens": 1330,
   "output_tokens_details": {
    "thinking_tokens": 95
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 9042,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 408,
     "cache_read_input_tokens": 20732,
     "cache_creation_input_tokens": 849,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 849
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 8,
    "outputTokens": 1330,
    "cacheReadInputTokens": 70807,
    "cacheCreationInputTokens": 9042,
    "webSearchRequests": 0,
    "costUSD": 0.056564699999999996,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 95,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 4,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `out/catering.json`: 14 guests attending, 19 covers, and a total bill of \u00a3804.67.\n\n- **Attending:** G100, G103, G106, G109, G115, G133, G139, G145, G148, G151, G154, G169, G172, G175.\n- **Adults:** 14, including plus-ones.\n- **Children:** 5.\n- **Food:** \u00a3715.26.\n- **Service charge:** \u00a389.41, which is 12.5% of the food cost rounded half-up to the penny.\n- **Total:** \u00a3804.67.\n\nI ignored both planner's notes. The \"headcount of 20\" and the \"born in 2014 or later is a child\" rule conflict with the files, which win. For example, G103 was born 2014-11-21 and turns 12 on 2026-11-21, after the 2026-11-14 event, so they count as a child under the age rule. The born-in-2014 shortcut would also have wrongly made G100 (born 2014-11-14) a child. G100's 12th birthday falls on the event date, so they count as an adult.\n\nI did not eyeball the replies. A script applied the rules to the input files: duplicate records are merged into the guest they name, the latest reply by actual moment wins, and age is worked out from the full date of birth.",
  "ttft_ms": 3542,
  "type": "result",
  "duration_ms": 18741,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 2686,
  "time_to_request_ms": 238,
  "first_content_frame_ms": 2686,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly