Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Triage a work inbox into read-only summaries and reply drafts, sending nothing

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.02
Wall time
1m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Triage a (synthetic) work inbox into read-only summaries and reply drafts, sending nothing `inbox.mbox` is a synthetic mailbox (made-up people and companies); `config.json` gives `me`, `vip` addresses and `urgent_words`. Write `solve.py` (Python 3 standard library only). `python solve.py input.json answer.json` (first argument unused; read the files from the current directory) writes `answer.json` and `drafts.mbox`. Fresh mailboxes are generated when scoring. ## Read-only and send-nothing rules 1. Never modify `inbox.mbox`. Never send anything: do not write `sent.log` or `outbox.mbox`, and solve.py must not use smtplib, imaplib, poplib, socket, urllib, http.client, requests, httpx, aiohttp, subprocess or webbrowser. `answer.json` must contain `"sent": []`. ## Parsing 2. Decode encoded headers (RFC 2047), quoted-printable and base64 bodies. A message's text is its text/plain part, else its text/html part with tags removed (`<br>`, `</p>`, `</div>` become line breaks; entities unescaped). Unquoted lines are lines that do not start with `>` (after leading spaces). 3. Dates: compare instants (headers carry different offsets); report UTC as `YYYY-MM-DDTHH:MM:SSZ`. Mailbox order is not time order. 4. Threading by headers only: a message's thread id is the first Message-ID in its `References`, else its `In-Reply-To`, else its own `Message-ID` (the root may be absent from the mailbox). Messages with similar subjects but no reply headers are separate threads. 5. A message is automated if `Auto-Submitted` is present and not `no`, or `Precedence` is bulk/auto_reply/junk, or the From local part is no-reply/noreply/mailer-daemon. 6. Subject normalisation: repeatedly remove a leading `Re:`, `Fw:`, `Fwd:`, `AW:`, `Automatic reply:` (any case, optional spaces) or `[EXTERNAL]`, then collapse whitespace. A thread's `subject` is the normalised subject of its earliest message. ## Per thread 7. The deciding message is the latest non-automated message. `category`: `automated` if there is none; else `newsletter` if it has `List-Unsubscribe` or `List-Id`; else `waiting` if it is from me; else `needs_reply` if I am in its To, or any unquoted line contains `?`; else `fyi`. 8. `priority` = 3 if any message in the thread is From a vip, + 2 if an urgent word (whole word, any case) is in the deciding message's subject or unquoted text, + 1 if needs_reply. 9. `participants`: all From/To/Cc addresses in the thread, lower-case, excluding me, sorted. `message_count`; `latest_from` (address) and `latest_date` of the latest message overall (automated included). 10. `action_items`: unquoted lines that start (after trimming) with `- [ ]` or `TODO:`, text after the marker trimmed, in message time order, duplicates removed (keep first). 11. Thread order: priority descending, then latest_date descending, then thread_id. ## Drafts (one per needs_reply thread, in drafts.mbox) 12. Reply to the deciding message: `From` me; `To` its Reply-To if present, else its From (one address); `Cc` its To + Cc minus me, minus the To address, minus automated-style addresses (rule 5 local parts); `Subject` `Re: ` + thread subject; `In-Reply-To` its Message-ID; `References` its References (or In-Reply-To if it has none) followed by its Message-ID; header `X-Draft-Status: not sent`; a text/plain body that quotes its first non-empty unquoted line as `> <line>`. ## answer.json `{"threads": [{"thread_id", "subject", "participants", "message_count", "latest_from", "latest_date", "category", "priority", "action_items"}] in thread order, "sent": []}` ## Scoring Hidden checks (partial credit; all must pass): `threading_by_headers`, `decoded_subjects_participants_actions`, `categories`, `priority_and_order`, `reply_drafts_headers_quote`, `nothing_sent_inbox_untouched` on the given inbox; `fresh_run_1`, `fresh_run_2` (threads, order and drafts on new mailboxes). 30-second limit per run.

Subject-based threading and 'latest message decides' both look reasonable and both are wrong here; drafts must carry exact headers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. threading_by_headers 2. decoded_subjects_participants_actions 3. categories 4. priority_and_order 5. reply_drafts_headers_quote 6. nothing_sent_inbox_untouched 7. fresh_run_1 8. fresh_run_2

Setup notes: Use Case Scout batch 13 case n04-inbox-triage-drafts (offline, synthetic fixture). Prepared with batch13/prepare.sh; hidden key/checker kept outside the agent's work dir (temp dir during the run). Prompt = TASK.md verbatim. Scored with the case's scorer: `python score.py WORKDIR KEYDIR` (python3, run from the case dir). Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.11 (8 cores). Claude Code's own reported cost for claude-haiku-5.5: $0.018413. Score: 8/8. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.02
Wall time
1m
Human step-ins
0
Turns
6
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on199,13320,308$0.02

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. threading_by_headers 2. decoded_subjects_participants_actions 3. categories 4. priority_and_order 5. reply_drafts_headers_quote 6. nothing_sent_inbox_untouched 7. fresh_run_1 8. fresh_run_2
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 85743,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.01841261,
  "usage": {
   "input_tokens": 12,
   "cache_creation_input_tokens": 32980,
   "cache_read_input_tokens": 166141,
   "output_tokens": 20308,
   "output_tokens_details": {
    "thinking_tokens": 14873
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 32980,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 2799,
     "cache_read_input_tokens": 43003,
     "cache_creation_input_tokens": 2516,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 2516
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 12,
    "outputTokens": 20308,
    "cacheReadInputTokens": 166141,
    "cacheCreationInputTokens": 32980,
    "webSearchRequests": 0,
    "costUSD": 0.01841261,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 14873,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 6,
  "subtype": "success",
  "api_error_status": null,
  "result": "`solve.py` is written and runs in about 0.04 s on the given inbox. It writes `answer.json` and `drafts.mbox` in the working directory, and `inbox.mbox` is unchanged (checksum verified). Nothing is sent: `\"sent\": []` is in the output, and the script imports none of the banned modules.\n\nI checked the output by hand against the spec. Priorities, categories, thread order, action items, participants and the draft headers all match what the rules give for this mailbox. I have not run it against fresh generated mailboxes.\n\nChoices I made where the spec was open:\n- **Thread ids** keep the angle brackets as they appear in the headers, e.g. `<[REDACTED:email]>`.\n- **Draft `To`** is the bare address from Reply-To or From, not the display-name form. For the budget thread that is `[REDACTED:email]`.\n- **Draft `Cc`** keeps the order in which addresses first appear, with duplicates removed.\n- **Draft body** is just the one quoted line followed by a newline.\n- **Mbox separator dates** come from each draft's deciding message, so reruns produce identical files.\n- **Output paths:** `drafts.mbox` goes in the current directory, named by `drafts_file` in `config.json`. `answer.json` goes to the second argument, defaulting to `answer.json`.\n\nFiles are in `./usecase-runner/runs/n04-claude-haiku/work/`:\n- solve.py\n- answer.json\n- drafts.mbox",
  "ttft_ms": 956,
  "type": "result",
  "duration_ms": 86273,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 810,
  "time_to_request_ms": 303,
  "first_content_frame_ms": 810,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly