Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Write release notes from a conventional-commit log (breaking changes, reverts, cherry-pick duplicates, semver bump)

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-19 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.0048
Wall time: 19s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: write the release notes `commits.txt` lists every commit since the last release, newest first, after two header lines (`previous:` = last released version, `release_date:`). Each commit is one line `<hash> <subject>`; indented lines after it are that commit's body. Write `RELEASE_NOTES.md` in this directory following these rules exactly. You may write it by hand or with a script. No network. 1. Conventional subjects look like `<type>(<scope>)!: <description> (#<PR>)`; the `(<scope>)`, the `!` and the ` (#<PR>)` suffix are each optional. A subject that does not have this shape (merge commits, "WIP ...") is excluded. 2. `feat` goes to **Features**, `fix` to **Fixes**, `perf` to **Performance**. All other types (`chore`, `docs`, `test`, `ci`, `style`, `refactor`, `build`, ...) are excluded, unless the commit is breaking. 3. A commit is breaking if it has `!` before the `:` or a body line starting with `BREAKING CHANGE: `. Breaking commits of ANY type are listed only under **Breaking changes** (not also in their type section). Their bullet text is the text after `BREAKING CHANGE: ` if there is such a line, otherwise the description. 4. A `revert:` commit whose body contains `This reverts commit <hash>.` removes itself and the commit with that hash. If that hash is not in the list, only the revert commit itself is excluded. Revert commits never appear in the notes. 5. If the same PR number appears on more than one included commit (cherry-picks), list it once, using the commit that appears first in the file. 6. Bullet format: `- **<scope>:** <text> (#<PR>)` when there is a scope, else `- <text> (#<PR>)`; leave out ` (#<PR>)` when the commit has no PR number. `<text>` is the description (or the breaking text from rule 3) with its first letter upper-cased and any trailing period removed; everything else is kept as written. 7. Inside a section, bullets with a PR number come first, sorted by PR number ascending, then bullets without a PR number in file order. 8. Version: from `previous` vX.Y.Z, bump major (to X+1.0.0) if anything breaking is included, else minor (X.Y+1.0) if any feature is included, else patch (X.Y.Z+1). 9. Layout: the first line is `# v<version> - <release_date>`. Then the non-empty sections in this order, each as a `## ` heading: `## Breaking changes`, `## Features`, `## Fixes`, `## Performance`, each followed by its bullets. The file contains only these heading lines, bullet lines and blank lines. The grader parses `RELEASE_NOTES.md` and compares the title line, the section order and every bullet line exactly (trailing spaces ignored).

Breaking footers on non-feature types, a revert that cancels a feature vs one whose target is absent, a duplicated PR, non-conventional subjects, PR-number ordering and a major bump all have one correct rendering. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. title_and_version_correct. 2. sections_correct_and_ordered. 3. breaking_bullets_exact. 4. features_bullets_exact. 5. fixes_bullets_exact. 6. performance_bullets_exact. 7. no_extra_lines: only headings, bullets and blank lines.

Setup notes: Use Case Scout batch 7 case g07-release-notes-from-commits (offline fixture). Prepared with batch7/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.004819. Score: 7/7 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.0048
Wall time: 19s
Human step-ins: 0
Turns: 3

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	64,310	3,956	$0.0048

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. title_and_version_correct. 2. sections_correct_and_ordered. 3. breaking_bullets_exact. 4. features_bullets_exact. 5. fixes_bullets_exact. 6. performance_bullets_exact. 7. no_extra_lines: only headings, bullets and blank lines.
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 17993,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.004818609999999999,
  "usage": {
   "input_tokens": 6,
   "cache_creation_input_tokens": 11563,
   "cache_read_input_tokens": 52741,
   "output_tokens": 3956,
   "output_tokens_details": {
    "thinking_tokens": 2956
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 11563,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 481,
     "cache_read_input_tokens": 20630,
     "cache_creation_input_tokens": 3472,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 3472
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 6,
    "outputTokens": 3956,
    "cacheReadInputTokens": 52741,
    "cacheCreationInputTokens": 11563,
    "webSearchRequests": 0,
    "costUSD": 0.004818609999999999,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 2956,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 3,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `RELEASE_NOTES.md` in the work directory. The release is **v3.0.0** (dated 2026-05-02), because a breaking commit is included, so the bump is major from v2.3.1.\n\n- **Breaking changes:** three commits, sorted by PR number. The #424 bullet uses the `BREAKING CHANGE:` line text, and the #411 bullet also uses its `BREAKING CHANGE:` text rather than its subject.\n- **Features:** four bullets in PR order (#407, #412, #417, #431).\n- **Fixes:** seven bullets. The no-PR bullet (db worker shutdown) comes last.\n- **Performance:** three bullets. The no-PR bullet (lazy-load report charts) comes last.\n\nExclusions:\n- The `chore`, `docs`, `test`, `ci`, `style`, `build` commits, the merge commit, and the `WIP` commit.\n- The dark mode feature (#420) and its revert, which removes it.\n- The progress-bar revert, because its target hash `9999999` is not in the list. Only the revert itself is dropped.\n- The second commit for PR #422, since the first occurrence (`1a9e0c2`) is kept.\n\nThe file was written by hand and checked against the rules. It is not a git repository, so I did not commit it.",
  "ttft_ms": 2745,
  "type": "result",
  "duration_ms": 18451,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 2381,
  "time_to_request_ms": 419,
  "first_content_frame_ms": 2382,
  "queued_turn_count": 0,
  "result_index": 0
 }
]