Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Find substantive changes hidden in a 'minor clean-up' contract redline

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.0091
Wall time
37s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Rescored on 2026-10-28. Previous score: 97.5 / 100, outcome: success. See the previous run.

Summary

Exact prompt

# Task: review a contract redline for the customer You represent the CUSTOMER. `msa_v1.txt` is the agreed draft; `msa_v2.txt` is the provider's new version, described by them as "minor clean-up only". Compare them. Write `changes.json`: a list of objects for every SUBSTANTIVE change (one that changes rights, obligations, amounts, durations, scope, or removes a clause). Ignore purely cosmetic edits (whitespace, capitalisation, punctuation, rephrasing with identical meaning). Each object: {"clause": "<number e.g. 2.2>", "kind": one of ["notice_period","payment_terms","cap_or_limit","duration","scope_narrowed","scope_broadened","ownership","new_obligation","governing_law","clause_removed","other"], "customer_impact": "worse"|"better"|"neutral", "summary": "<one sentence>"} Also write `memo.md` (≤250 words) telling the customer which 3 changes matter most and why. No network.

Review a contract redline for the customer: compare the agreed draft with a 'minor clean-up' version from the provider and list every substantive change. Ten real changes, some subtle (a liability cap cut from 50% to 15%, a dropped 'gross negligence' carve-out, a deleted clause), are mixed with eight cosmetic edits that should not be flagged. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. changes.json must flag at least 9 of the 10 substantive changes (matched by clause number). 2. The 'kind' value must be correct for at least 8 of the substantive changes. 3. At most 1 clause that only has cosmetic edits may be flagged. 4. customer_impact must be 'worse' for at least 9 of the substantive changes. 5. memo.md must exist and be <= 250 words.

Setup notes: Use Case Scout batch 3 case c06-contract-redline (offline fixture). Prepared with the case's setup script; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored automatically with the scout's scorer. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Rescored 2026-10-16 with Scout's fixed scorer. The original score was fail, and the model's output is unchanged. Claude Code's own reported cost for claude-haiku-5.5: $0.009096. Score: pass. Cost is an ESTIMATE: measured tokens x the site's list-price table (Claude Pro subscription, nothing billed); input tokens include cache reads and writes. Single run. Full scorer detail is kept in our audit records.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.0091
Wall time
37s
Human step-ins
0
Turns
12
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on211,5757,316$0.0091

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. changes.json must flag at least 9 of the 10 substantive changes (matched by clause number). 2. The 'kind' value must be correct for at least 8 of the substantive changes. 3. At most 1 clause that only has cosmetic edits may be flagged. 4. customer_impact must be 'worse' for at least 9 of the substantive changes. 5. memo.md must exist and be <= 250 words.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 35596,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.00909639,
  "usage": {
   "input_tokens": 16,
   "cache_creation_input_tokens": 17480,
   "cache_read_input_tokens": 194079,
   "output_tokens": 7316,
   "output_tokens_details": {
    "thinking_tokens": 3603
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 17480,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 548,
     "cache_read_input_tokens": 29540,
     "cache_creation_input_tokens": 479,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 479
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 16,
    "outputTokens": 7316,
    "cacheReadInputTokens": 194079,
    "cacheCreationInputTokens": 17480,
    "webSearchRequests": 0,
    "costUSD": 0.00909639,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 3603,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 12,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `changes.json` (10 substantive changes) and `memo.md` (250 words) in the working directory. The provider's \"minor clean-up only\" label is wrong: every substantive change is adverse to the customer.\n\n**The three that matter most:**\n1. **IP ownership (7.1):** the provider now owns all outputs generated by the Service. In v1 the customer owns outputs generated from its data.\n2. **Data location (6.1):** customer data may now be stored in the US as well as the EU. This may conflict with GDPR transfer rules or residency commitments.\n3. **Liability carve-out (5.2):** gross negligence is now subject to the liability cap. Combined with service credits falling from 50% to 15% (3.2), the customer's recovery for serious failure shrinks.\n\n**Other substantive changes:** renewal notice 90\u2192120 days (1.2), payment 45 days from receipt\u219230 days from invoice date (2.2), confidentiality 5\u21922 years (4.1), early termination fee added (8.1), governing law Ireland\u2192New York with Dublin venue unchanged (9.1), and the annual audit right deleted (10.3).\n\n**Cosmetic, ignored:** whitespace, the \"1 %\" spacing, capitalisation of \"Fees\", comma and hyphen changes in 4.2 and 10.2, \"that precede\" vs \"preceding\" in 5.1, \"shall have\" in 9.2, and trailing spaces in 7.2.\n\nI checked the result against a unified diff of the two files, which confirmed the list.",
  "ttft_ms": 1470,
  "type": "result",
  "duration_ms": 36099,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1166,
  "time_to_request_ms": 228,
  "first_content_frame_ms": 1166,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly