Find substantive changes hidden in a 'minor clean-up' contract redline
Exact prompt
# Task: review a contract redline for the customer You represent the CUSTOMER. `msa_v1.txt` is the agreed draft; `msa_v2.txt` is the provider's new version, described by them as "minor clean-up only". Compare them. Write `changes.json`: a list of objects for every SUBSTANTIVE change (one that changes rights, obligations, amounts, durations, scope, or removes a clause). Ignore purely cosmetic edits (whitespace, capitalisation, punctuation, rephrasing with identical meaning). Each object: {"clause": "<number e.g. 2.2>", "kind": one of ["notice_period","payment_terms","cap_or_limit","duration","scope_narrowed","scope_broadened","ownership","new_obligation","governing_law","clause_removed","other"], "customer_impact": "worse"|"better"|"neutral", "summary": "<one sentence>"} Also write `memo.md` (≤250 words) telling the customer which 3 changes matter most and why. No network.
Review a contract redline for the customer: compare the agreed draft with a 'minor clean-up' version from the provider and list every substantive change. Ten real changes, some subtle (a liability cap cut from 50% to 15%, a dropped 'gross negligence' carve-out, a deleted clause), are mixed with eight cosmetic edits that should not be flagged. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. changes.json must flag at least 9 of the 10 substantive changes (matched by clause number). 2. The 'kind' value must be correct for at least 8 of the substantive changes. 3. At most 1 clause that only has cosmetic edits may be flagged. 4. customer_impact must be 'worse' for at least 9 of the substantive changes. 5. memo.md must exist and be <= 250 words.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0091, 37s).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.04 | 52s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.02 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0091 | 37s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.19 | 38s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 29s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.