Find vendor contracts whose non-renewal notice deadline is near, through amendments, month-end dates and business-day notice periods
Exact prompt
# Which vendor contracts need a non-renewal decision soon? Inputs: `contracts.csv`, `amendments.csv` and `calendar.json` (today's date and public holidays). Today is the date in `calendar.json`. For every contract that auto-renews, find the current term's end date and the last day on which a non-renewal notice can still be given. - A term of N months starting on date S ends the day before S plus N months. If that month is shorter, S plus N months falls on its last day (so 31 January plus 1 month is 28 or 29 February). - A contract renews for its renewal term each time a term ends, so the current term is the one that contains today. Each renewal term starts on the day after the previous term ends. - Notice must arrive at least the notice period before the term ends: the deadline is the end date minus the notice period. Periods are in days, months (same month rule as above) or business days (Monday to Friday, not holidays; count back from the end date, not counting the end date itself). - Only signed amendments change a contract; a later signed amendment to the same field replaces an earlier one, by effective date. - If the current term's deadline has already passed, the contract will renew: report the next term instead (its end date and deadline). Write `out/renewals.json`: `{"contracts": {"<contract_id>": {"term_end": "YYYY-MM-DD", "notice_deadline": "YYYY-MM-DD"}, ...}, "decide_within_30_days": [<ids whose reported deadline is from today to today + 30 days inclusive, sorted>], "missed_current_term": [<ids whose current-term deadline has passed, sorted>]}`. Leave contracts that don't auto-renew out of `contracts`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Work out the current term end and notice deadline for 18-24 auto-renewing contracts as of a fixed date. Notice periods mix days, months and business days with holidays, several start on the 29th-31st, amendments include drafts that must not apply, and some deadlines have already passed so the next term is the one to report. Thirty-day months, applying drafts, or treating business days as calendar days each change the answer. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Folder-restore note: this run's output files under work/out/ were lost in a 1:22 AM ET restore on Oct 21, 2026. The trace, step-in exchanges and output-hash list were checked against the recorded SHA-256 values and match; the hashes of the individual output files inside that list can no longer be recomputed. Measured data unchanged.
Success criteria: deadlines decide_soon missed amendments
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0073, 38s).
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.13 | 27s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 | |
| AClaude Haiku 5.5default · thinking on | $0.0073 | 38s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 18s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.