Turn a meeting transcript into tracked action items (owners, relative due dates, reassignments, cancellations) plus a 120-word summary
Exact prompt
# Task: turn the meeting transcript into tracked action items `transcript.md` is the transcript of a team meeting held on Tuesday 2026-03-10. Produce two files in this directory: ## 1. `actions.json` A JSON object with one entry for every ticket ID (`OPS-<number>`) mentioned in the transcript, and no other keys: `{"OPS-101": {"owner": "<full name>", "due": "YYYY-MM-DD" or null, "status": "open" | "done" | "cancelled"}, ...}`. - `owner`: the person's full name exactly as written in the attendee list. "I'll take it" / "I'll own it" / "I fixed it" makes the speaker the owner. - When the transcript changes an owner, due date or status later on, the LATEST statement wins. - `status`: `done` if it is said to be done/fixed/finished, `cancelled` if it is dropped/cancelled, otherwise `open`. A done or cancelled item keeps its last due date (or null). - `due`: null if no due date is ever given. Resolve relative dates against the meeting date (Tuesday 2026-03-10), with weeks running Monday-Sunday: - "by/on <weekday>" or just "<weekday>": the first such day strictly after the meeting date (so "by Friday" = 2026-03-13 and "by Tuesday" = 2026-03-17). - "next <weekday>": that weekday in the following calendar week (Mon 2026-03-16 - Sun 2026-03-22). - "end of this week" / "end of next week": the Friday of the meeting week / of the following week. - "in N days" / "in N weeks": the meeting date plus N days / plus 7xN days. - "end of the month": the last day of the meeting's month; "end of <month name>": the last day of that month in 2026. - "the <Nth>": that day of the meeting's month if it is after the meeting date, otherwise of the next month. - Dates mentioned that are not given as the due date (for example when something expires) are not due dates. ## 2. `summary.md` A short status summary for people who missed the meeting: at most 120 words (counted as whitespace-separated tokens), mentioning the ID of every ticket that is still open, and not mentioning the ID of any cancelled ticket. The grader compares `actions.json` with the correct answers field by field (every ticket must be fully correct) and checks `summary.md` against the three rules above. No network.
Later statements override earlier ones (owner hand-off, earlier due date, drop, done), and relative dates ('by Friday' vs 'next Friday', 'in 10 days', 'the 17th', an expiry date that is not a due date) must be resolved with the stated rules. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. tickets_complete: exactly the 10 ticket IDs. 2. owners_correct. 3. due_dates_correct. 4. statuses_correct. 5. summary_word_limit: summary.md 1-120 whitespace tokens. 6. summary_mentions_open: every open ticket ID appears. 7. summary_omits_cancelled: no cancelled ticket ID appears.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0056, 20s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.04 | 38s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.0056 | 20s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Opus 5.5default · thinking on | $0.11 | 19s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Sonnet 5.5default · thinking on | $0.05 | 12s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.