Skip to content

Triage free-text incident reports by inferred impact rather than the reporter's label

Exact prompt

# Triage the overnight incident reports Input: `reports.json`, 30-45 free-text incident reports. Each has a reporter-chosen priority label, which is often wrong. Judge severity from what the text says actually happened. Severity rules: - SEV1: any exposure of one customer's data to another party, or data loss, affecting external customers; or at least 20% of external users blocked with no workaround. - SEV2: at least 5% of external users blocked (with or without a workaround), unless it is SEV1; or data loss affecting internal users only. - SEV3: any other impact on external users (for example slowness, or a cosmetic problem). - SEV4: everything else (internal users only, no data loss). - "Blocked" means users cannot complete the task at all. A workaround counts only if users can apply it themselves. - Shares may be given as a percentage, as "N of M", or as "1 in N". Write `out/triage.json`: `{"severity": {"<report_id>": "SEV1" | "SEV2" | "SEV3" | "SEV4", ...}, "page_now": [<SEV1 report ids, sorted>], "top5": [<5 report ids>]}`. top5 = the five most urgent reports: by severity, then by larger share of users affected, then by report id. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Assign severities to 30-45 free-text incident reports from what each report says happened: who is affected (customers or staff), what share (as a percentage, 'N of M' or '1 in N'), whether users are fully blocked, whether a self-service workaround exists, and any data exposure or loss. Reporter labels and alarmed or casual tone are deliberately unreliable. Scoring checks the paging list, every severity and the top-5 order. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Folder-restore note: this run's output files under work/out/ were lost in a 1:22 AM ET restore on Oct 21, 2026. The trace, step-in exchanges and output-hash list were checked against the recorded SHA-256 values and match; the hashes of the individual output files inside that list can no longer be recomputed. Measured data unchanged.

Success criteria: page_now all_severities workaround_judged internal_vs_external labels_overridden top5

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0081, 30s).

Run this on another model

All runs (4)

50% pass rate across 4 eligible runs: 2 pass, 0 partial, 2 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
MMistral Small 3.2 24B Instruct (Venice)thinking on$0.001213s0Fully autonomousfailTrust 80%2026-10-28
AClaude Opus 5.5default · thinking on$0.1524s0Fully autonomouspassTrust 80%2026-10-21
AClaude Haiku 5.5default · thinking on$0.008130s0Fully autonomouspassTrust 80%2026-10-21
AClaude Sonnet 5.5default · thinking on$0.0716s0Fully autonomousfailTrust 80%2026-10-21

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Triage free-text incident reports by inferred impact rather than the reporter's label

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.