Write a blameless postmortem from a timeline + chat export (automated rubric)

Write a blameless postmortem from an incident timeline and a chat export, with eight required sections in a fixed order and 400-700 words. Numbers and durations must match the sources, the employee who left must not be named, and action items need role owners. The scorer also rejects any number that does not appear in the sources. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. Exactly these H2 sections in this order: Summary, Impact, Timeline, Root cause, Detection, What went well, What went wrong, Action items. 2. 400–700 words. 3. States the correct incident duration (as hours+minutes or total minutes), and both the start and end times from the sources. 4. States the exact failed-payment count and affected-customer count from the sources. 5. Root cause section mentions an expired certificate; Detection section mentions customers/tweets/support and the alert/threshold problem. 6. Blameless: the departed employee is not named; no 'fault', 'blame', 'careless', 'negligent' wording. 7. At least 4 action-item bullets, every one containing 'TBD', and no person names in the Action items section. 8. No invented numbers: every number in the text must appear in the source files (a few small integers are whitelisted).

Best model for this task so far: DeepSeek V4.1 Flash (Venice) (thinking on) (success, $0.0046, 27s).

Run this on another model

All runs (5)

100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ