Skip to content

Pick the real root cause of an outage from a call graph, a change log and two hours of error logs

Exact prompt

# Find the root cause of last night's outage Inputs: `services.json` (which service calls which), `changes.json` (every deploy, config change and certificate rotation in the period), `errors.log` (one line per logged error, 02:00-04:00 UTC), `alert.json` (when the page fired). Definitions: - The baseline hour is 02:00-02:59 UTC and the incident window is 03:00-03:59 UTC. A service's baseline peak is its highest count of ERROR lines in any single baseline minute. - A service's onset is the first incident-window minute in which its ERROR count exceeds its baseline peak + 5. A service with no onset is unaffected. - The origin service is the affected service with the earliest onset whose onset is not preceded by the onset of a service it calls (directly or indirectly). If several qualify, the one with the earliest onset is the origin. - The root cause is the latest change to the origin service made at or before its onset minute. - For every other change, give the reason it is ruled out: "after_onset" (it happened after the origin's onset), "unaffected_service" (its service never had an onset), or "not_origin" (its service was affected, but only as a consequence of the origin). Write `out/rca.json`: `{"root_cause_change": "<change_id>", "origin_service": "<name>", "onset_utc": "HH:MM", "affected_services": [<sorted>], "ruled_out": {"<change_id>": "<reason>", ...}}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Find the origin of a cascading outage among 8-11 services: baseline-adjusted onsets from about 2,000 error lines, then the origin service whose onset no dependency's onset precedes, then the latest change to it before onset. Plausible decoys include a hotfix deployed just before the page fired, a change on a chronically noisy service, a mid-chain deploy and a later certificate rotation. Each change must also be ruled out with the right reason. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Folder-restore note: this run's output files under work/out/ were lost in a 1:22 AM ET restore on Oct 21, 2026. The trace, step-in exchanges and output-hash list were checked against the recorded SHA-256 values and match; the hashes of the individual output files inside that list can no longer be recomputed. Measured data unchanged.

Success criteria: root_cause origin_service onset_minute affected_services alert_adjacent_deploy_ruled_out all_ruled_out_reasons

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0060, 23s).

Run this on another model

All runs (4)

50% pass rate across 4 eligible runs: 2 pass, 1 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
MMistral Small 3.2 24B Instruct (Venice)thinking on$0.006116s0Fully autonomousfailTrust 80%2026-10-28
AClaude Opus 5.5default · thinking on$0.1526s0Fully autonomouspartialTrust 80%2026-10-21
AClaude Haiku 5.5default · thinking on$0.006023s0Fully autonomouspassTrust 80%2026-10-21
AClaude Sonnet 5.5default · thinking on$0.0614s0Fully autonomouspassTrust 80%2026-10-21

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Pick the real root cause of an outage from a call graph, a change log and two hours of error logs

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.