Classify 12 faulty sensors (frozen/°F/spike/gap/drift) with fault start hours
Exact prompt
# Task: find faulty sensors `readings.csv` has hourly temperature readings (sensor_id, hour 0–167, temp_c) from 40 indoor sensors over one week. Normal sensors follow a daily cycle around 18–24 °C with small noise. Some sensors developed exactly one fault each, starting at some hour. Fault types: `frozen` (value stops changing), `unit_fahrenheit` (starts reporting °F), `spike` (a single absurd reading), `offline_gap` (a run of missing hours), `drift` (steady upward drift). Write `faults.json`: `{"S07": {"type": "drift", "start_hour": 55}, ...}` listing ONLY faulty sensors. No network.
Find the faulty sensors among 40 indoor temperature sensors over one week of hourly readings, and name each fault type and the hour it started. Faults include frozen values, a switch to Fahrenheit, single spikes, offline gaps and slow drift, and flagging a healthy but noisy sensor counts against the answer. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. Fault type correct for at least 11 of the 12 faulty sensors. 2. Zero false positives: no healthy sensor may appear in faults.json. 3. start_hour (integer) within ±3 hours of the true start for at least 9 sensors (±8 hours allowed for drift faults).
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 53s).
All runs (5)
80% pass rate across 5 eligible runs: 4 pass, 0 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.05 | 46s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.06 | 9m | 0 | Fully autonomous | fail | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.01 | 53s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.19 | 38s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.08 | 23s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.