Water-quality sensor anomaly flags and daily means
Exact prompt
# Water-quality sensor anomaly flags `input.json` (synthetic buoy data) names `readings.csv` (`ts` "YYYY-MM-DDTHH:MM", `sensor`, `value`; rows are shuffled) and gives `stuck_n`, `window`, `spike_k`, and per-sensor `config` (`min`, `max`, `spike_floor`, `max_step`). Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json` writes `answer.json` and `chart.svg`. Fresh inputs are used too. ## Rules (per sensor, rows in ascending ts) 1. **missing**: value is blank or not a finite number (e.g. `err`, `nan`). It is skipped entirely (it does not break or extend a stuck run). 2. For other values, track the run of consecutive non-missing readings with exactly the same value (all flags included). Then the first match wins: - **range**: value < min or > max. - **stuck**: the current run length >= `stuck_n`. - **spike**: at least `window` ok values exist; with the last `window` ok values, med = median and MAD = median of |x - med|; flag if |v - med| > max(spike_k * MAD, spike_floor). - **rate**: |v - last ok value| > max_step. - else **ok**. Only ok values are used as "ok values", in daily means and in the chart. 3. `daily_means[sensor][date]` = mean of ok values that day (3 decimals); `counts[sensor]` has keys ok, missing, range, stuck, spike, rate; `flags` lists every non-ok reading `{"ts", "sensor", "flag"}` sorted by ts then sensor. ## Visual output: chart.svg Static, self-contained SVG (root `<svg>`, no scripts, images, links or external assets; renders headless) with `<polyline id="series-<sensor>">` for each sensor (`series-temp_c`, `series-do_mgl`, `series-ph`) and an element `id="flag-summary"`. The drawing itself is not graded. ## answer.json `{"counts": {...}, "daily_means": {...}, "flags": [...]}` ## Scoring Hidden checks (partial credit; all must pass): `missing_and_range`, `stuck_detection`, `spike_and_rate` (with ok counts), `flag_list`, `daily_means` (within 0.0011), `chart_svg_static` (chart.svg exists, follows the static rules, has the ids/tags above), `fresh_run_1`, `fresh_run_2`. 60-second limit per run.
Shuffled rows, a stuck run that ignores missing values but includes flagged ones, a robust median/MAD spike test with a floor that only uses ok history, a rate test against the last ok value, and flag precedence. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Benchmark code for comparing AI models, not navigation or safety software.
Success criteria: 1. missing_and_range 2. stuck_detection 3. spike_and_rate 4. flag_list 5. daily_means 6. chart_svg_static 7. fresh_run_1 8. fresh_run_2
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0094, 42s).
Latest output per model
All runs (8)
75% pass rate across 8 eligible runs: 6 pass, 1 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| MMistral Small 3.2 24B Instruct (Venice)thinking on | $0.0014 | 28s | 0 | Fully autonomous | fail | Trust 80% | 2026-10-27 | |
| QQwen3 Coder 480B Turbo (Venice)thinking on | $0.02 | 43s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-23 | |
| GGemini 3.8 Flashmedium · thinking on | $0.46 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| GGemini 4 Argonmedium · thinking on | $0.32 | 4m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| OGPT-6.1 Soldefault · thinking on | $0.06 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.0094 | 42s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Opus 5.5default · thinking on | $0.17 | 35s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 25s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.