Skip to content

Water-quality sensor anomaly flags and daily means

Exact prompt

# Water-quality sensor anomaly flags `input.json` (synthetic buoy data) names `readings.csv` (`ts` "YYYY-MM-DDTHH:MM", `sensor`, `value`; rows are shuffled) and gives `stuck_n`, `window`, `spike_k`, and per-sensor `config` (`min`, `max`, `spike_floor`, `max_step`). Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json` writes `answer.json` and `chart.svg`. Fresh inputs are used too. ## Rules (per sensor, rows in ascending ts) 1. **missing**: value is blank or not a finite number (e.g. `err`, `nan`). It is skipped entirely (it does not break or extend a stuck run). 2. For other values, track the run of consecutive non-missing readings with exactly the same value (all flags included). Then the first match wins: - **range**: value < min or > max. - **stuck**: the current run length >= `stuck_n`. - **spike**: at least `window` ok values exist; with the last `window` ok values, med = median and MAD = median of |x - med|; flag if |v - med| > max(spike_k * MAD, spike_floor). - **rate**: |v - last ok value| > max_step. - else **ok**. Only ok values are used as "ok values", in daily means and in the chart. 3. `daily_means[sensor][date]` = mean of ok values that day (3 decimals); `counts[sensor]` has keys ok, missing, range, stuck, spike, rate; `flags` lists every non-ok reading `{"ts", "sensor", "flag"}` sorted by ts then sensor. ## Visual output: chart.svg Static, self-contained SVG (root `<svg>`, no scripts, images, links or external assets; renders headless) with `<polyline id="series-<sensor>">` for each sensor (`series-temp_c`, `series-do_mgl`, `series-ph`) and an element `id="flag-summary"`. The drawing itself is not graded. ## answer.json `{"counts": {...}, "daily_means": {...}, "flags": [...]}` ## Scoring Hidden checks (partial credit; all must pass): `missing_and_range`, `stuck_detection`, `spike_and_rate` (with ok counts), `flag_list`, `daily_means` (within 0.0011), `chart_svg_static` (chart.svg exists, follows the static rules, has the ids/tags above), `fresh_run_1`, `fresh_run_2`. 60-second limit per run.

Shuffled rows, a stuck run that ignores missing values but includes flagged ones, a robust median/MAD spike test with a floor that only uses ok history, a rate test against the last ok value, and flag precedence. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Benchmark code for comparing AI models, not navigation or safety software.

Success criteria: 1. missing_and_range 2. stuck_detection 3. spike_and_rate 4. flag_list 5. daily_means 6. chart_svg_static 7. fresh_run_1 8. fresh_run_2

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0094, 42s).

Latest output per model

Run this on another model

All runs (8)

75% pass rate across 8 eligible runs: 6 pass, 1 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
MMistral Small 3.2 24B Instruct (Venice)thinking on$0.001428s0Fully autonomousfailTrust 80%2026-10-27
QQwen3 Coder 480B Turbo (Venice)thinking on$0.0243s0Fully autonomouspartialTrust 80%2026-10-23
GGemini 3.8 Flashmedium · thinking on$0.463m0Fully autonomouspassTrust 80%2026-10-22
GGemini 4 Argonmedium · thinking on$0.324m0Fully autonomouspassTrust 80%2026-10-22
OGPT-6.1 Soldefault · thinking on$0.061m0Fully autonomouspassTrust 80%2026-10-22
AClaude Haiku 5.5default · thinking on$0.009442s0Fully autonomouspassTrust 80%2026-10-20
AClaude Opus 5.5default · thinking on$0.1735s0Fully autonomouspassTrust 80%2026-10-20
AClaude Sonnet 5.5default · thinking on$0.0725s0Fully autonomouspassTrust 80%2026-10-20

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Water-quality sensor anomaly flags and daily means

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.