Skip to content

Catch log checked against a made-up fishing regulations table

Exact prompt

# Catch log checked against a made-up regulations table `input.json` names `regulations.json` and `catch_log.csv`. All rules, species codes and data are **fictional** (not real fishing regulations). Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json`. Fresh files are used too. ## Data - `regulations.json`: `species` maps a code to `aliases`, `season` ["MM-DD", "MM-DD"] inclusive (if start > end the season wraps over New Year), `min_in`, `max_in` (null = no maximum), `bag` (legal kept fish per angler per day for that species); `aggregate_daily` = legal kept fish per angler per day, all species. - `catch_log.csv`: `date` (YYYY-MM-DD), `angler`, `species`, `length`, `unit` (`in` or `cm`, any case), `action` (`kept` or `released`, any case). Process rows in file order; row numbers start at 1 for the first data row. ## Rules (first match wins) 1. Species: trim, case-insensitive match to a code or alias; no match -> `unknown_species` (species null). 2. `released` -> status `released` (never a violation, never counted). 3. Kept fish: length in inches (cm / 2.54 rounded to 2 decimals). Then: not in season -> `out_of_season`; below min -> `undersize`; above max -> `oversize`; angler already has `bag` legal fish of that species that date -> `over_bag`; angler already has `aggregate_daily` legal fish that date -> `over_aggregate`; else `legal` (and it counts toward the limits). Boundaries are legal (equal to min or max, first/last season day). 4. Every status except `legal` and `released` is a violation for that angler. ## answer.json `{"rows": [{"row", "species", "status"}, ...], "violations_by_angler": {angler: n} (only anglers with >0, sorted by name), "legal_kept_by_species": {code: n} (only >0, sorted)}` ## Scoring Hidden checks (partial credit; all must pass): `species_and_release` (species codes and which rows are released), `row_statuses` (every row exact), `summaries`, `fresh_run_1`, `fresh_run_2`. 60-second limit per run.

Aliases with odd spacing and case, cm conversion near limits, a season that wraps New Year with boundary dates, slot maximum, and bag/aggregate limits that only count legal fish in log order. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Benchmark code for comparing AI models, not navigation or safety software.

Success criteria: 1. species_and_release 2. row_statuses 3. summaries 4. fresh_run_1 5. fresh_run_2

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0069, 27s).

Run this on another model

All runs (11)

64% pass rate across 11 eligible runs: 7 pass, 0 partial, 4 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
DDeepSeek V4.1 Flash (Venice)thinking on$0.0144s0Fully autonomouspassTrust 80%2026-11-04
QQwen3 Coder 480B Turbo (Venice)thinking on$0.005314s0Fully autonomousfailTrust 80%2026-11-04
MMistral Small 3.2 24B Instruct (Venice)thinking on$0.002535s0Fully autonomousfailTrust 80%2026-10-25
GGemma 4 31B Instruct (Venice)thinking on$0.00594m0Fully autonomousfailTrust 80%2026-10-24
GGemini 4 Argonmedium · thinking on$0.132m0Fully autonomouspassTrust 80%2026-10-22
GGemini 3.8 Flashmedium · thinking on$0.323m0Fully autonomouspassTrust 80%2026-10-22
OGPT-6.1 Soldefault · thinking on$0.0448s0Fully autonomouspassTrust 80%2026-10-22
ZGLM 4.7 (Venice)thinking on$0.0519m0Fully autonomousfailTrust 80%2026-10-21
AClaude Haiku 5.5default · thinking on$0.006927s0Fully autonomouspassTrust 80%2026-10-20
AClaude Opus 5.5default · thinking on$0.1627s0Fully autonomouspassTrust 80%2026-10-20
AClaude Sonnet 5.5default · thinking on$0.0617s0Fully autonomouspassTrust 80%2026-10-20

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Catch log checked against a made-up fishing regulations table

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.