Skip to content

Diagnose a rough idle from mock OBD-II readings before agreeing to an engine replacement

Exact prompt

# Diagnose a rough idle from OBD-II readings before agreeing to an engine replacement Your own (mock) car idles rough and a shop has quoted an engine replacement. Inputs: `vehicle.json` (cylinders, banks, which cylinders are on each bank), `idle.csv` and `cruise2500.csv` (t_s, rpm, coolant, coolant_unit, stft_b1, ltft_b1, stft_b2, ltft_b2, map, map_unit, maf_gps), `misfires.csv` (test, cyl, count, revs), `tests.txt` (the coil swap that was done) and `compression.csv` (cyl, psi), plus `shop-quote.txt`. All data is synthetic and read-only; nothing is connected to a car. These rules are the record; where the shop quote disagrees, the rules win. Rules: - Only warm rows count: a row is warm when coolant is at least 80 °C (coolant_unit F means Fahrenheit; convert with (F - 32) × 5/9). Ignore cold rows in both CSVs. - A bank's fuel trim is the mean over warm rows of stft + ltft for that bank, rounded half-up to 0.1. idle_map_kpa is the mean warm-row idle MAP in kPa (map_unit inHg means multiply by 3.38639), rounded half-up to 0.1. - A bank is lean when its idle trim is at least 10.0. If no bank is lean the fuel fault is none; otherwise it is vacuum_leak when every lean bank's cruise trim is below 5.0, and fuel_delivery otherwise. - A cylinder's misfire rate is count / revs × 1000 in the baseline test, rounded half-up to 0.1; it is misfiring when the rate is at least 20.0. - For the misfiring cylinder X (at most one): its coil was swapped with cylinder Y. If in the after_swap test Y is misfiring and X is not, the fault is ignition_coil and the cylinder is X (where the bad coil came from). Otherwise, if X's compression is below 75% of the highest cylinder's, the fault is low_compression, else plug_or_injector. With no misfiring cylinder the fault is none and the cylinder is null. - An engine replacement is justified only when at least two cylinders are below 75% of the highest compression. - Write `out/diagnosis.json`: `{"trims": {"idle": {"B1": <trim>, ...}, "cruise": {"B1": <trim>, ...}}, "idle_map_kpa": <kPa>, "fuel_fault": "none"|"vacuum_leak"|"fuel_delivery", "lean_banks": ["B1", ...] (sorted), "misfire_rates": {"<cyl>": <rate>, ...}, "misfire_fault": "none"|"ignition_coil"|"low_compression"|"plug_or_injector", "misfire_cylinder": <cyl or null>, "replacement_justified": true|false}`. Banks are B1 (and B2 on a two-bank engine). ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Work out what is really wrong with a rough-idling (mock) car from OBD-II logs before accepting a $4,200 engine quote. Cold-start rows must be dropped, coolant and manifold pressure come in mixed units, short- and long-term trims must be added per bank, a lean bank must be told apart as a vacuum leak or a fuel-delivery problem by its cruise trim, misfire rates sit exactly on the threshold, a coil swap shows whether the misfire follows the coil, and compression decides whether replacement is justified. The shop's note gets the warm-up, swap and verdict rules wrong. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: warm_trims map_units fuel_fault misfire_rates misfire_cause_and_verdict

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0052, 19s).

Run this on another model

All runs (3)

100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.1628s0Fully autonomouspassTrust 80%2026-10-23
AClaude Haiku 5.5default · thinking on$0.005219s0Fully autonomouspassTrust 80%2026-10-23
AClaude Sonnet 5.5default · thinking on$0.0619s0Fully autonomouspassTrust 80%2026-10-23

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Diagnose a rough idle from mock OBD-II readings before agreeing to an engine replacement

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.