Diagnose a rough idle from mock OBD-II readings before agreeing to an engine replacement
Exact prompt
# Diagnose a rough idle from OBD-II readings before agreeing to an engine replacement Your own (mock) car idles rough and a shop has quoted an engine replacement. Inputs: `vehicle.json` (cylinders, banks, which cylinders are on each bank), `idle.csv` and `cruise2500.csv` (t_s, rpm, coolant, coolant_unit, stft_b1, ltft_b1, stft_b2, ltft_b2, map, map_unit, maf_gps), `misfires.csv` (test, cyl, count, revs), `tests.txt` (the coil swap that was done) and `compression.csv` (cyl, psi), plus `shop-quote.txt`. All data is synthetic and read-only; nothing is connected to a car. These rules are the record; where the shop quote disagrees, the rules win. Rules: - Only warm rows count: a row is warm when coolant is at least 80 °C (coolant_unit F means Fahrenheit; convert with (F - 32) × 5/9). Ignore cold rows in both CSVs. - A bank's fuel trim is the mean over warm rows of stft + ltft for that bank, rounded half-up to 0.1. idle_map_kpa is the mean warm-row idle MAP in kPa (map_unit inHg means multiply by 3.38639), rounded half-up to 0.1. - A bank is lean when its idle trim is at least 10.0. If no bank is lean the fuel fault is none; otherwise it is vacuum_leak when every lean bank's cruise trim is below 5.0, and fuel_delivery otherwise. - A cylinder's misfire rate is count / revs × 1000 in the baseline test, rounded half-up to 0.1; it is misfiring when the rate is at least 20.0. - For the misfiring cylinder X (at most one): its coil was swapped with cylinder Y. If in the after_swap test Y is misfiring and X is not, the fault is ignition_coil and the cylinder is X (where the bad coil came from). Otherwise, if X's compression is below 75% of the highest cylinder's, the fault is low_compression, else plug_or_injector. With no misfiring cylinder the fault is none and the cylinder is null. - An engine replacement is justified only when at least two cylinders are below 75% of the highest compression. - Write `out/diagnosis.json`: `{"trims": {"idle": {"B1": <trim>, ...}, "cruise": {"B1": <trim>, ...}}, "idle_map_kpa": <kPa>, "fuel_fault": "none"|"vacuum_leak"|"fuel_delivery", "lean_banks": ["B1", ...] (sorted), "misfire_rates": {"<cyl>": <rate>, ...}, "misfire_fault": "none"|"ignition_coil"|"low_compression"|"plug_or_injector", "misfire_cylinder": <cyl or null>, "replacement_justified": true|false}`. Banks are B1 (and B2 on a two-bank engine). ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Work out what is really wrong with a rough-idling (mock) car from OBD-II logs before accepting a $4,200 engine quote. Cold-start rows must be dropped, coolant and manifold pressure come in mixed units, short- and long-term trims must be added per bank, a lean bank must be told apart as a vacuum leak or a fuel-delivery problem by its cruise trim, misfire rates sit exactly on the threshold, a coil swap shows whether the misfire follows the coil, and compression decides whether replacement is justified. The shop's note gets the warm-up, swap and verdict rules wrong. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: warm_trims map_units fuel_fault misfire_rates misfire_cause_and_verdict
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0052, 19s).
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.16 | 28s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-23 | |
| AClaude Haiku 5.5default · thinking on | $0.0052 | 19s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-23 | |
| AClaude Sonnet 5.5default · thinking on | $0.06 | 19s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-23 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.