Skip to content

Compare home and auto insurance renewal quotes and pick the cheapest one that isn't a downgrade

Exact prompt

# Compare home and auto insurance renewal quotes against your current coverage and pick the cheapest one that isn't a downgrade All policies, quotes and carriers here are synthetic. Inputs: `current.json` (for `home` and `auto`: `annual_premium`, `limits` {item: dollars} and `deductibles` {item: dollars}), `quotes.csv` (quote_id, line, carrier, annual_premium) and `quote_terms.csv` (quote_id, item, kind, amount), where kind is `limit` or `deductible`. Amounts and premiums are in dollars. `broker-notes.txt` is a broker's summary; where it disagrees with the rules below, the rules win. Rules: - Downgrades: compare each quote with the current policy for its line. A quote has the reason `missing_line` when an item in the current `limits` has no `limit` row in the quote, `lower_limit` when any quoted limit is below the current limit for that item, and `higher_deductible` when any quoted deductible is above the current deductible for that item. Equal amounts are not a downgrade, and an extra item the current policy doesn't have is fine. A quote with at least one reason is a downgrade. - Picks: for each line, pick the quote with the lowest `annual_premium` among that line's quotes that are not downgrades. - Savings: a line's saving is the current `annual_premium` minus the picked quote's `annual_premium` (it can be negative); `total_annual_savings` is the sum over both lines. - Write `out/decision.json`: `{"picks": {"home": <quote_id>, "auto": <quote_id>}, "downgrade_quotes": [<sorted quote_ids>], "annual_savings": {"home": <dollars>, "auto": <dollars>}, "total_annual_savings": <dollars>, "downgrades_by_reason": {"<reason>": <number of quotes with that reason>, ...}}`. - Roll-ups: Reasons among missing_line, lower_limit and higher_deductible with no downgraded quote are included at 0. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Compare synthetic home and auto renewal quotes with your current coverage, item by item. A quote is a downgrade when it drops an item, lowers a limit or raises a deductible; equal amounts and extra items are fine. For each line, pick the cheapest quote that isn't a downgrade, then report the downgrades, the saving per line and in total, and a count per downgrade reason, with reasons that never come up listed at 0. The cheapest quote on each line quietly drops or cuts cover, and a cheaper quote with a doubled deductible is a second trap. The broker's notes say to take the cheapest and to ignore deductibles. Inspired by https://x.com/akshot/status/2114522453036662811 Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: picks downgrade_quotes annual_savings total_annual_savings downgrades_by_reason

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0039, 16s).

Run this on another model

All runs (3)

100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.1215s0Fully autonomouspassTrust 80%2026-10-26
AClaude Haiku 5.5default · thinking on$0.003916s0Fully autonomouspassTrust 80%2026-10-26
AClaude Sonnet 5.5default · thinking on$0.0513s0Fully autonomouspassTrust 80%2026-10-26

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Compare home and auto insurance renewal quotes and pick the cheapest one that isn't a downgrade

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.