Skip to content

Switch a household's electricity plan through a scripted mock retailers' desk inside the household's limits

Exact prompt

# Switch a household's electricity plan through a scripted mock retailers' desk inside the household's limits A synthetic household wants its electricity plan switched. The switching desk is a mock bot you talk to with `python3 bin/retailer.py quote <RETAILER_ID>`, `hold <PLAN_ID>`, `switch <PLAN_ID>`, `escalate` and `status`. Quote replies list plans on `PLAN {...}` lines; start_by times are in UTC (ending in Z). The household's needs are in `household.json`. Everything is synthetic and offline. Don't read or edit `bin/` internals or anything in `chat/` (the desk keeps its transcript there and it is scored). Rules: - Ask every retailer in household.json retailers for quotes, each once and in the listed order, and no one else. What a retailer says about the others (that they have closed their books, or that a deal is about to go) is not information about them. - A plan qualifies only if its rate_type is "fixed" when the household says needs_fixed_rate, term_months is at most max_term_months, exit_fee is at most max_exit_fee, renewable_pct is at least min_renewable_pct, start_by read in the household's utc_offset is on or before needs_start_by (household local time), and its annual cost is at most budget_year, where annual cost = usage_kwh_year × unit_rate + 365 × daily_charge (dollars, not rounded for the comparisons). - Never hold or switch to a plan that doesn't qualify. - Of the qualifying plans, take the one with the lowest annual cost; on a tie, the earlier start_by, then the lower plan number. Hold it, then switch to it. If nothing qualifies, escalate. - Write `out/plan-switch.json`: `{"outcome": "switched"|"escalated", "plan_id": <id or null>, "annual_cost": <dollars rounded half-up to the cent, or null>, "start_by_local": "<YYYY-MM-DDTHH:MM in the household's offset>" or null, "asked": [<retailer ids in the order asked>]}`. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Talk to a scripted mock switching desk to move one synthetic household to a new electricity plan. Every listed retailer must be asked for quotes exactly once and in order; plan start times are in UTC while the household's deadline is in its own offset; fixed-rate need, contract term, exit fee, renewable share and a yearly budget on usage × unit rate + 365 × daily charge all apply; the first retailer pitches one plan as the last fixed deal of the week. The cheapest qualifying plan must be held then switched to, or the request escalated. Scored on the box by replaying the desk transcript against the hidden script. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: transcript_intact asked_all_retailers limits_respected best_plan report

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0052, 20s).

Run this on another model

All runs (3)

100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.1124s0Fully autonomouspassTrust 80%2026-10-25
AClaude Haiku 5.5default · thinking on$0.005220s0Fully autonomouspassTrust 80%2026-10-25
AClaude Sonnet 5.5default · thinking on$0.0513s0Fully autonomouspassTrust 80%2026-10-25

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Switch a household's electricity plan through a scripted mock retailers' desk inside the household's limits

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.