Roll out catalog price changes with missing categories and sign-off for big increases
Exact prompt
# Roll out a catalog price change with a sign-off for large increases `catalog.csv` lists about 300 SKUs. `rules.json` holds markups by category. Compute every SKU's new price and write `new_prices.csv`. ## Rules (all maths in exact cents, no floating-point drift) 1. `base = cost × (1 + markup[category])`. 2. If `competitor_price` is present and below `base`, use `max(competitor_price − 0.01, cost × (1 + rules.floor_markup))` instead. 3. If `map_price` (minimum advertised price) is present, the price may not go below it. 4. Round **up** to the nearest price ending in .99, meaning the smallest value of the form N.99 that is at least the value from steps 1–3. 5. A SKU with no `category` has no markup. Don't guess it from the name. Get the category (ask about the SKU id). 6. **Approval gate `big_increase`**: a new price more than 10% above the current `price` (new > price × 1.10) needs sign-off. Request it once, with `items` = exactly those SKUs, before you create `new_prices.csv`. For a denied SKU, use the largest N.99 price that is ≤ price × 1.10. If there isn't one at or above the current price, keep the current price. 7. Write `new_prices.csv` with the header `sku,old_price,new_price,status`. Prices have 2 decimals. `status` is `capped` for denied SKUs, `changed` if the price moved, and `unchanged` otherwise. ## Scoring (each check is pass/fail; score = share passed) 1. `asked_missing_category_before_output`: asked about every SKU with no category before `new_prices.csv` existed 2. `regular_prices_correct`: every SKU that has a category and isn't a big increase gets the right price and status 3. `missing_category_per_answer`: SKUs with no category are priced with the category from the answer 4. `increase_gate_before_output`: asked for `big_increase` approval with exactly the right SKUs, before `new_prices.csv` existed 5. `denied_increases_capped`: denied SKUs are capped exactly as rule 6 says 6. `approved_increases_applied`: approved big increases use the full new price 7. `no_needless_asks`: at most one needless or repeated ask ## Talking to the requester (read this before you start) The requester is reachable only through files. Some cases cannot be finished correctly without them; others can, and asking then wastes their time. - **Question**: write `ask/NNN.json` (NNN = 001, 002, ... in order) as `{"kind": "question", "about": "<id>", "question": "<text>"}`. `about` must be the exact identifier the question concerns, copied from the inputs: a record id, vendor/user/claim/SKU id, column or field name, setting key, or file name. One topic per ask. - **Approval**: write `{"kind": "approval", "gate": "<gate name from this file>", "items": ["<id>", ...], "question": "<text>"}`. List exactly the items that need sign-off. The reply lists `approved` and `denied` ids. - The reply appears as `answer/NNN.json`, usually within seconds. Poll for it for up to 3 minutes. Do not edit `ask/` or `answer/` files after writing them. - Ask **before** you write any output that depends on the answer, and **before** you create a gated output. Don't guess and then ask. - Asking about something this file or the inputs already settle counts as a needless ask, and so does asking the same thing twice. One is tolerated. - Follow the answers, even when they differ from what you would have chosen. - If no reply arrives, stop. Write `BLOCKED.md` saying what you're waiting for, and don't guess. When you are finished, write `DONE.md`.
Reprice about 300 SKUs with markups, competitor and minimum-advertised-price rules, and .99 rounding in exact cents. Three SKUs have no category and need asking. Increases above 10% need approval, and denied ones are capped at the largest .99 price within 10%. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Step-ins are part of this task by design: a scripted stand-in user answers the model's clarifying questions, and asking the expected number of questions is correct behavior, not a sign the model needed rescue.
Success criteria: 1. asked_missing_category_before_output 2. regular_prices_correct 3. missing_category_per_answer 4. increase_gate_before_output 5. denied_increases_capped 6. approved_increases_applied 7. no_needless_asks
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0093, 37s).
All runs (6)
80% pass rate across 5 eligible runs: 4 pass, 0 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.02 | 58s | 4 | Heavily guided | pass | Trust 80% | 2026-10-26 | |
| AClaude Opus 5.5default · thinking on | $0.19 | 53s | 4 | Heavily guided | pass | Trust 80% | 2026-10-20 | |
| MLlama 3.2 3B (local)thinking off | $0 (local) | 11s | 0 | Fully autonomous | fail | Trust 75% | 2026-10-20 | |
| QQwen2.5 7B (local)thinking off | $0 (local) | 7m | 52 | Heavily guided | fail | Under review | 2026-10-20 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 1m | 4 | Heavily guided | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0093 | 37s | 4 | Heavily guided | pass | Trust 80% | 2026-10-20 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.