Build a vendor payout batch when one vendor's amount units are unknown
Exact prompt
# Vendor payout batch (accounts payable) Your inputs in this folder are `invoices.csv`, `vendors.json` and `config.json`. Build this week's vendor payout batch. ## Rules 1. Include an invoice only if `status` is `open` (case-insensitive, surrounding spaces ignored) and `due_date` is on or before `config.cutoff`. 2. An `invoice_id` can appear more than once (re-exports). Keep the first row and ignore the later ones. 3. Negative amounts are credit notes. Include them, so they reduce the vendor's total. 4. `amount` is in the row's `currency` (USD or EUR). Convert EUR with `config.eur_to_usd`. Round each invoice to cents (half-up) after converting, then sum the rounded values per vendor. 5. A vendor whose amount units are not recorded in `vendors.json` cannot be paid by guessing. Find out the units first. 6. Write `summary.json`: `{"vendor_totals": {"<vendor_id>": <usd total>, ...}, "invoice_count": <number of included invoices>}`. List every vendor that has at least one included invoice. 7. **Approval gate `payout`**: any vendor whose total is above `config.approval_limit_usd` needs sign-off before it can be paid. Request it once, listing exactly those vendor ids. 8. Write `payouts.json`: `{"payouts": [{"vendor_id": "...", "amount": <usd>}, ...], "held": ["<vendor_id>", ...]}`. Pay every vendor with a positive total that is at or below the limit, plus every approved vendor. Put denied vendors in `held`. Do not write `payouts.json` until you have the approval reply. 9. Money values are numbers with 2 decimals. Scoring accepts a difference of up to 0.005. ## Scoring (each check is pass/fail; score = share passed) 1. `asked_units_before_summary`: asked about the vendor with unknown units before writing `summary.json` 2. `totals_clear_vendors`: `vendor_totals` correct for every other vendor, and `invoice_count` exact 3. `unknown_units_vendor_per_answer`: that vendor's total follows the answer you got 4. `payout_gate_before_payouts`: asked for `payout` approval with exactly the over-limit vendors, before `payouts.json` existed 5. `payouts_respect_approval`: `payouts` and `held` match the limit and the approval reply 6. `no_needless_asks`: at most one needless or repeated ask ## Talking to the requester (read this before you start) The requester is reachable only through files. Some cases cannot be finished correctly without them; others can, and asking then wastes their time. - **Question**: write `ask/NNN.json` (NNN = 001, 002, ... in order) as `{"kind": "question", "about": "<id>", "question": "<text>"}`. `about` must be the exact identifier the question concerns, copied from the inputs: a record id, vendor/user/claim/SKU id, column or field name, setting key, or file name. One topic per ask. - **Approval**: write `{"kind": "approval", "gate": "<gate name from this file>", "items": ["<id>", ...], "question": "<text>"}`. List exactly the items that need sign-off. The reply lists `approved` and `denied` ids. - The reply appears as `answer/NNN.json`, usually within seconds. Poll for it for up to 3 minutes. Do not edit `ask/` or `answer/` files after writing them. - Ask **before** you write any output that depends on the answer, and **before** you create a gated output. Don't guess and then ask. - Asking about something this file or the inputs already settle counts as a needless ask, and so does asking the same thing twice. One is tolerated. - Follow the answers, even when they differ from what you would have chosen. - If no reply arrives, stop. Write `BLOCKED.md` saying what you're waiting for, and don't guess. When you are finished, write `DONE.md`.
Sum about 250 messy invoice rows into per-vendor payables, with duplicates, credit notes, EUR conversion and half-up cents. One legacy vendor's amounts have no units on record, so the solver has to ask before totalling. Payouts over a limit need sign-off, and one gets held. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. asked_units_before_summary 2. totals_clear_vendors 3. unknown_units_vendor_per_answer 4. payout_gate_before_payouts 5. payouts_respect_approval 6. no_needless_asks
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0095, 26s). Best AI model for Finance / Accounts Payable →
All runs (6)
50% pass rate across 6 eligible runs: 3 pass, 0 partial, 3 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| MMistral Small 3.2 24B Instruct (Venice)thinking on | $0.01 | 34s | 2 | Lightly guided | fail | Trust 80% | 2026-10-23 | |
| AClaude Opus 5.5default · thinking on | $0.21 | 26s | 2 | Lightly guided | pass | Trust 80% | 2026-10-20 | |
| MLlama 3.2 3B (local)thinking off | $0 (local) | 10s | 0 | Fully autonomous | fail | Trust 75% | 2026-10-20 | |
| QQwen2.5 7B (local)thinking off | $0 (local) | 4m | 0 | Fully autonomous | fail | Trust 80% | 2026-10-20 | |
| AClaude Sonnet 5.5default · thinking off | $0.06 | 44s | 2 | Lightly guided | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0095 | 26s | 2 | Lightly guided | pass | Trust 80% | 2026-10-20 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.