Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Reconcile bank payments against open invoices (memo references, partial payments, exact-amount and 2-3 invoice combinations, ambiguity)
AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener
- Estimated cost
- $0.01
- Wall time
- 50s
- Human step-ins
- 0
- Outcome
- Pass
Compare with other models on this task →
Summary
Exact prompt
# Reconcile bank payments against open invoices `invoices.csv` (invoice_id, customer, amount, issued) and `payments.csv` (payment_id, date, payer, amount, memo) are this quarter's exports. Write `recon.py` (Python 3 stdlib only) that, run as `python recon.py` in this directory, writes `recon.csv` and `open_invoices.csv` following these rules. We re-run it on freshly generated files, so do not hard-code anything. ## Normalisation 1. **Money:** remove `$` and `,`, strip, parse as an exact decimal (use `decimal.Decimal`, never floats). Output money with exactly 2 decimals. 2. **Names:** a payer matches a customer when their normalised names are equal. Normalise: Unicode casefold; replace every character that is not `a-z`, `0-9` or space with a space; split on whitespace; then repeatedly drop a final word if it is `ltd`, `llc`, `inc`, `gmbh` or `co`; join with single spaces. (`"Acme Tools, Ltd."` -> `acme tools`, `"Birch & Sons LLC"` -> `birch sons`.) 3. **Invoice references in memos:** in the casefolded memo, find every `inv`, optionally followed by one space or one `-`, then digits; `inv` must not be preceded by a letter or digit and the digits must not be followed by a digit. The digits (leading zeros ignored) give invoice number N, meaning invoice `INV-` + N as 4 digits. ## Processing 4. Process payments in order of (date, payment_id). Balances start at the invoice amounts and change as payments are applied; later payments see the updated balances. 5. **Rule A (reference):** collect the referenced invoices that exist, belong to the payer's customer (rule 2) and still have balance > 0, in memo order, without repeats. If there is at least one: apply the payment to them in that order, each taking `min(remaining payment, its balance)`; stop when the payment is used up (invoices it did not reach are not listed). Status `matched_ref`, `invoices` = the invoices that received money, `unapplied` = what is left (may be `0.00`). Dates are not checked in rule A. 6. Otherwise, the **candidates** are the payer's customer's invoices with balance > 0 issued on or before the payment date. 7. **Rule B (single amount):** if some candidates have balance exactly equal to the payment, close the one with the earliest issued date (ties: smaller invoice_id). Status `matched_amount`, `unapplied` `0.00`. 8. **Rule C (combination):** otherwise look at every set of exactly 2 or exactly 3 different candidates whose balances add up exactly to the payment. Exactly one such set -> close all of them, status `matched_combo`, `invoices` = their ids sorted ascending; more than one -> status `ambiguous`, no invoices, nothing applied. 9. Otherwise status `unmatched`, nothing applied. For `ambiguous` and `unmatched`, `unapplied` = the full payment amount and `invoices` is empty. ## Output 10. `recon.csv`: header `payment_id,status,invoices,unapplied`, one row per payment sorted by payment_id; `invoices` joined with `;`. 11. `open_invoices.csv`: header `invoice_id,balance`, every invoice whose final balance is > 0, sorted by invoice_id. ## Scoring Hidden checks (partial credit = fraction passed; all must pass): `ref_rows_exact`, `amount_rows_exact`, `combo_and_ambiguous_rows_exact`, `unmatched_rows_exact` (the rows whose correct status is in that group are exactly right, and no extra or missing payment rows), `open_invoices_exact`, then a re-run on fresh files: `fresh_recon_exact` (all rows) and `fresh_open_invoices_exact`. Each run has a 60-second limit.
Order-dependent balances, name normalisation with legal suffixes, memo reference parsing, issue-date filtering only for amount rules, unique-combination search and exact Decimal output; graded by outcome group plus a fresh-seed rerun. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact
Setup notes: Use Case Scout batch 8 case i07-payment-reconciliation (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.010942. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.
Estimated cost List-price estimate; not actually billed.
Metrics
- Estimated cost
- $0.01
- Wall time
- 50s
- Human step-ins
- 0
- Turns
- 6
| Model | Input tok | Output tok | Cost |
|---|---|---|---|
| AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on | 163,507 | 10,489 | $0.01 |
Reproduce this run
- Use the use case prompt on the use case page.
- Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via
Claude Code 2.1.299 (headless -p), cloud. - It counts as a success only if it meets every success criterion: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact
- Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.
How runs are scored: Methodology.
Was this useful?
Automated checks
Trust 80% comes from these checks (how Trust % works).
- Run date is not in the future : passed
- Model is in the dated pricing table : passed
- Cost recomputed from tokens matches : passed
- Cached tokens are consistent : passed
- Output throughput is plausible : passed
- Autonomy matches step-ins : passed
- Step-ins do not exceed turns : passed
- Retries agree with first-try result : passed
- Evidence log SHA-256 matches : passed
- Evidence log is substantial : passed
Evidence log
[
{
"duration_api_ms": 49243,
"stop_reason": "end_turn",
"session_id": "[REDACTED]",
"total_cost_usd": 0.01094171,
"usage": {
"input_tokens": 12,
"cache_creation_input_tokens": 21374,
"cache_read_input_tokens": 142121,
"output_tokens": 10489,
"output_tokens_details": {
"thinking_tokens": 6999
},
"server_tool_use": {
"web_search_requests": 0,
"web_fetch_requests": 0
},
"service_tier": "standard",
"cache_creation": {
"ephemeral_1h_input_tokens": 21374,
"ephemeral_5m_input_tokens": 0
},
"inference_geo": "global",
"iterations": [
{
"input_tokens": 2,
"output_tokens": 2491,
"cache_read_input_tokens": 32329,
"cache_creation_input_tokens": 1584,
"cache_creation": {
"ephemeral_5m_input_tokens": 0,
"ephemeral_1h_input_tokens": 1584
},
"type": "message"
}
],
"speed": "standard",
"fallback_credit": null
},
"modelUsage": {
"claude-haiku-5-5": {
"inputTokens": 12,
"outputTokens": 10489,
"cacheReadInputTokens": 142121,
"cacheCreationInputTokens": 21374,
"webSearchRequests": 0,
"costUSD": 0.01094171,
"contextWindow": 1000000,
"maxOutputTokens": 128000,
"thinkingTokens": 6999,
"canonicalModel": "claude-haiku-5-5",
"provider": "firstParty",
"costBasis": "list"
}
},
"permission_denials": [],
"terminal_reason": "completed",
"fast_mode_state": "off",
"fast_mode_disabled_reason": "sdk_opt_in_required",
"subagent_stats": {
"spawned": 0,
"requested": {
"background": 0,
"foreground": 0,
"unset": 0
},
"started_in_background": 0,
"max_depth": 0,
"spawned_by_subagents": 0,
"completed": 0,
"failed": 0,
"killed": {
"parent": 0,
"user": 0,
"system": 0
},
"refused": {
"depth_limit": 0,
"concurrency_limit": 0,
"budget": 0
},
"by_type": {}
},
"safety_stops": 0,
"is_error": false,
"num_turns": 6,
"subtype": "success",
"api_error_status": null,
"result": "I wrote `recon.py` and ran it. It produced `recon.csv` and `open_invoices.csv` in this directory.\n\nOne thing to know: this machine has no `python` binary, only `python3`, so `python recon.py` fails with \"command not found\". I ran it as `python3 recon.py`. The script itself is stdlib-only and has no version-specific code. If your re-run environment uses `python`, it should work unchanged.\n\nResults: 55 payments, 42 open invoices. Status counts are 27 `matched_ref`, 6 `matched_amount`, 3 `matched_combo`, and 19 `unmatched`. There are no `ambiguous` rows in this data. Each `matched_ref` row also has a non-zero `unapplied` only where the payment exceeded the referenced balances (P024, P028, P036).\n\nChoices I made where the spec left room:\n- **Name matching:** the suffix-stripping loop removes trailing `ltd`, `llc`, `inc`, `gmbh`, `co` only, as written. \"Cobalt\" matches \"Cobalt GmbH\", and \"Birch Sons\" matches \"Birch & Sons LLC\".\n- **Memo regex:** `(?<![^\\W_])inv[ -]?([0-9]+)(?![0-9])` on the casefolded memo. The lookbehind excludes letters and digits but not underscore, which is what the spec says.\n- **Money parsing:** removes `$` and `,`, then strips surrounding whitespace. Outputs use `ROUND_HALF_UP` to 2 decimals. The inputs already have 2 decimals, so no rounding actually happens.\n- **Rule C counting:** 2-sets and 3-sets are counted together. More than one hit across both sizes gives `ambiguous`.\n- **Output:** `invoices` for `matched_ref` keeps memo order, and for `matched_combo` it is sorted ascending. Rows are sorted by `payment_id`. Open invoices are sorted by `invoice_id`.\n\nI spot-checked several rows by hand: P013 (2109.01 + 218.11 = 2327.12), P028 (2585.94 + 269.06 + 5.00 unapplied), P024 (INV-0038 already closed by P036, so only INV-0042 applies), and P004 (878.79 leaves 0.01 on INV-0026). I did not hand-verify the full run. The unmatched rows, such as P001, are the ones I'd look at first if the hidden checks disagree.\n\nThe directory is not a git repo, so I did not commit anything. Files are in `./usecase-runner/runs/i07-claude-haiku/work/`: `recon.py`, `recon.csv`, `open_invoices.csv`.",
"ttft_ms": 1720,
"type": "result",
"duration_ms": 49662,
"uuid": "[REDACTED]",
"ttft_stream_ms": 1643,
"time_to_request_ms": 247,
"first_content_frame_ms": 1644,
"queued_turn_count": 0,
"result_index": 0
}
]