Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Reconcile bank payments against open invoices (memo references, partial payments, exact-amount and 2-3 invoice combinations, ambiguity)

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.01
Wall time
50s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Reconcile bank payments against open invoices `invoices.csv` (invoice_id, customer, amount, issued) and `payments.csv` (payment_id, date, payer, amount, memo) are this quarter's exports. Write `recon.py` (Python 3 stdlib only) that, run as `python recon.py` in this directory, writes `recon.csv` and `open_invoices.csv` following these rules. We re-run it on freshly generated files, so do not hard-code anything. ## Normalisation 1. **Money:** remove `$` and `,`, strip, parse as an exact decimal (use `decimal.Decimal`, never floats). Output money with exactly 2 decimals. 2. **Names:** a payer matches a customer when their normalised names are equal. Normalise: Unicode casefold; replace every character that is not `a-z`, `0-9` or space with a space; split on whitespace; then repeatedly drop a final word if it is `ltd`, `llc`, `inc`, `gmbh` or `co`; join with single spaces. (`"Acme Tools, Ltd."` -> `acme tools`, `"Birch & Sons LLC"` -> `birch sons`.) 3. **Invoice references in memos:** in the casefolded memo, find every `inv`, optionally followed by one space or one `-`, then digits; `inv` must not be preceded by a letter or digit and the digits must not be followed by a digit. The digits (leading zeros ignored) give invoice number N, meaning invoice `INV-` + N as 4 digits. ## Processing 4. Process payments in order of (date, payment_id). Balances start at the invoice amounts and change as payments are applied; later payments see the updated balances. 5. **Rule A (reference):** collect the referenced invoices that exist, belong to the payer's customer (rule 2) and still have balance > 0, in memo order, without repeats. If there is at least one: apply the payment to them in that order, each taking `min(remaining payment, its balance)`; stop when the payment is used up (invoices it did not reach are not listed). Status `matched_ref`, `invoices` = the invoices that received money, `unapplied` = what is left (may be `0.00`). Dates are not checked in rule A. 6. Otherwise, the **candidates** are the payer's customer's invoices with balance > 0 issued on or before the payment date. 7. **Rule B (single amount):** if some candidates have balance exactly equal to the payment, close the one with the earliest issued date (ties: smaller invoice_id). Status `matched_amount`, `unapplied` `0.00`. 8. **Rule C (combination):** otherwise look at every set of exactly 2 or exactly 3 different candidates whose balances add up exactly to the payment. Exactly one such set -> close all of them, status `matched_combo`, `invoices` = their ids sorted ascending; more than one -> status `ambiguous`, no invoices, nothing applied. 9. Otherwise status `unmatched`, nothing applied. For `ambiguous` and `unmatched`, `unapplied` = the full payment amount and `invoices` is empty. ## Output 10. `recon.csv`: header `payment_id,status,invoices,unapplied`, one row per payment sorted by payment_id; `invoices` joined with `;`. 11. `open_invoices.csv`: header `invoice_id,balance`, every invoice whose final balance is > 0, sorted by invoice_id. ## Scoring Hidden checks (partial credit = fraction passed; all must pass): `ref_rows_exact`, `amount_rows_exact`, `combo_and_ambiguous_rows_exact`, `unmatched_rows_exact` (the rows whose correct status is in that group are exactly right, and no extra or missing payment rows), `open_invoices_exact`, then a re-run on fresh files: `fresh_recon_exact` (all rows) and `fresh_open_invoices_exact`. Each run has a 60-second limit.

Order-dependent balances, name normalisation with legal suffixes, memo reference parsing, issue-date filtering only for amount rules, unique-combination search and exact Decimal output; graded by outcome group plus a fresh-seed rerun. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact

Setup notes: Use Case Scout batch 8 case i07-payment-reconciliation (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.010942. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.01
Wall time
50s
Human step-ins
0
Turns
6
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on163,50710,489$0.01

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 49243,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.01094171,
  "usage": {
   "input_tokens": 12,
   "cache_creation_input_tokens": 21374,
   "cache_read_input_tokens": 142121,
   "output_tokens": 10489,
   "output_tokens_details": {
    "thinking_tokens": 6999
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 21374,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 2491,
     "cache_read_input_tokens": 32329,
     "cache_creation_input_tokens": 1584,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 1584
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 12,
    "outputTokens": 10489,
    "cacheReadInputTokens": 142121,
    "cacheCreationInputTokens": 21374,
    "webSearchRequests": 0,
    "costUSD": 0.01094171,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 6999,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 6,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `recon.py` and ran it. It produced `recon.csv` and `open_invoices.csv` in this directory.\n\nOne thing to know: this machine has no `python` binary, only `python3`, so `python recon.py` fails with \"command not found\". I ran it as `python3 recon.py`. The script itself is stdlib-only and has no version-specific code. If your re-run environment uses `python`, it should work unchanged.\n\nResults: 55 payments, 42 open invoices. Status counts are 27 `matched_ref`, 6 `matched_amount`, 3 `matched_combo`, and 19 `unmatched`. There are no `ambiguous` rows in this data. Each `matched_ref` row also has a non-zero `unapplied` only where the payment exceeded the referenced balances (P024, P028, P036).\n\nChoices I made where the spec left room:\n- **Name matching:** the suffix-stripping loop removes trailing `ltd`, `llc`, `inc`, `gmbh`, `co` only, as written. \"Cobalt\" matches \"Cobalt GmbH\", and \"Birch Sons\" matches \"Birch & Sons LLC\".\n- **Memo regex:** `(?<![^\\W_])inv[ -]?([0-9]+)(?![0-9])` on the casefolded memo. The lookbehind excludes letters and digits but not underscore, which is what the spec says.\n- **Money parsing:** removes `$` and `,`, then strips surrounding whitespace. Outputs use `ROUND_HALF_UP` to 2 decimals. The inputs already have 2 decimals, so no rounding actually happens.\n- **Rule C counting:** 2-sets and 3-sets are counted together. More than one hit across both sizes gives `ambiguous`.\n- **Output:** `invoices` for `matched_ref` keeps memo order, and for `matched_combo` it is sorted ascending. Rows are sorted by `payment_id`. Open invoices are sorted by `invoice_id`.\n\nI spot-checked several rows by hand: P013 (2109.01 + 218.11 = 2327.12), P028 (2585.94 + 269.06 + 5.00 unapplied), P024 (INV-0038 already closed by P036, so only INV-0042 applies), and P004 (878.79 leaves 0.01 on INV-0026). I did not hand-verify the full run. The unmatched rows, such as P001, are the ones I'd look at first if the hidden checks disagree.\n\nThe directory is not a git repo, so I did not commit anything. Files are in `./usecase-runner/runs/i07-claude-haiku/work/`: `recon.py`, `recon.csv`, `open_invoices.csv`.",
  "ttft_ms": 1720,
  "type": "result",
  "duration_ms": 49662,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1643,
  "time_to_request_ms": 247,
  "first_content_frame_ms": 1644,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly