Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Reconcile bank payments against open invoices (memo references, partial payments, exact-amount and 2-3 invoice combinations, ambiguity)

AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.23
Wall time: 38s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Reconcile bank payments against open invoices `invoices.csv` (invoice_id, customer, amount, issued) and `payments.csv` (payment_id, date, payer, amount, memo) are this quarter's exports. Write `recon.py` (Python 3 stdlib only) that, run as `python recon.py` in this directory, writes `recon.csv` and `open_invoices.csv` following these rules. We re-run it on freshly generated files, so do not hard-code anything. ## Normalisation 1. **Money:** remove `$` and `,`, strip, parse as an exact decimal (use `decimal.Decimal`, never floats). Output money with exactly 2 decimals. 2. **Names:** a payer matches a customer when their normalised names are equal. Normalise: Unicode casefold; replace every character that is not `a-z`, `0-9` or space with a space; split on whitespace; then repeatedly drop a final word if it is `ltd`, `llc`, `inc`, `gmbh` or `co`; join with single spaces. (`"Acme Tools, Ltd."` -> `acme tools`, `"Birch & Sons LLC"` -> `birch sons`.) 3. **Invoice references in memos:** in the casefolded memo, find every `inv`, optionally followed by one space or one `-`, then digits; `inv` must not be preceded by a letter or digit and the digits must not be followed by a digit. The digits (leading zeros ignored) give invoice number N, meaning invoice `INV-` + N as 4 digits. ## Processing 4. Process payments in order of (date, payment_id). Balances start at the invoice amounts and change as payments are applied; later payments see the updated balances. 5. **Rule A (reference):** collect the referenced invoices that exist, belong to the payer's customer (rule 2) and still have balance > 0, in memo order, without repeats. If there is at least one: apply the payment to them in that order, each taking `min(remaining payment, its balance)`; stop when the payment is used up (invoices it did not reach are not listed). Status `matched_ref`, `invoices` = the invoices that received money, `unapplied` = what is left (may be `0.00`). Dates are not checked in rule A. 6. Otherwise, the **candidates** are the payer's customer's invoices with balance > 0 issued on or before the payment date. 7. **Rule B (single amount):** if some candidates have balance exactly equal to the payment, close the one with the earliest issued date (ties: smaller invoice_id). Status `matched_amount`, `unapplied` `0.00`. 8. **Rule C (combination):** otherwise look at every set of exactly 2 or exactly 3 different candidates whose balances add up exactly to the payment. Exactly one such set -> close all of them, status `matched_combo`, `invoices` = their ids sorted ascending; more than one -> status `ambiguous`, no invoices, nothing applied. 9. Otherwise status `unmatched`, nothing applied. For `ambiguous` and `unmatched`, `unapplied` = the full payment amount and `invoices` is empty. ## Output 10. `recon.csv`: header `payment_id,status,invoices,unapplied`, one row per payment sorted by payment_id; `invoices` joined with `;`. 11. `open_invoices.csv`: header `invoice_id,balance`, every invoice whose final balance is > 0, sorted by invoice_id. ## Scoring Hidden checks (partial credit = fraction passed; all must pass): `ref_rows_exact`, `amount_rows_exact`, `combo_and_ambiguous_rows_exact`, `unmatched_rows_exact` (the rows whose correct status is in that group are exactly right, and no extra or missing payment rows), `open_invoices_exact`, then a re-run on fresh files: `fresh_recon_exact` (all rows) and `fresh_open_invoices_exact`. Each run has a 60-second limit.

Order-dependent balances, name normalisation with legal suffixes, memo reference parsing, issue-date filtering only for amount rules, unique-combination search and exact Decimal output; graded by outcome group plus a fresh-seed rerun. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact

Setup notes: Use Case Scout batch 8 case i07-payment-reconciliation (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-opus-5.5: $0.225854. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.23
Wall time: 38s
Human step-ins: 0
Turns: 7

Model usage
Model	Input tok	Output tok	Cost
AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on	136,542	4,293	$0.23

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. ref_rows_exact 2. amount_rows_exact 3. combo_and_ambiguous_rows_exact 4. unmatched_rows_exact 5. open_invoices_exact 6. fresh_recon_exact 7. fresh_open_invoices_exact
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 37318,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.2258538,
  "usage": {
   "input_tokens": 12,
   "cache_creation_input_tokens": 14441,
   "cache_read_input_tokens": 122089,
   "output_tokens": 4293,
   "output_tokens_details": {
    "thinking_tokens": 993
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 14441,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 738,
     "cache_read_input_tokens": 25991,
     "cache_creation_input_tokens": 596,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 596
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 12,
    "outputTokens": 4293,
    "cacheReadInputTokens": 122089,
    "cacheCreationInputTokens": 14441,
    "webSearchRequests": 0,
    "costUSD": 0.2258538,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 993,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 7,
  "subtype": "success",
  "api_error_status": null,
  "result": "I've written `recon.py` and run it against the current exports. It produced `recon.csv` with 55 rows, one per payment, and `open_invoices.csv` with 34 open invoices. I didn't have the hidden checks, so I verified it by spot-checking results and testing the name and memo-reference parsing directly.\n\n- **Results:** the 55 payments came out as 29 `matched_ref`, 6 `matched_amount`, 3 `matched_combo`, 17 `unmatched` and 0 `ambiguous`. Because nothing in this data is ambiguous, that branch hasn't been exercised by a real run yet.\n- **Spot checks:** the parsing tests passed. Name normalisation gives `\"Acme Tools, Ltd.\"` \u2192 `acme tools`, `\"Birch & Sons LLC\"` \u2192 `birch sons`, and both `\"Delta Co.\"` and `DELTA` \u2192 `delta`. Memo parsing picks up `INV-0010` and `INV 41`, accepts `inv 0007a` (a letter after the digits is allowed), and rejects `xinv5` and `inv--3`.\n- **Partial payments:** these look right. P003 paid 1,390.00 against INV-0007 (2,780.00), so 1,390.00 stays open. P028 and P036 each covered two referenced invoices and had 5.00 left over.\n\nA few small decisions the rules don't cover:\n- Invoice numbers are padded to at least 4 digits, so `inv12345` becomes `INV-12345`.\n- Dates are read as ISO `YYYY-MM-DD`, which is the only format in these files.\n- When a payer's name matches no customer, the payment comes out `unmatched` (unless it matches nothing in rule A either), since it has no candidate invoices.\n\nThe script reads everything from the input files and hard-codes nothing, so it can be re-run on fresh exports. It takes well under a second here.",
  "ttft_ms": 2105,
  "type": "result",
  "duration_ms": 37775,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1099,
  "time_to_request_ms": 258,
  "first_content_frame_ms": 1099,
  "queued_turn_count": 0,
  "result_index": 0
 }
]