Skip to content

Fix a multi-currency brokerage ledger codebase with interacting planted bugs

Exact prompt

# Fix the multi-currency brokerage ledger The working directory contains a small ledger codebase (`solve.py` and the `ledger/` package, about 290 lines) plus sample inputs. Its output does not follow the specification below: **several bugs are planted, and some interact**. Fix the code (you may restructure it freely, Python 3 standard library only) so that `python solve.py input.json answer.json` follows this specification exactly. Hidden checks also run it on freshly generated inputs. ## Inputs `input.json`: `base` (base currency), `as_of` (valuation date), file names for `transactions`, `rates`, `prices`. CSV files may start with a UTF-8 BOM, contain `#` comment lines and blank lines (ignored), padded header names (case-insensitive) and quoted fields with commas. Transaction rows are numbered from 0 = first data row after the header, counting only non-comment, non-blank rows (`line`). - `rates.csv` `date,currency,rate`: units of base per 1 unit of currency. Not every currency has a rate every day. - `prices.csv` `date,symbol,close,currency`. - `transactions.csv` `date,type,account,amount,currency,symbol,qty,memo`. ## Rules 1. All money is exact decimal (no binary floats in calculations). "Round" means to 0.01, **half away from zero for .5 cases (ROUND_HALF_UP)**. 2. FX rate for (currency, date) = the rate with the latest date **on or before** that date (a rate dated exactly that day counts). The base currency's rate is always 1. No such rate means none. 3. Numbers: surrounding spaces and `_` are ignored; `(x)` means -x. A non-number in amount/qty -> reject `bad_number`. Empty amount/qty = 0. Empty currency = base. 4. Validation order per row, first failure wins: unknown type (case-insensitive DEPOSIT, WITHDRAW, TRANSFER, BUY, SELL, DIV, FEE) -> `bad_type`; empty account -> `bad_account`; `bad_number`; amount or qty < 0 -> `negative_value`; no FX rate for the row's currency on its date -> `no_rate`. Then the type-specific checks below. A rejected row changes nothing. Rejects are `{"line", "reason"}` in file order; `applied` counts rows not rejected. 5. Cash is kept per account **per currency** (never converted). 6. DEPOSIT adds cash. WITHDRAW: needs cash in that currency >= amount, else `insufficient_cash`. 7. TRANSFER: memo must be `to:<account>` (prefix case-insensitive, account trimmed, non-empty, different from the source) else `bad_transfer`; needs cash >= amount else `insufficient_cash`; moves the amount **in the same currency** to the destination account (which is created if new). 8. BUY: symbol (upper-cased) non-empty and qty > 0 else `bad_trade`; needs cash >= amount else `insufficient_cash`; pays `amount` (total cost) in the row currency; adds a lot of qty with cost basis = round(amount * rate on the trade date) in base. 9. SELL: symbol/qty as BUY (`bad_trade`); held qty in that account must be >= qty else `insufficient_qty`. Lots are relieved **FIFO** (oldest first). A fully used lot relieves its whole remaining cost; a partly used lot relieves round(lot_cost * take / lot_qty) and keeps the rest (so cost always sums exactly). Proceeds in base = round(amount * rate); realized gain += proceeds - relieved cost. Cash += amount in the row currency. 10. DIV: cash += amount; income += round(amount * rate). FEE: cash -= amount (may go negative); income -= round(amount * rate). 11. Valuation at `as_of`: each holding (qty > 0) uses the last close on or before `as_of` in its quote currency; value_base = round(qty * close * rate(currency, as_of)), or null if no price. Cash in base = sum over currencies of round(balance * rate(currency, as_of)). total_value_base = round(cash in base + sum of holding values). grand_total_base = sum of account totals. ## answer.json `{"accounts": {account: {"cash": {ccy: balance (non-zero only, sorted)}, "holdings": {SYMBOL: {"qty", "cost_base", "value_base"}} (qty > 0 only), "realized_base", "income_base", "total_value_base"}}, "rejects": [...], "applied": n, "grand_total_base"}`. Accounts are every account that ever had cash, lots, gains or income. Money is written as a JSON number with 2 decimals. ## Scoring Hidden checks (partial credit = fraction passed; all must pass): `rejects_and_applied`, `cash_balances`, `holdings_quantities`, `cost_basis`, `realized_gains`, `income`, `valuation` (account totals and grand total) on the given input; `fresh_run_1`, `fresh_run_2` (the whole answer) on fresh inputs. All values exact. 30-second limit per run.

Each bug is plausible code; several masks another (wrong rate date changes the same totals as rounding). Fixing only the obvious ones leaves every money total wrong. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. rejects_and_applied 2. cash_balances 3. holdings_quantities 4. cost_basis 5. realized_gains 6. income 7. valuation 8. fresh_run_1 9. fresh_run_2

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 1m).

Run this on another model

All runs (4)

100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
OGPT-6.1 Soldefault · thinking on$0.102m0Fully autonomouspassTrust 80%2026-10-22
AClaude Haiku 5.5default · thinking on$0.021m0Fully autonomouspassTrust 80%2026-10-20
AClaude Opus 5.5default · thinking on$0.381m0Fully autonomouspassTrust 80%2026-10-20
AClaude Sonnet 5.5default · thinking on$0.1346s0Fully autonomouspassTrust 80%2026-10-20

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Fix a multi-currency brokerage ledger codebase with interacting planted bugs

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.