GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking on) on Fix a lock-ordering deadlock, a lost-update race and torn snapshots in a threaded bank ledger
OGPT-6.1 Soldefault · thinking on · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-22 · by @bobsiopener
- Estimated cost
- $0.06
- Wall time
- 1m
- Human step-ins
- 0
- Outcome
- Pass
Compare with other models on this task →
Summary
Exact prompt
# Task: fix the concurrency bugs in bank.py `bank.py` is called from many worker threads at once. In production the worker pool sometimes hangs forever, money occasionally appears or disappears (the bank total drifts), balances have gone negative, and auditors say `total()`/`snapshot()` sometimes report a state that never existed. Fix `bank.py` so that, under heavy concurrent use: - `transfer` can never deadlock, never loses or creates money, never makes a balance negative, and raises `InsufficientFunds` (leaving everything unchanged) when the source lacks funds; - `total()` and `snapshot()` return a consistent point-in-time view even while transfers are running; - every successful transfer is in `log` exactly once, failed ones not at all. Keep the public API (class names, method names/signatures, `bal` and `log` attributes). Do not make it single-threaded by removing threading support from callers; a global lock is acceptable only if it keeps throughput reasonable (the grader runs 8 threads x 20,000 transfers in under 30 s). Standard library only. No network.
Three distinct bugs: check-then-act outside the lock (lost updates/negatives), opposite lock order (deadlock), unlocked snapshot/total (torn reads). Fixing one exposes the others; the stress test forces frequent thread switches. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).
Success criteria: 1. no_deadlock: each of 3 stress runs (8 threads x 20,000 random transfers plus a concurrent auditor) finishes within 30 s. 2. final_total_ok: total money is conserved in every run. 3. no_negative: no balance is ever negative (final state and every audited snapshot). 4. log_exact: log has exactly one entry per successful transfer. 5. snapshots_consistent: every snapshot()/total() taken during the run sums to the true total. 6. nsf_raises_unchanged: an overdraft raises InsufficientFunds and changes neither balances nor log.
Setup notes: Use Case Scout batch5 case e03-bank-deadlock-race (offline fixture). Prepared with batch5/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the scout's scorer: `python score.py WORKDIR KEYDIR`. Single run, no retries. User-level AGENTS.md (846 bytes, sha256 83ec4fe50da8) may have been in context; --ignore-user-config does not document excluding it. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 6/6 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 4 (1 failed, recovered). Single run.
Estimated cost List-price estimate; not actually billed.
Metrics
- Estimated cost
- $0.06
- Wall time
- 1m
- Human step-ins
- 0
- Turns
- 1
| Model | Input tok | Output tok | Cost |
|---|---|---|---|
| OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking on | 100,157 | 1,957 | $0.06 |
Reproduce this run
- Use the use case prompt on the use case page.
- Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking on) via
Codex CLI 0.163.0 (codex exec --json), cloud. - It counts as a success only if it meets every success criterion: 1. no_deadlock: each of 3 stress runs (8 threads x 20,000 random transfers plus a concurrent auditor) finishes within 30 s. 2. final_total_ok: total money is conserved in every run. 3. no_negative: no balance is ever negative (final state and every audited snapshot). 4. log_exact: log has exactly one entry per successful transfer. 5. snapshots_consistent: every snapshot()/total() taken during the run sums to the true total. 6. nsf_raises_unchanged: an overdraft raises InsufficientFunds and changes neither balances nor log.
- Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.
How runs are scored: Methodology.
Was this useful?
Automated checks
Trust 80% comes from these checks (how Trust % works).
- Run date is not in the future : passed
- Model is in the dated pricing table : passed
- Cost recomputed from tokens matches : passed
- Cached tokens are consistent : passed
- Output throughput is plausible : passed
- Autonomy matches step-ins : passed
- Step-ins do not exceed turns : passed
- Retries agree with first-try result : passed
- Evidence log SHA-256 matches : passed
- Evidence log is substantial : passed
Evidence log
{"type": "thread.started", "thread_id": "01a1642e-8656-7b93-a261-80d3d5d3aee8"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll inspect `bank.py` and the local tests, fix the locking, then check correctness and throughput under concurrent load.\n"}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd; rg --files -g 'AGENTS.md' -g '*bank*' -g '*test*' -g 'pyproject.toml'\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd; rg --files -g 'AGENTS.md' -g '*bank*' -g '*test*' -g 'pyproject.toml'\"", "aggregated_output": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/162_e03/work\nbank.py\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat bank.py; ls -la'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat bank.py; ls -la'", "aggregated_output": "\"\"\"In-memory bank used by the payments worker pool (many threads call transfer concurrently).\"\"\"\nimport threading\n\nclass InsufficientFunds(Exception):\n pass\n\nclass Bank:\n def __init__(self, balances):\n self.bal = dict(balances)\n self.locks = {k: threading.Lock() for k in self.bal}\n self.log = []\n\n def _record(self, src, dst, amt):\n self.log.append(f\"{src}->{dst}:{amt}\")\n\n def transfer(self, src, dst, amt):\n if amt <= 0:\n raise ValueError(\"amount must be positive\")\n if src == dst:\n return\n current = self.bal[src]\n if current < amt:\n raise InsufficientFunds(src)\n with self.locks[src]:\n with self.locks[dst]:\n self._record(src, dst, amt)\n self.bal[src] = current - amt\n self.bal[dst] = self.bal[dst] + amt\n\n def total(self):\n return sum(self.bal.values())\n\n def snapshot(self):\n return dict(self.bal)\ntotal 16\ndrwxr-xr-x 6 bob bob 4096 Oct 22 11:15 .\ndrwxrwxr-x 3 bob bob 4096 Oct 22 11:15 ..\ndr-xr-xr-x 2 bob bob 40 Oct 22 11:15 .agents\ndr-xr-xr-x 2 bob bob 40 Oct 22 11:15 .aws\ndr-xr-xr-x 2 bob bob 40 Oct 22 11:15 .codex\ndr-xr-xr-x 2 bob bob 40 Oct 22 11:15 .git\n-rw-r--r-- 1 bob bob 1077 Oct 22 10:08 TASK.md\n-rw-r--r-- 1 bob bob 989 Oct 22 10:08 bank.py\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "agent_message", "text": "The balance check runs before locking, opposite-direction transfers can deadlock, and readers can see a partially applied transfer. I\u2019ll use one lock for transfers, logging, and reads, then benchmark it against the required workload. Self-transfers will check funds and log once, as required for every successful transfer.\n"}}
{"type": "item.started", "item": {"id": "item_4", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/162_e03/work/bank.py", "kind": "update"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/162_e03/work/bank.py", "kind": "update"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"cat > test_bank.py <<'PY'\nimport random\nimport threading\nimport time\nimport unittest\nfrom concurrent.futures import ThreadPoolExecutor\n\nfrom bank import Bank, InsufficientFunds\n\n\nclass BankTests(unittest.TestCase):\n def test_failed_transfers_leave_state_unchanged(self):\n bank = Bank({'a': 5, 'b': 0})\n for args, error in [\n (('a', 'b', 6), InsufficientFunds),\n (('a', 'a', 6), InsufficientFunds),\n (('a', 'b', 0), ValueError),\n (('a', 'b', -1), ValueError),\n (('a', 'missing', 1), KeyError),\n ]:\n with self.assertRaises(error):\n bank.transfer(*args)\n self.assertEqual(bank.snapshot(), {'a': 5, 'b': 0})\n self.assertEqual(bank.log, [])\n bank.transfer('a', 'a', 5)\n self.assertEqual(bank.snapshot(), {'a': 5, 'b': 0})\n self.assertEqual(bank.log, ['a->a:5'])\n\n def test_competing_withdrawals(self):\n bank = Bank({'a': 1, 'b': 0})\n barrier = threading.Barrier(8)\n\n def withdraw(_):\n barrier.wait()\n try:\n bank.transfer('a', 'b', 1)\n return 1\n except InsufficientFunds:\n return 0\n\n with ThreadPoolExecutor(8) as pool:\n self.assertEqual(sum(pool.map(withdraw, range(8))), 1)\n self.assertEqual(bank.snapshot(), {'a': 0, 'b': 1})\n self.assertEqual(bank.log, ['a->b:1'])\n\n def test_concurrent_transfers_and_point_in_time_reads(self):\n bank = Bank({i: 100 for i in range(8)})\n barrier = threading.Barrier(9)\n snapshots = []\n errors = []\n successes = [0] * 8\n stop = threading.Event()\n\n def worker(index):\n try:\n rng = random.Random(index)\n barrier.wait()\n for _ in range(20_000):\n src, dst = rng.sample(range(8), 2)\n try:\n bank.transfer(src, dst, rng.randint(1, 10))\n successes[index] += 1\n except InsufficientFunds:\n pass\n except BaseException as exc:\n errors.append(exc)\n\n threads = [threading.Thread(target=worker, args=(i,), daemon=True)\n for i in range(8)]\n start = time.monotonic()\n for thread in threads:\n thread.start()\n barrier.wait()\n deadline = start + 30\n while any(thread.is_alive() for thread in threads):\n self.assertLess(time.monotonic(), deadline, 'workers hung or too slow')\n snapshots.append(tuple(bank.snapshot().values()))\n self.assertEqual(bank.total(), 800)\n time.sleep(0.001)\n for thread in threads:\n thread.join()\n self.assertFalse(errors)\n self.assertEqual(len(bank.log), sum(successes))\n balances = [100] * 8\n observed_states = {tuple(balances)}\n for entry in bank.log:\n route, amount = entry.split(':')\n src, dst = map(int, route.split('->'))\n amount = int(amount)\n self.assertGreaterEqual(balances[src], amount)\n balances[src] -= amount\n balances[dst] += amount\n observed_states.add(tuple(balances))\n self.assertEqual(tuple(bank.snapshot().values()), tuple(balances))\n self.assertTrue(all(state in observed_states for state in snapshots))\n self.assertEqual(sum(balances), 800)\n print(f'160,000 concurrent transfer attempts: {time.monotonic() - start:.2f}s')\n\n\nif __name__ == '__main__':\n unittest.main()\nPY\npython -m unittest -v test_bank.py\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"cat > test_bank.py <<'PY'\nimport random\nimport threading\nimport time\nimport unittest\nfrom concurrent.futures import ThreadPoolExecutor\n\nfrom bank import Bank, InsufficientFunds\n\n\nclass BankTests(unittest.TestCase):\n def test_failed_transfers_leave_state_unchanged(self):\n bank = Bank({'a': 5, 'b': 0})\n for args, error in [\n (('a', 'b', 6), InsufficientFunds),\n (('a', 'a', 6), InsufficientFunds),\n (('a', 'b', 0), ValueError),\n (('a', 'b', -1), ValueError),\n (('a', 'missing', 1), KeyError),\n ]:\n with self.assertRaises(error):\n bank.transfer(*args)\n self.assertEqual(bank.snapshot(), {'a': 5, 'b': 0})\n self.assertEqual(bank.log, [])\n bank.transfer('a', 'a', 5)\n self.assertEqual(bank.snapshot(), {'a': 5, 'b': 0})\n self.assertEqual(bank.log, ['a->a:5'])\n\n def test_competing_withdrawals(self):\n bank = Bank({'a': 1, 'b': 0})\n barrier = threading.Barrier(8)\n\n def withdraw(_):\n barrier.wait()\n try:\n bank.transfer('a', 'b', 1)\n return 1\n except InsufficientFunds:\n return 0\n\n with ThreadPoolExecutor(8) as pool:\n self.assertEqual(sum(pool.map(withdraw, range(8))), 1)\n self.assertEqual(bank.snapshot(), {'a': 0, 'b': 1})\n self.assertEqual(bank.log, ['a->b:1'])\n\n def test_concurrent_transfers_and_point_in_time_reads(self):\n bank = Bank({i: 100 for i in range(8)})\n barrier = threading.Barrier(9)\n snapshots = []\n errors = []\n successes = [0] * 8\n stop = threading.Event()\n\n def worker(index):\n try:\n rng = random.Random(index)\n barrier.wait()\n for _ in range(20_000):\n src, dst = rng.sample(range(8), 2)\n try:\n bank.transfer(src, dst, rng.randint(1, 10))\n successes[index] += 1\n except InsufficientFunds:\n pass\n except BaseException as exc:\n errors.append(exc)\n\n threads = [threading.Thread(target=worker, args=(i,), daemon=True)\n for i in range(8)]\n start = time.monotonic()\n for thread in threads:\n thread.start()\n barrier.wait()\n deadline = start + 30\n while any(thread.is_alive() for thread in threads):\n self.assertLess(time.monotonic(), deadline, 'workers hung or too slow')\n snapshots.append(tuple(bank.snapshot().values()))\n self.assertEqual(bank.total(), 800)\n time.sleep(0.001)\n for thread in threads:\n thread.join()\n self.assertFalse(errors)\n self.assertEqual(len(bank.log), sum(successes))\n balances = [100] * 8\n observed_states = {tuple(balances)}\n for entry in bank.log:\n route, amount = entry.split(':')\n src, dst = map(int, route.split('->'))\n amount = int(amount)\n self.assertGreaterEqual(balances[src], amount)\n balances[src] -= amount\n balances[dst] += amount\n observed_states.add(tuple(balances))\n self.assertEqual(tuple(bank.snapshot().values()), tuple(balances))\n self.assertTrue(all(state in observed_states for state in snapshots))\n self.assertEqual(sum(balances), 800)\n print(f'160,000 concurrent transfer attempts: {time.monotonic() - start:.2f}s')\n\n\nif __name__ == '__main__':\n unittest.main()\nPY\npython -m unittest -v test_bank.py\"", "aggregated_output": "/bin/bash: line 103: python: command not found\n", "exit_code": 127, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_6", "type": "command_execution", "command": "/bin/bash -lc \"sed -i '/ stop = threading.Event()/d' test_bank.py\npython3 -m unittest -v test_bank.py\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "command_execution", "command": "/bin/bash -lc \"sed -i '/ stop = threading.Event()/d' test_bank.py\npython3 -m unittest -v test_bank.py\"", "aggregated_output": "test_competing_withdrawals (test_bank.BankTests.test_competing_withdrawals) ... ok\ntest_concurrent_transfers_and_point_in_time_reads (test_bank.BankTests.test_concurrent_transfers_and_point_in_time_reads) ... ok\ntest_failed_transfers_leave_state_unchanged (test_bank.BankTests.test_failed_transfers_leave_state_unchanged) ... ok\n\n----------------------------------------------------------------------\nRan 3 tests in 0.460s\n\nOK\n160,000 concurrent transfer attempts: 0.44s\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_7", "type": "agent_message", "text": "Fixed `bank.py` with shared locking around transfers, logging, `total()`, and `snapshot()`. Funds checks now happen inside the lock, preventing overdrafts and lost updates. Successful self-transfers check funds and log once.\n\nAdded tests for competing withdrawals, unchanged state after failures, and snapshot consistency against log replay.\n\nAll 3 tests pass. Eight threads completed 160,000 transfer attempts in **0.44 seconds**, below the 30-second limit."}}
{"type": "turn.completed", "usage": {"input_tokens": 100157, "cached_input_tokens": 86656, "cache_write_input_tokens": 0, "output_tokens": 1957, "reasoning_output_tokens": 179}}