Skip to content

GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking on) on Fix a flaky test suite caused by set ordering, ignored date parameter and global random state - without touching the tests

OGPT-6.1 Soldefault · thinking on · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-22 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.03
Wall time
34s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Task: fix the flaky billing tests `tests/test_billing.py` passes on some CI runs and fails on others (and fails on every run on certain days). Make `billing.py` deterministic so the suite passes on every run, on any date, under any `PYTHONHASHSEED`. Rules: - **Do not modify `tests/test_billing.py`** - the tests describe the intended behaviour. - Invoices are ordered by customer name, case-insensitively (ties broken by the exact name), and numbered in that order. - `today` defaults to the current date only when not passed. - The audit sample must be reproducible: the same `orders`, `today` and `audit_size` always give the same sample, independent of the global `random` state and of process/hash seed - and must not modify the global `random` state. It should still look random (not simply the first N invoices): different months should generally give different samples. - Keep the function signature (you may add keyword-only params with defaults). Standard library only. Run tests with `python3 -m pytest tests` if pytest is available, or call the test functions directly. No network.

Three independent sources of flakiness (hash-seeded set order, `today` overwritten, unseeded global random). The common shortcut random.seed(...) mutates global state and is caught, as is returning the first N invoices. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).

Success criteria: 1. tests_intact: tests/test_billing.py is byte-identical to the original. 2. visible_tests_all_seeds: all visible tests pass under 6 different PYTHONHASHSEED values. 3. cross_process_deterministic: a hidden 29-customer batch is identical across 4 processes with different hash seeds. 4. global_random_untouched: build_batch does not change the global random state. 5. audit_looks_random: audit samples across 12 months take at least 6 distinct values and are not always the first N invoices. 6. default_today_works: omitting today uses the current date.

Setup notes: Use Case Scout batch5 case e12-flaky-nondeterminism (offline fixture). Prepared with batch5/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the scout's scorer: `python score.py WORKDIR KEYDIR`. Single run, no retries. User-level AGENTS.md (846 bytes, sha256 83ec4fe50da8) may have been in context; --ignore-user-config does not document excluding it. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 6/6 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 4 (1 failed, recovered). Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.03
Wall time
34s
Human step-ins
0
Turns
1
Model usage
ModelInput tokOutput tokCost
OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking on81,8291,060$0.03

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking on) via Codex CLI 0.163.0 (codex exec --json), cloud.
  3. It counts as a success only if it meets every success criterion: 1. tests_intact: tests/test_billing.py is byte-identical to the original. 2. visible_tests_all_seeds: all visible tests pass under 6 different PYTHONHASHSEED values. 3. cross_process_deterministic: a hidden 29-customer batch is identical across 4 processes with different hash seeds. 4. global_random_untouched: build_batch does not change the global random state. 5. audit_looks_random: audit samples across 12 months take at least 6 distinct values and are not always the first N invoices. 6. default_today_works: omitting today uses the current date.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{"type": "thread.started", "thread_id": "01a16436-ea19-7793-8739-d93a70db7afb"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll inspect the billing code and tests, fix the sources of nondeterminism, then run the suite under several hash seeds."}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g '*billing*' -g 'pyproject.toml' -g 'pytest.ini'\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g '*billing*' -g 'pyproject.toml' -g 'pytest.ini'\"", "aggregated_output": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/170_e12/work\nbilling.py\ntests/test_billing.py\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat billing.py tests/test_billing.py'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat billing.py tests/test_billing.py'", "aggregated_output": "\"\"\"Monthly invoice batch.\"\"\"\nimport datetime, random\n\ndef build_batch(orders, today=None, audit_size=2):\n    \"\"\"orders: list of dicts {customer, amount_cents}. Returns dict with invoices and an audit sample.\"\"\"\n    today = datetime.date.today()\n    customers = set(o[\"customer\"] for o in orders)\n    invoices = []\n    for n, c in enumerate(customers, 1):\n        total = sum(o[\"amount_cents\"] for o in orders if o[\"customer\"] == c)\n        due = today + datetime.timedelta(days=30)\n        invoices.append({\"number\": f\"INV-{today:%Y%m}-{n:03d}\", \"customer\": c, \"total_cents\": total, \"due\": due.isoformat()})\n    audit = random.sample([i[\"number\"] for i in invoices], min(audit_size, len(invoices)))\n    return {\"invoices\": invoices, \"audit\": audit}\nimport datetime, random, sys, os\nsys.path.insert(0, os.path.join(os.path.dirname(__file__), \"..\"))\nfrom billing import build_batch\n\nORDERS = [{\"customer\": c, \"amount_cents\": a} for c, a in\n          [(\"zeta\", 500), (\"acme\", 1250), (\"Bravo\", 300), (\"acme\", 50), (\"delta\", 999), (\"zeta\", 1)]]\n\ndef test_numbers_follow_customer_order():\n    b = build_batch(ORDERS, today=datetime.date(2026, 3, 15))\n    assert [i[\"customer\"] for i in b[\"invoices\"]] == [\"acme\", \"Bravo\", \"delta\", \"zeta\"]\n    assert [i[\"number\"] for i in b[\"invoices\"]] == [\"INV-202603-001\", \"INV-202603-002\", \"INV-202603-003\", \"INV-202603-004\"]\n    assert [i[\"total_cents\"] for i in b[\"invoices\"]] == [1300, 300, 999, 501]\n\ndef test_due_dates():\n    b = build_batch(ORDERS, today=datetime.date(2026, 3, 15))\n    assert {i[\"due\"] for i in b[\"invoices\"]} == {\"2026-04-14\"}\n    b = build_batch(ORDERS, today=datetime.date(2026, 1, 31))\n    assert {i[\"due\"] for i in b[\"invoices\"]} == {\"2026-03-02\"}\n\ndef test_audit_is_reproducible():\n    a = build_batch(ORDERS, today=datetime.date(2026, 3, 15), audit_size=2)[\"audit\"]\n    random.random()\n    b = build_batch(ORDERS, today=datetime.date(2026, 3, 15), audit_size=2)[\"audit\"]\n    assert a == b and len(a) == 2\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/170_e12/work/billing.py", "kind": "update"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/170_e12/work/billing.py", "kind": "update"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc 'python3 -m pytest tests'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc 'python3 -m pytest tests'", "aggregated_output": "/usr/bin/python3: No module named pytest\n", "exit_code": 1, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 - <<'PY'\nimport datetime\nimport os\nimport random\nimport runpy\nimport subprocess\nimport sys\n\nsuite = runpy.run_path('tests/test_billing.py')\nfor name, test in suite.items():\n    if name.startswith('test_'):\n        test()\n        print(f'{name}: PASS')\nfrom billing import build_batch\norders = suite['ORDERS']\ndate = datetime.date(2026, 3, 15)\nstate = random.getstate()\nexpected = build_batch(orders, today=date)\nassert random.getstate() == state\ncode = \\\"import datetime,json; from billing import build_batch; orders=\\\" + repr(orders) + \\\"; print(json.dumps(build_batch(orders,today=datetime.date(2026,3,15)),sort_keys=True))\\\"\noutputs = [subprocess.check_output([sys.executable, '-c', code], env=dict(os.environ, PYTHONHASHSEED=seed)) for seed in ('0', '1', '42', 'random')]\nassert len(set(outputs)) == 1\nnames = ['acme', 'ACME', 'Acme', 'Bravo']\nassert [i['customer'] for i in build_batch([{'customer': n, 'amount_cents': 1} for n in names], today=date)['invoices']] == ['ACME', 'Acme', 'acme', 'Bravo']\nsamples = {tuple(build_batch(orders, today=datetime.date(2026, month, 15))['audit']) for month in range(1, 13)}\nassert len(samples) > 1\nassert build_batch([], today=date) == {'invoices': [], 'audit': []}\nassert build_batch(orders)['invoices'][0]['number'].startswith(f'INV-{datetime.date.today():%Y%m}-')\nprint('Additional checks: PASS (global random state, four hash seeds, name ties, monthly variation, empty batch, default date)')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 - <<'PY'\nimport datetime\nimport os\nimport random\nimport runpy\nimport subprocess\nimport sys\n\nsuite = runpy.run_path('tests/test_billing.py')\nfor name, test in suite.items():\n    if name.startswith('test_'):\n        test()\n        print(f'{name}: PASS')\nfrom billing import build_batch\norders = suite['ORDERS']\ndate = datetime.date(2026, 3, 15)\nstate = random.getstate()\nexpected = build_batch(orders, today=date)\nassert random.getstate() == state\ncode = \\\"import datetime,json; from billing import build_batch; orders=\\\" + repr(orders) + \\\"; print(json.dumps(build_batch(orders,today=datetime.date(2026,3,15)),sort_keys=True))\\\"\noutputs = [subprocess.check_output([sys.executable, '-c', code], env=dict(os.environ, PYTHONHASHSEED=seed)) for seed in ('0', '1', '42', 'random')]\nassert len(set(outputs)) == 1\nnames = ['acme', 'ACME', 'Acme', 'Bravo']\nassert [i['customer'] for i in build_batch([{'customer': n, 'amount_cents': 1} for n in names], today=date)['invoices']] == ['ACME', 'Acme', 'acme', 'Bravo']\nsamples = {tuple(build_batch(orders, today=datetime.date(2026, month, 15))['audit']) for month in range(1, 13)}\nassert len(samples) > 1\nassert build_batch([], today=date) == {'invoices': [], 'audit': []}\nassert build_batch(orders)['invoices'][0]['number'].startswith(f'INV-{datetime.date.today():%Y%m}-')\nprint('Additional checks: PASS (global random state, four hash seeds, name ties, monthly variation, empty batch, default date)')\nPY\"", "aggregated_output": "test_numbers_follow_customer_order: PASS\ntest_due_dates: PASS\ntest_audit_is_reproducible: PASS\nAdditional checks: PASS (global random state, four hash seeds, name ties, monthly variation, empty batch, default date)\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "agent_message", "text": "Fixed [billing.py](/home/bob/usecase-runs/batch/out/gpt-6.1-sol/170_e12/work/billing.py):\n\n- Sorts customers case-insensitively, with exact-name tie breaks.\n- Honors supplied `today`.\n- Uses a stable digest and private random generator for reproducible audit samples without changing global random state.\n\nTests remain unchanged. Pytest wasn\u2019t installed, so I called all three tests directly; all passed. Additional checks passed for four hash seeds, monthly variation, name ties, and global random state preservation."}}
{"type": "turn.completed", "usage": {"input_tokens": 81829, "cached_input_tokens": 75904, "cache_write_input_tokens": 0, "output_tokens": 1060, "reasoning_output_tokens": 37}}

Get new runs and ranking changes weekly