Fix a flaky test suite caused by set ordering, ignored date parameter and global random state - without touching the tests
Exact prompt
# Task: fix the flaky billing tests `tests/test_billing.py` passes on some CI runs and fails on others (and fails on every run on certain days). Make `billing.py` deterministic so the suite passes on every run, on any date, under any `PYTHONHASHSEED`. Rules: - **Do not modify `tests/test_billing.py`** - the tests describe the intended behaviour. - Invoices are ordered by customer name, case-insensitively (ties broken by the exact name), and numbered in that order. - `today` defaults to the current date only when not passed. - The audit sample must be reproducible: the same `orders`, `today` and `audit_size` always give the same sample, independent of the global `random` state and of process/hash seed - and must not modify the global `random` state. It should still look random (not simply the first N invoices): different months should generally give different samples. - Keep the function signature (you may add keyword-only params with defaults). Standard library only. Run tests with `python3 -m pytest tests` if pytest is available, or call the test functions directly. No network.
Three independent sources of flakiness (hash-seeded set order, `today` overwritten, unseeded global random). The common shortcut random.seed(...) mutates global state and is caught, as is returning the first N invoices. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. tests_intact: tests/test_billing.py is byte-identical to the original. 2. visible_tests_all_seeds: all visible tests pass under 6 different PYTHONHASHSEED values. 3. cross_process_deterministic: a hidden 29-customer batch is identical across 4 processes with different hash seeds. 4. global_random_untouched: build_batch does not change the global random state. 5. audit_looks_random: audit samples across 12 months take at least 6 distinct values and are not always the first N invoices. 6. default_today_works: omitting today uses the current date.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0050, 20s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.03 | 34s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.0050 | 20s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.12 | 19s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.05 | 12s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.