Skip to content

GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking off) on Seeded Monte Carlo retirement odds against a stated success threshold

OGPT-6.1 Soldefault · thinking off · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.04
Wall time
1m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Seeded Monte Carlo retirement odds `input.json` holds the assumptions for a Monte Carlo retirement simulation (synthetic). Write `solve.py` (Python 3 standard library only) so that `python solve.py input.json answer.json` runs the simulation exactly as specified and writes the statistics. Because the random sequence is fixed by the seed, the answer is exact. We re-run it on fresh inputs, so do not hard-code anything. ## Input `seed`, `sims`, `years` (integers), `start_balance`, `annual_spend` (first-year spending), `mean_return`, `sd_return`, `mean_inflation`, `sd_inflation`, `success_threshold` (fractions). ## Rules 1. Create one generator `rng = random.Random(seed)` and use it for everything, in this order. Run simulations s = 0 .. sims-1 one after another. 2. Each simulation starts with balance = `start_balance` and spend = `annual_spend` (as floats). For each year y = 0 .. years-1: draw `r = rng.gauss(mean_return, sd_return)`, then `i = rng.gauss(mean_inflation, sd_inflation)` (both are drawn every year, in that order, before anything else), then balance -= spend. 3. If the balance is now below 0, the simulation **fails** in year y+1: its ending balance is 0 and it stops immediately (no more draws for it). Otherwise balance *= (1 + r) and spend *= (1 + i). 4. A simulation that never fails succeeds; its ending balance is the balance after the last year. 5. `success_rate` = successes / sims (round to 6 decimals). `meets_threshold` = (unrounded success rate >= `success_threshold`). `sims_failed` = number of failures. 6. Percentiles: sort all ending balances ascending (failed ones count as 0) into E; `pX` = E[int(X/100 * (sims - 1))] for X = 10, 50, 90 (round to 2 decimals). 7. `median_failure_year`: sort the failure years ascending into F and take F[int(0.5 * (len(F) - 1))]; `null` if nothing failed. 8. `best_sim` = index of the highest ending balance (lowest index on ties). `earliest_failure_sim` = index of the simulation that failed in the earliest year (lowest index on ties); `null` if nothing failed. ## Output `answer.json`: an object with keys `success_rate`, `meets_threshold`, `sims_failed`, `p10`, `p50`, `p90`, `median_failure_year`, `best_sim`, `earliest_failure_sim`. ## Chart (required visual) Also write `chart.svg` in the working directory: a static, self-contained SVG (root element `<svg>`, no scripts, no external images, fonts, stylesheets or links) that can be rendered headless. It must contain a percentile fan: `<polyline id="p10">`, `<polyline id="p50">`, `<polyline id="p90">`, each the 10th/50th/90th percentile end-of-year balance across simulations for each year (a failed simulation counts as 0 from its failure year). ## Scoring The `chart_svg_static` check (one more check, counted in partial credit) passes only if `chart.svg` exists after the fixture run, parses, has root `<svg>`, contains the ids above with the stated tags, and has no `<script>`, `<img>`/`<image>`, `<link>`, `<iframe>`/`<object>`/`<embed>`, `on*` handlers, or `href`/`src` pointing anywhere except `#...` or `data:`. Its drawing is not graded. Hidden checks (partial credit = fraction passed; all must pass): `success_rate_and_threshold` (rate within 0.000001, the other two exact), `percentiles` (within 0.011), `failure_stats` (exact), `best_sim` (exact) on the given input; `fresh_run_1` and `fresh_run_2`: every key on a fresh input (money within 0.011, rates within 0.011, counts and indexes exact). Each run has a 120-second limit.

With a fixed seed the success rate is exact only if the draw order (return then inflation, every year, stop on failure) and the percentile/tie rules are followed exactly; approximate simulations score near zero. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).

These are synthetic test cases for comparing AI models, not financial advice.

Success criteria: 1. success_rate_and_threshold 2. percentiles 3. failure_stats 4. best_sim 5. fresh_run_1 6. fresh_run_2 7. chart_svg_static

Setup notes: Batch 9 case j02-retirement-monte-carlo (offline fixture). Prepared with batch9/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the case scorer using absolute paths: `python score.py WORKDIR KEYDIR`. Single run, no retries. A global user-level AGENTS.md was present on the agent machine and may have been in context. The served model id is not echoed in the event log. Visual output (chart.svg) rendered offline to PNG with google-chrome headless, network blocked, 1280x800 at ~3 s after load. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 5 (1 failed, recovered). Single run.

Estimated cost List-price estimate; not actually billed.

Output

Metrics

Estimated cost
$0.04
Wall time
1m
Human step-ins
0
Turns
1
Model usage
ModelInput tokOutput tokCost
OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking off65,7481,760$0.04

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking off) via Codex CLI 0.163.0 (codex exec --json), cloud.
  3. It counts as a success only if it meets every success criterion: 1. success_rate_and_threshold 2. percentiles 3. failure_stats 4. best_sim 5. fresh_run_1 6. fresh_run_2 7. chart_svg_static
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{"type": "thread.started", "thread_id": "01a15941-7c55-77a0-80f1-088c36eb290c"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll implement the exact draw order and failure rules, and use the yearly balances to generate the percentile chart."}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc 'ls -la'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc 'ls -la'", "aggregated_output": "total 16\ndrwxr-xr-x 6 bob bob 4096 Oct 20 08:21 .\ndrwxrwxr-x 3 bob bob 4096 Oct 20 08:20 ..\ndr-xr-xr-x 2 bob bob   40 Oct 20 08:21 .agents\ndr-xr-xr-x 2 bob bob   40 Oct 20 08:21 .aws\ndr-xr-xr-x 2 bob bob   40 Oct 20 08:21 .codex\ndr-xr-xr-x 2 bob bob   40 Oct 20 08:21 .git\n-rw-r--r-- 1 bob bob 3486 Oct 20 08:17 TASK.md\n-rw-r--r-- 1 bob bob  226 Oct 20 08:17 input.json\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "{\n \"seed\": 485855,\n \"sims\": 1500,\n \"years\": 26,\n \"start_balance\": 2000000,\n \"annual_spend\": 77900.0,\n \"mean_return\": 0.0525,\n \"sd_return\": 0.1039,\n \"mean_inflation\": 0.0234,\n \"sd_inflation\": 0.0114,\n \"success_threshold\": 0.9\n}", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/j02/work/solve.py", "kind": "add"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/j02/work/solve.py", "kind": "add"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc 'python solve.py input.json answer.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc 'python solve.py input.json answer.json'", "aggregated_output": "/bin/bash: line 1: python: command not found\n", "exit_code": 127, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc 'python3 solve.py input.json answer.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc 'python3 solve.py input.json answer.json'", "aggregated_output": "", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_6", "type": "command_execution", "command": "/bin/bash -lc \"python3 - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import simulate\nprint(json.load(open('answer.json')))\nr=ET.parse('chart.svg').getroot()\nassert r.tag == '{http://www.w3.org/2000/svg}svg'\nassert {e.attrib['id'] for e in r.findall('{http://www.w3.org/2000/svg}polyline')} == {'p10','p50','p90'}\nd=dict(seed=1,sims=3,years=2,start_balance=10,annual_spend=6,mean_return=0,sd_return=0,mean_inflation=0,sd_inflation=0,success_threshold=0.5)\na,f=simulate(d)\nassert a == dict(success_rate=0.0,meets_threshold=False,sims_failed=3,p10=0.0,p50=0.0,p90=0.0,median_failure_year=2,best_sim=0,earliest_failure_sim=0)\nassert f == {10:[4.0,0.0],50:[4.0,0.0],90:[4.0,0.0]}\nd['annual_spend']=1\na,f=simulate(d)\nassert a['success_rate']==1 and a['best_sim']==0 and a['median_failure_year'] is None\nprint('Deterministic cases and SVG validation passed.')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "command_execution", "command": "/bin/bash -lc \"python3 - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import simulate\nprint(json.load(open('answer.json')))\nr=ET.parse('chart.svg').getroot()\nassert r.tag == '{http://www.w3.org/2000/svg}svg'\nassert {e.attrib['id'] for e in r.findall('{http://www.w3.org/2000/svg}polyline')} == {'p10','p50','p90'}\nd=dict(seed=1,sims=3,years=2,start_balance=10,annual_spend=6,mean_return=0,sd_return=0,mean_inflation=0,sd_inflation=0,success_threshold=0.5)\na,f=simulate(d)\nassert a == dict(success_rate=0.0,meets_threshold=False,sims_failed=3,p10=0.0,p50=0.0,p90=0.0,median_failure_year=2,best_sim=0,earliest_failure_sim=0)\nassert f == {10:[4.0,0.0],50:[4.0,0.0],90:[4.0,0.0]}\nd['annual_spend']=1\na,f=simulate(d)\nassert a['success_rate']==1 and a['best_sim']==0 and a['median_failure_year'] is None\nprint('Deterministic cases and SVG validation passed.')\nPY\"", "aggregated_output": "{'success_rate': 0.824667, 'meets_threshold': False, 'sims_failed': 263, 'p10': 0.0, 'p50': 1473403.0, 'p90': 5177505.89, 'median_failure_year': 23, 'best_sim': 692, 'earliest_failure_sim': 294}\nDeterministic cases and SVG validation passed.\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_7", "type": "agent_message", "text": "Created `solve.py`, `answer.json`, and `chart.svg`. Verified deterministic failure/tie cases and SVG structure.\n\nThe fixture\u2019s success rate is **0.824667**, with **263 failures**.\n\nRun with `python3 solve.py input.json answer.json` (`python` is unavailable in this environment)."}}
{"type": "turn.completed", "usage": {"input_tokens": 65748, "cached_input_tokens": 56960, "cache_write_input_tokens": 0, "output_tokens": 1760, "reasoning_output_tokens": 0}}

Get new runs and ranking changes weekly