Skip to content

GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking off) on Progressive tax on fictional brackets, and pre-tax vs after-tax savings comparison

OGPT-6.1 Soldefault · thinking off · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-22 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.05
Wall time
48s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Progressive tax on fictional brackets, and pre-tax ("traditional") vs after-tax ("Roth") savings `input.json` contains a **fictional** tax system (brackets and standard deductions per filing status), a list of incomes to tabulate, and savings scenarios. Write `solve.py` (Python 3 standard library only) so that `python solve.py input.json answer.json` computes the answers below. Use only the numbers in the input. We re-run it on fresh inputs, so do not hard-code anything. ## Rules 1. Taxable income = max(0, gross income - `standard_deduction[filing]`). 2. Brackets for a filing status are a list of `[lower_bound, rate]` sorted by bound. Rate j applies to the part of taxable income above `lower_bound_j` and up to `lower_bound_{j+1}` (the last bracket has no upper bound). Tax = sum over brackets. 3. Marginal rate = the rate of the last bracket whose lower bound is strictly below taxable income; if taxable income is 0, the first bracket's rate. 4. `tax_table`: for each `[filing, income]` in input order output `{"filing", "income", "tax", "effective_rate", "marginal_rate"}`; effective_rate = tax / income (0 when income is 0), 6 decimals. 5. Scenarios (each has `name`, `filing`, `wages`, `contribution`, `years`, `growth`, `retire_filing`, `retire_other_income`): - `tax_now_traditional` = tax on (wages - contribution); `tax_now_roth` = tax on wages; `tax_saved_now` = their difference; `marginal_rate_now` = marginal rate (rule 3) at the taxable income of (wages - contribution). - `traditional_balance` F = contribution * (1 + growth) ** years. `withdrawal_tax` = tax(retire_other_income + F) - tax(retire_other_income), both with `retire_filing`, taken as one withdrawal in a single year. `traditional_after_tax` = F - withdrawal_tax. - Same out-of-pocket cost: the after-tax account receives (contribution - tax_saved_now), so `roth_after_tax` = (contribution - tax_saved_now) * (1 + growth) ** years. - `winner` = "traditional" or "roth", whichever after-tax value is larger, or "tie" if they differ by less than 0.005. 6. Round money to 2 decimals and rates to 6 only when writing (bracket rates are written as given). ## Output `answer.json`: `{"tax_table": [...], "scenarios": [{"name", "tax_now_traditional", "tax_now_roth", "tax_saved_now", "marginal_rate_now", "traditional_balance", "withdrawal_tax", "traditional_after_tax", "roth_after_tax", "winner"}, ...]}` in input order. ## Scoring Hidden checks (partial credit = fraction passed; all must pass), money within 0.011, rates within 0.000001, winners exact: `tax_table_amounts`, `effective_and_marginal`, `scenario_tax_now`, `scenario_future_values`, `scenario_winners` on the given input; `fresh_run_1`, `fresh_run_2`: the whole answer on fresh inputs (tolerance 0.011). Each run has a 60-second limit.

Incomes placed exactly on bracket boundaries test the strict marginal-rate rule; the equal out-of-pocket comparison and the lump-withdrawal tax at retirement decide winners. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).

These are synthetic test cases for comparing AI models, not financial advice.

Success criteria: 1. tax_table_amounts 2. effective_and_marginal 3. scenario_tax_now 4. scenario_future_values 5. scenario_winners 6. fresh_run_1 7. fresh_run_2

Setup notes: Use Case Scout batch9 case j05-tax-brackets-roth-vs-traditional (offline fixture). Prepared with batch9/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the scout's scorer: `python score.py WORKDIR KEYDIR`. Single run, no retries. User-level AGENTS.md (846 bytes, sha256 83ec4fe50da8) may have been in context; --ignore-user-config does not document excluding it. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 4 (2 failed, recovered). Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.05
Wall time
48s
Human step-ins
0
Turns
1
Model usage
ModelInput tokOutput tokCost
OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking off90,1861,676$0.05

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking off) via Codex CLI 0.163.0 (codex exec --json), cloud.
  3. It counts as a success only if it meets every success criterion: 1. tax_table_amounts 2. effective_and_marginal 3. scenario_tax_now 4. scenario_future_values 5. scenario_winners 6. fresh_run_1 7. fresh_run_2
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{"type": "thread.started", "thread_id": "01a163f1-1730-7102-83ec-beb2ccae83ff"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll check the input structure, write `solve.py`, and verify the tax and savings calculations.\n"}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc 'ls -la; cat input.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc 'ls -la; cat input.json'", "aggregated_output": "total 16\ndrwxr-xr-x 6 bob bob 4096 Oct 22 10:08 .\ndrwxrwxr-x 3 bob bob 4096 Oct 22 10:08 ..\ndr-xr-xr-x 2 bob bob   40 Oct 22 10:08 .agents\ndr-xr-xr-x 2 bob bob   40 Oct 22 10:08 .aws\ndr-xr-xr-x 2 bob bob   40 Oct 22 10:08 .codex\ndr-xr-xr-x 2 bob bob   40 Oct 22 10:08 .git\n-rw-r--r-- 1 bob bob 2801 Oct 22 10:08 TASK.md\n-rw-r--r-- 1 bob bob 2423 Oct 22 10:08 input.json\n{\n \"note\": \"Fictional tax system for this exercise. Brackets are [lower_bound, rate]; each rate applies to taxable income above its lower bound up to the next bound.\",\n \"brackets\": {\n  \"single\": [\n   [\n    0,\n    0.08\n   ],\n   [\n    12480,\n    0.14\n   ],\n   [\n    49920,\n    0.21\n   ],\n   [\n    109200,\n    0.27\n   ],\n   [\n    208000,\n    0.33\n   ],\n   [\n    499200,\n    0.39\n   ]\n  ],\n  \"joint\": [\n   [\n    0,\n    0.08\n   ],\n   [\n    24960,\n    0.14\n   ],\n   [\n    99840,\n    0.21\n   ],\n   [\n    218400,\n    0.27\n   ],\n   [\n    416000,\n    0.33\n   ],\n   [\n    582400,\n    0.39\n   ]\n  ],\n  \"head\": [\n   [\n    0,\n    0.08\n   ],\n   [\n    18720,\n    0.14\n   ],\n   [\n    74880,\n    0.21\n   ],\n   [\n    163800,\n    0.27\n   ],\n   [\n    312000,\n    0.33\n   ],\n   [\n    748800,\n    0.39\n   ]\n  ]\n },\n \"standard_deduction\": {\n  \"single\": 15600,\n  \"joint\": 31200,\n  \"head\": 23400\n },\n \"tax_table\": [\n  [\n   \"joint\",\n   0\n  ],\n  [\n   \"joint\",\n   15600\n  ],\n  [\n   \"single\",\n   28080\n  ],\n  [\n   \"joint\",\n   65520\n  ],\n  [\n   \"joint\",\n   683000\n  ],\n  [\n   \"single\",\n   669000\n  ],\n  [\n   \"single\",\n   295000\n  ],\n  [\n   \"joint\",\n   144000\n  ],\n  [\n   \"single\",\n   92000\n  ],\n  [\n   \"head\",\n   603000\n  ],\n  [\n   \"joint\",\n   51000\n  ],\n  [\n   \"joint\",\n   714000\n  ]\n ],\n \"scenarios\": [\n  {\n   \"name\": \"S1\",\n   \"filing\": \"single\",\n   \"wages\": 349000,\n   \"contribution\": 23000,\n   \"years\": 16,\n   \"growth\": 0.0401,\n   \"retire_filing\": \"joint\",\n   \"retire_other_income\": 57000\n  },\n  {\n   \"name\": \"S2\",\n   \"filing\": \"single\",\n   \"wages\": 393000,\n   \"contribution\": 12000,\n   \"years\": 24,\n   \"growth\": 0.0431,\n   \"retire_filing\": \"head\",\n   \"retire_other_income\": 55000\n  },\n  {\n   \"name\": \"S3\",\n   \"filing\": \"joint\",\n   \"wages\": 394000,\n   \"contribution\": 12000,\n   \"years\": 9,\n   \"growth\": 0.0477,\n   \"retire_filing\": \"joint\",\n   \"retire_other_income\": 89000\n  },\n  {\n   \"name\": \"S4\",\n   \"filing\": \"single\",\n   \"wages\": 252000,\n   \"contribution\": 6000,\n   \"years\": 24,\n   \"growth\": 0.0398,\n   \"retire_filing\": \"joint\",\n   \"retire_other_income\": 5000\n  },\n  {\n   \"name\": \"S5\",\n   \"filing\": \"joint\",\n   \"wages\": 94000,\n   \"contribution\": 23000,\n   \"years\": 15,\n   \"growth\": 0.048,\n   \"retire_filing\": \"head\",\n   \"retire_other_income\": 116000\n  },\n  {\n   \"name\": \"S6\",\n   \"filing\": \"single\",\n   \"wages\": 93000,\n   \"contribution\": 6000,\n   \"years\": 6,\n   \"growth\": 0.0562,\n   \"retire_filing\": \"head\",\n   \"retire_other_income\": 74000\n  }\n ]\n}", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc \"rg --files -g AGENTS.md -g '\"'!input.json'\"' . /home/bob/usecase-runs/batch/out 2>/dev/null\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc \"rg --files -g AGENTS.md -g '\"'!input.json'\"' . /home/bob/usecase-runs/batch/out 2>/dev/null\"", "aggregated_output": "", "exit_code": 1, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/100_j05/work/solve.py", "kind": "add"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/100_j05/work/solve.py", "kind": "add"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json\npython - <<'PY'\nfrom solve import solve\nx = {'brackets': {'x': [[0, .1], [100, .2]]}, 'standard_deduction': {'x': 10}, 'tax_table': [['x', 0], ['x', 110], ['x', 111]], 'scenarios': [{'name': 'tie', 'filing': 'x', 'wages': 60, 'contribution': 20, 'years': 0, 'growth': 0, 'retire_filing': 'x', 'retire_other_income': 20}]}\nr = solve(x)\nassert [v['tax'] for v in r['tax_table']] == [0, 10, 10.2]\nassert [v['marginal_rate'] for v in r['tax_table']] == [.1, .1, .2]\nassert r['scenarios'][0]['winner'] == 'tie'\nassert r['scenarios'][0]['traditional_after_tax'] == 18\nassert r['scenarios'][0]['roth_after_tax'] == 18\nprint('Boundary and equal-cost scenario checks passed.')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json\npython - <<'PY'\nfrom solve import solve\nx = {'brackets': {'x': [[0, .1], [100, .2]]}, 'standard_deduction': {'x': 10}, 'tax_table': [['x', 0], ['x', 110], ['x', 111]], 'scenarios': [{'name': 'tie', 'filing': 'x', 'wages': 60, 'contribution': 20, 'years': 0, 'growth': 0, 'retire_filing': 'x', 'retire_other_income': 20}]}\nr = solve(x)\nassert [v['tax'] for v in r['tax_table']] == [0, 10, 10.2]\nassert [v['marginal_rate'] for v in r['tax_table']] == [.1, .1, .2]\nassert r['scenarios'][0]['winner'] == 'tie'\nassert r['scenarios'][0]['traditional_after_tax'] == 18\nassert r['scenarios'][0]['roth_after_tax'] == 18\nprint('Boundary and equal-cost scenario checks passed.')\nPY\"", "aggregated_output": "/bin/bash: line 1: python: command not found\n/bin/bash: line 2: python: command not found\n", "exit_code": 127, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json\npython3 - <<'PY'\nfrom solve import solve\nx = {'brackets': {'x': [[0, .1], [100, .2]]}, 'standard_deduction': {'x': 10}, 'tax_table': [['x', 0], ['x', 110], ['x', 111]], 'scenarios': [{'name': 'tie', 'filing': 'x', 'wages': 60, 'contribution': 20, 'years': 0, 'growth': 0, 'retire_filing': 'x', 'retire_other_income': 20}]}\nr = solve(x)\nassert [v['tax'] for v in r['tax_table']] == [0, 10, 10.2]\nassert [v['marginal_rate'] for v in r['tax_table']] == [.1, .1, .2]\nassert r['scenarios'][0]['winner'] == 'tie'\nassert r['scenarios'][0]['traditional_after_tax'] == 18\nassert r['scenarios'][0]['roth_after_tax'] == 18\nprint('Boundary and equal-cost scenario checks passed.')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json\npython3 - <<'PY'\nfrom solve import solve\nx = {'brackets': {'x': [[0, .1], [100, .2]]}, 'standard_deduction': {'x': 10}, 'tax_table': [['x', 0], ['x', 110], ['x', 111]], 'scenarios': [{'name': 'tie', 'filing': 'x', 'wages': 60, 'contribution': 20, 'years': 0, 'growth': 0, 'retire_filing': 'x', 'retire_other_income': 20}]}\nr = solve(x)\nassert [v['tax'] for v in r['tax_table']] == [0, 10, 10.2]\nassert [v['marginal_rate'] for v in r['tax_table']] == [.1, .1, .2]\nassert r['scenarios'][0]['winner'] == 'tie'\nassert r['scenarios'][0]['traditional_after_tax'] == 18\nassert r['scenarios'][0]['roth_after_tax'] == 18\nprint('Boundary and equal-cost scenario checks passed.')\nPY\"", "aggregated_output": "Boundary and equal-cost scenario checks passed.\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "agent_message", "text": "Created `solve.py` and generated `answer.json`. Verified bracket boundaries, zero income, and an equal-cost tie scenario.\n\nRun with `python3 solve.py input.json answer.json` (`python` is unavailable here)."}}
{"type": "turn.completed", "usage": {"input_tokens": 90186, "cached_input_tokens": 78976, "cache_write_input_tokens": 0, "output_tokens": 1676, "reasoning_output_tokens": 0}}

Get new runs and ranking changes weekly