Skip to content

GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking off) on Mortgage amortization with extra monthly payments and lump sums

OGPT-6.1 Soldefault · thinking off · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-22 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.04
Wall time
1m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Mortgage amortization with extra monthly payments and lump sums `input.json` describes a fixed-rate loan plus planned prepayments (synthetic). Write `solve.py` (Python 3 standard library only) so that `python solve.py input.json answer.json` produces the payment, the baseline payoff, and the full schedule with prepayments. We re-run it on fresh inputs, so do not hard-code anything. ## Input `principal`, `annual_rate` (percent, e.g. 6.125), `years`, `extra_monthly`, `extra_start_month` (first month the extra is paid; months count from 1), `lump_sums` (list of `{"month", "amount"}`; several can fall in the same month and then add up). ## Rules 1. Money is exact cents with `decimal.Decimal` and ROUND_HALF_UP to 0.01 (convert input numbers with `Decimal(str(x))`). 2. Scheduled payment: with float r = annual_rate / 1200 and n = years * 12, x = principal * r / (1 - (1 + r) ** -n) computed in floats; payment = `Decimal(str(x))` rounded to cents. 3. Each month m: interest = round(balance * annual_rate / 1200) (in Decimal). Prepayment = extra_monthly if m >= extra_start_month (else 0) plus the lump sums for month m. If payment + prepayment >= balance + interest, this is the last month: the amount paid is balance + interest and the balance becomes 0. Otherwise the amount paid is payment + prepayment and the balance drops by (amount paid - interest). 4. The baseline is the same loan with no extra and no lump sums. 5. `interest_saved` = baseline total interest - total interest with prepayments; `months_saved` = baseline months - months with prepayments. ## Output `answer.json`: ``` {"payment": ..., "baseline": {"months": ..., "total_interest": ..., "last_payment": ...}, "with_extra": {"months": ..., "total_interest": ..., "schedule": [[month, amount_paid, interest, principal_paid, balance_after], ...]}, "interest_saved": ..., "months_saved": ...} ``` One schedule row per month, money as numbers with 2 decimals. ## Chart (required visual) Also write `chart.svg` in the working directory: a static, self-contained SVG (root element `<svg>`, no scripts, no external images, fonts, stylesheets or links) that can be rendered headless. It must contain `<polyline id="balance">` drawing the with-extra remaining balance by month. ## Scoring The `chart_svg_static` check (one more check, counted in partial credit) passes only if `chart.svg` exists after the fixture run, parses, has root `<svg>`, contains the ids above with the stated tags, and has no `<script>`, `<img>`/`<image>`, `<link>`, `<iframe>`/`<object>`/`<embed>`, `on*` handlers, or `href`/`src` pointing anywhere except `#...` or `data:`. Its drawing is not graded. Hidden checks (partial credit = fraction passed; all must pass), money within 0.011, counts exact: `payment`, `baseline`, `schedule_rows` (every row), `savings` on the given input; `fresh_run_1`, `fresh_run_2` (the whole answer on fresh inputs; one has two lump sums in the same month). Each run has a 60-second limit.

Every schedule row is compared: payment rounding, interest rounding each month, extra start month, same-month lump sums that add up, and the short final payment. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).

These are synthetic test cases for comparing AI models, not financial advice.

Success criteria: 1. payment 2. baseline 3. schedule_rows 4. savings 5. fresh_run_1 6. fresh_run_2 7. chart_svg_static

Setup notes: Use Case Scout batch9 case j08-amortization-extra-payments (offline fixture). Prepared with batch9/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the scout's scorer: `python score.py WORKDIR KEYDIR`. Single run, no retries. User-level AGENTS.md (846 bytes, sha256 83ec4fe50da8) may have been in context; --ignore-user-config does not document excluding it. Visual output (chart.svg) rendered offline to PNG with google-chrome headless, network blocked, 1280x800 at ~3 s after load. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 4 (1 failed, recovered). Single run.

Estimated cost List-price estimate; not actually billed.

Output

Metrics

Estimated cost
$0.04
Wall time
1m
Human step-ins
0
Turns
1
Model usage
ModelInput tokOutput tokCost
OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking off84,7462,130$0.04

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking off) via Codex CLI 0.163.0 (codex exec --json), cloud.
  3. It counts as a success only if it meets every success criterion: 1. payment 2. baseline 3. schedule_rows 4. savings 5. fresh_run_1 6. fresh_run_2 7. chart_svg_static
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{"type": "thread.started", "thread_id": "01a163f3-1f55-7e43-b4a5-dac0e3f51be7"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll inspect the input, implement the amortization rules, and generate the balance chart.\n"}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g 'input.json' -g 'solve.py' -g '\"'!node_modules'\"'\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g 'input.json' -g 'solve.py' -g '\"'!node_modules'\"'\"", "aggregated_output": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/103_j08/work\ninput.json\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "{\n \"principal\": 285000,\n \"annual_rate\": 2.942,\n \"years\": 15,\n \"extra_monthly\": 400,\n \"extra_start_month\": 2,\n \"lump_sums\": [\n  {\n   \"month\": 42,\n   \"amount\": 7000\n  }\n ]\n}", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/103_j08/work/solve.py", "kind": "add"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/103_j08/work/solve.py", "kind": "add"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json && python - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom decimal import Decimal\nv=json.load(open('answer.json'),parse_float=Decimal)\nprint({k: val for k,val in v.items() if k\"'!='\"'with_extra'})\ns=v['with_extra']['schedule']\nassert len(s)==v['with_extra']['months']\nassert s[-1][-1]==0\nassert sum(row[3] for row in s)==Decimal('285000')\nassert sum(row[2] for row in s)==v['with_extra']['total_interest']\nroot=ET.parse('chart.svg').getroot()\nassert root.find('{http://www.w3.org/2000/svg}polyline').attrib['id']=='balance'\nprint('Verified schedule accounting and SVG; months:',len(s))\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json && python - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom decimal import Decimal\nv=json.load(open('answer.json'),parse_float=Decimal)\nprint({k: val for k,val in v.items() if k\"'!='\"'with_extra'})\ns=v['with_extra']['schedule']\nassert len(s)==v['with_extra']['months']\nassert s[-1][-1]==0\nassert sum(row[3] for row in s)==Decimal('285000')\nassert sum(row[2] for row in s)==v['with_extra']['total_interest']\nroot=ET.parse('chart.svg').getroot()\nassert root.find('{http://www.w3.org/2000/svg}polyline').attrib['id']=='balance'\nprint('Verified schedule accounting and SVG; months:',len(s))\nPY\"", "aggregated_output": "/bin/bash: line 1: python: command not found\n", "exit_code": 127, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json && python3 - <<'PY'\nimport json\nimport tempfile\nimport subprocess\nimport xml.etree.ElementTree as ET\nfrom decimal import Decimal\nv=json.load(open('answer.json'),parse_float=Decimal)\ns=v['with_extra']['schedule']\nassert s[-1][-1]==0\nassert sum(row[3] for row in s)==Decimal('285000')\nassert sum(row[2] for row in s)==v['with_extra']['total_interest']\nassert ET.parse('chart.svg').getroot().find('{http://www.w3.org/2000/svg}polyline').attrib['id']=='balance'\nprint('Payment:',v['payment'],'baseline months:',v['baseline']['months'],'extra months:',len(s))\nwith tempfile.TemporaryDirectory() as d:\n    data=dict(principal=1200,annual_rate=0,years=1,extra_monthly=10,extra_start_month=2,lump_sums=[dict(month=2,amount=50),dict(month=2,amount=25)])\n    open(d+'/input.json','w').write(json.dumps(data))\n    subprocess.run(['python3','solve.py',d+'/input.json',d+'/answer.json'],check=True)\n    out=json.load(open(d+'/answer.json'))\n    assert out['payment']==100\n    assert out['with_extra']['schedule'][1][1]==185\n    assert out['baseline']['months']==12\n    assert out['with_extra']['schedule'][-1][-1]==0\nprint('Verified accounting, duplicate lump sums, zero rate, and SVG.')\nPY\npython3 solve.py input.json answer.json\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json && python3 - <<'PY'\nimport json\nimport tempfile\nimport subprocess\nimport xml.etree.ElementTree as ET\nfrom decimal import Decimal\nv=json.load(open('answer.json'),parse_float=Decimal)\ns=v['with_extra']['schedule']\nassert s[-1][-1]==0\nassert sum(row[3] for row in s)==Decimal('285000')\nassert sum(row[2] for row in s)==v['with_extra']['total_interest']\nassert ET.parse('chart.svg').getroot().find('{http://www.w3.org/2000/svg}polyline').attrib['id']=='balance'\nprint('Payment:',v['payment'],'baseline months:',v['baseline']['months'],'extra months:',len(s))\nwith tempfile.TemporaryDirectory() as d:\n    data=dict(principal=1200,annual_rate=0,years=1,extra_monthly=10,extra_start_month=2,lump_sums=[dict(month=2,amount=50),dict(month=2,amount=25)])\n    open(d+'/input.json','w').write(json.dumps(data))\n    subprocess.run(['python3','solve.py',d+'/input.json',d+'/answer.json'],check=True)\n    out=json.load(open(d+'/answer.json'))\n    assert out['payment']==100\n    assert out['with_extra']['schedule'][1][1]==185\n    assert out['baseline']['months']==12\n    assert out['with_extra']['schedule'][-1][-1]==0\nprint('Verified accounting, duplicate lump sums, zero rate, and SVG.')\nPY\npython3 solve.py input.json answer.json\"", "aggregated_output": "Payment: 1960.22 baseline months: 180 extra months: 140\nVerified accounting, duplicate lump sums, zero rate, and SVG.\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "agent_message", "text": "Created [solve.py](solve.py), [answer.json](answer.json), and [chart.svg](chart.svg).\n\nVerified schedule accounting, duplicate lump sums, zero-interest handling, and SVG structure. The supplied loan pays off in 140 months versus 180 baseline.\n\nRun with `python3 solve.py input.json answer.json` (`python` isn\u2019t installed here)."}}
{"type": "turn.completed", "usage": {"input_tokens": 84746, "cached_input_tokens": 80000, "cache_write_input_tokens": 0, "output_tokens": 2130, "reasoning_output_tokens": 0}}

Get new runs and ranking changes weekly