GPT-6.1 Sol (effort default (no reasoning-effort override) · thinking off) on AIS collision risk: closest point of approach and alert levels
OGPT-6.1 Soldefault · thinking off · Codex CLI 0.163.0 (codex exec --json) · cloud · 2026-10-22 · by @bobsiopener
- Estimated cost
- $0.05
- Wall time
- 1m
- Human step-ins
- 0
- Outcome
- Pass
Compare with other models on this task →
Summary
Exact prompt
# AIS collision risk: CPA, TCPA and alert levels `input.json` (synthetic; not for collision avoidance): `own` ship (`lat`, `lon`, `cog` degrees true, `sog` knots), `targets` (each `mmsi`, `lat`, `lon`, `cog`, `sog`), thresholds `cpa_danger_nm`, `cpa_caution_nm`, `tcpa_max_min`. Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json` writes `answer.json` and `radar.svg`. Fresh inputs are used too. ## Rules 1. Skip a target (list its mmsi in `skipped`, ascending) if lat, lon, cog or sog is missing/null, sog >= 102.3 (AIS "not available"), or cog >= 360 (not available). Keep the rest in input order. 2. Local flat frame around own ship in nm: x = (lon - own lon) * 60 * cos(own lat), y = (lat - own lat) * 60. Velocity of a ship = (sog * sin(cog), sog * cos(cog)) knots. 3. Relative position r = target - own, relative velocity v = v_target - v_own. If |v|^2 < 1e-9, TCPA = 0. Otherwise TCPA (hours) = max(0, -(r . v) / |v|^2). CPA = |r + v * TCPA|. 4. `range_nm` = |r|, `bearing_deg` = atan2(x, y) mod 360 (true bearing from own ship), `tcpa_min` = TCPA * 60. 5. Level: `danger` if CPA <= cpa_danger and tcpa_min <= tcpa_max; else `caution` if CPA <= cpa_caution and tcpa_min <= tcpa_max; else `none`. 6. `alerts`: mmsi of danger then caution targets, each group sorted by tcpa_min then mmsi. Round range/CPA to 3 decimals, bearing to 1, tcpa_min to 2. ## Visual output: radar.svg Static, self-contained SVG (root `<svg>`, no scripts, images, links or external assets; renders headless) with an element `id="own-ship"` and one element `id="target-<mmsi>"` for every non-skipped target. The drawing itself is not graded. ## answer.json `{"targets": [{"mmsi", "range_nm", "bearing_deg", "cpa_nm", "tcpa_min", "level"}, ...], "skipped": [...], "alerts": [...]}` ## Scoring Hidden checks (partial credit; all must pass): `filtering`, `range_bearing`, `cpa_tcpa`, `levels_and_alerts`, `radar_svg_static` (radar.svg exists, follows the static rules, has `own-ship` and every `target-<mmsi>`), `fresh_run_1`, `fresh_run_2`. Tolerances: distances 0.002 nm, bearings 0.11, minutes 0.011 (fresh runs 0.011 for all numbers). 60-second limit per run.
AIS not-available sentinels (sog 102.3, cog 360), a stationary target, diverging targets with TCPA clamped to 0, longitude scaling, near-threshold crossing targets, and a two-key alert sort. Original benchmark task written for usecaserank.com. Headless, offline run on a reproducible fixture. Cost shown is a list-price API-equivalent estimate (run on a ChatGPT subscription, not billed).
Benchmark code for comparing AI models, not navigation or safety software.
Success criteria: 1. filtering 2. range_bearing 3. cpa_tcpa 4. levels_and_alerts 5. radar_svg_static 6. fresh_run_1 7. fresh_run_2
Setup notes: Use Case Scout batch10 case k08-ais-cpa-tcpa (offline fixture). Prepared with batch10/prepare.sh on the shared box; only TASK.md + starting files were copied to the agent machine, hidden key/tests stayed on the box. Prompt = TASK.md verbatim on stdin. Flags: --json -s workspace-write --skip-git-repo-check --ephemeral --ignore-user-config --ignore-rules -C <workdir>. Scored on the box with the scout's scorer: `python score.py WORKDIR KEYDIR`. Single run, no retries. User-level AGENTS.md (846 bytes, sha256 83ec4fe50da8) may have been in context; --ignore-user-config does not document excluding it. Visual output (radar.svg) rendered offline to PNG with google-chrome headless, network blocked, 1280x800 at ~3 s after load. Serving model not confirmed: requested via -m; the event log does not echo the serving model id. Price source is secondary (pricing.json verification='secondary', https://www.actuia.com/en/news/ai-api-pricing-in-october-2026-050-to-50-for-output/), not an official OpenAI price page. Score: 7/7. Cost is an ESTIMATE: measured tokens x pricing.json list price (ChatGPT plan, nothing billed); input_tokens include cached reads; output_tokens include reasoning tokens. Tool calls 4 (1 failed, recovered). Single run.
Estimated cost List-price estimate; not actually billed.
Output
Metrics
- Estimated cost
- $0.05
- Wall time
- 1m
- Human step-ins
- 0
- Turns
- 1
| Model | Input tok | Output tok | Cost |
|---|---|---|---|
| OGPT-6.1 Solopenai/gpt-6.1-soleffort default (no reasoning-effort override) · thinking off | 86,684 | 2,043 | $0.05 |
Reproduce this run
- Use the use case prompt on the use case page.
- Same setup: GPT-6.1 Sol (openai/gpt-6.1-sol, effort default (no reasoning-effort override) · thinking off) via
Codex CLI 0.163.0 (codex exec --json), cloud. - It counts as a success only if it meets every success criterion: 1. filtering 2. range_bearing 3. cpa_tcpa 4. levels_and_alerts 5. radar_svg_static 6. fresh_run_1 7. fresh_run_2
- Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.
How runs are scored: Methodology.
Was this useful?
Automated checks
Trust 80% comes from these checks (how Trust % works).
- Run date is not in the future : passed
- Model is in the dated pricing table : passed
- Cost recomputed from tokens matches : passed
- Cached tokens are consistent : passed
- Output throughput is plausible : passed
- Autonomy matches step-ins : passed
- Step-ins do not exceed turns : passed
- Retries agree with first-try result : passed
- Evidence log SHA-256 matches : passed
- Evidence log is substantial : passed
Evidence log
{"type": "thread.started", "thread_id": "01a163fe-9915-71e1-b2be-5f0c6a17e201"}
{"type": "turn.started"}
{"type": "item.completed", "item": {"id": "item_0", "type": "agent_message", "text": "I\u2019ll implement the calculations and static SVG output, then verify them with a synthetic input.\n"}}
{"type": "item.started", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g 'input.json' -g 'solve.py' -g '\"'!node_modules'\"'\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_1", "type": "command_execution", "command": "/bin/bash -lc \"pwd && rg --files -g 'AGENTS.md' -g 'input.json' -g 'solve.py' -g '\"'!node_modules'\"'\"", "aggregated_output": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/115_k08/work\ninput.json\n", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_2", "type": "command_execution", "command": "/bin/bash -lc 'cat input.json'", "aggregated_output": "{\n \"own\": {\n \"lat\": 46.95882,\n \"lon\": -127.13341,\n \"cog\": 298,\n \"sog\": 4.2\n },\n \"targets\": [\n {\n \"mmsi\": 795070736,\n \"lat\": 46.83549,\n \"lon\": -127.20639,\n \"cog\": 14.0,\n \"sog\": 41.3\n },\n {\n \"mmsi\": 276139249,\n \"lat\": 47.04512,\n \"lon\": -127.14288,\n \"cog\": 274.7,\n \"sog\": 14.0\n },\n {\n \"mmsi\": 660159254,\n \"lat\": 46.94775,\n \"lon\": -127.11005,\n \"cog\": 158.2,\n \"sog\": 102.3\n },\n {\n \"mmsi\": 265918776,\n \"lat\": 47.00577,\n \"lon\": -127.29469,\n \"cog\": 118.4,\n \"sog\": 65.0\n },\n {\n \"mmsi\": 743277030,\n \"lat\": 46.96679,\n \"lon\": -126.9877,\n \"cog\": 276.4,\n \"sog\": 15.0\n },\n {\n \"mmsi\": 562375419,\n \"lat\": 46.94375,\n \"lon\": -127.0556,\n \"cog\": 298.3,\n \"sog\": 11.9\n },\n {\n \"mmsi\": 613463677,\n \"lat\": 47.0609,\n \"lon\": -127.13599,\n \"cog\": 187.0,\n \"sog\": 26.0\n },\n {\n \"mmsi\": 340778672,\n \"lat\": 46.96234,\n \"lon\": -127.07372,\n \"cog\": 360.0,\n \"sog\": 11.2\n },\n {\n \"mmsi\": 286629533,\n \"lat\": 47.0064,\n \"lon\": -127.19976,\n \"cog\": 214.3,\n \"sog\": 0.0\n },\n {\n \"mmsi\": 637679438,\n \"lat\": 46.90575,\n \"lon\": -127.01879,\n \"cog\": 27.4,\n \"sog\": 19.4\n },\n {\n \"mmsi\": 327721356,\n \"lat\": 46.9028,\n \"lon\": -127.06074,\n \"cog\": 204.2,\n \"sog\": 10.7\n },\n {\n \"mmsi\": 777617410,\n \"lat\": 46.84177,\n \"lon\": -127.07896,\n \"cog\": 351.5,\n \"sog\": 8.5\n }\n ],\n \"cpa_danger_nm\": 0.5,\n \"cpa_caution_nm\": 1.0,\n \"tcpa_max_min\": 30\n}", "exit_code": 0, "status": "completed"}}
{"type": "item.started", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/115_k08/work/solve.py", "kind": "add"}], "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_3", "type": "file_change", "changes": [{"path": "/home/bob/usecase-runs/batch/out/gpt-6.1-sol/115_k08/work/solve.py", "kind": "add"}], "status": "completed"}}
{"type": "item.started", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json && python - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import solve\nr=json.load(open('answer.json'))\nroot=ET.parse('radar.svg').getroot()\nids={e.get('id') for e in root.iter()}\nassert 'own-ship' in ids\nassert all('target-'+str(t['mmsi']) in ids for t in r['targets'])\nbase={'own':{'lat':0,'lon':0,'cog':0,'sog':0},'cpa_danger_nm':0.5,'cpa_caution_nm':1,'tcpa_max_min':60}\nbase['targets']=[{'mmsi':2,'lat':1/60,'lon':0,'cog':180,'sog':1},{'mmsi':1,'lat':0,'lon':0,'cog':0,'sog':0},{'mmsi':3,'lat':None}]\na,_=solve(base)\nassert a['targets'][0]['tcpa_min']==60\nassert a['targets'][0]['cpa_nm']==0\nassert a['alerts']==[1,2]\nassert a['skipped']==[3]\nprint('Verified sample output, SVG IDs, closing approach, equal velocities, filtering, and alert ordering.')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_4", "type": "command_execution", "command": "/bin/bash -lc \"python solve.py input.json answer.json && python - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import solve\nr=json.load(open('answer.json'))\nroot=ET.parse('radar.svg').getroot()\nids={e.get('id') for e in root.iter()}\nassert 'own-ship' in ids\nassert all('target-'+str(t['mmsi']) in ids for t in r['targets'])\nbase={'own':{'lat':0,'lon':0,'cog':0,'sog':0},'cpa_danger_nm':0.5,'cpa_caution_nm':1,'tcpa_max_min':60}\nbase['targets']=[{'mmsi':2,'lat':1/60,'lon':0,'cog':180,'sog':1},{'mmsi':1,'lat':0,'lon':0,'cog':0,'sog':0},{'mmsi':3,'lat':None}]\na,_=solve(base)\nassert a['targets'][0]['tcpa_min']==60\nassert a['targets'][0]['cpa_nm']==0\nassert a['alerts']==[1,2]\nassert a['skipped']==[3]\nprint('Verified sample output, SVG IDs, closing approach, equal velocities, filtering, and alert ordering.')\nPY\"", "aggregated_output": "/bin/bash: line 1: python: command not found\n", "exit_code": 127, "status": "failed"}}
{"type": "item.started", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json && python3 - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import solve\nr=json.load(open('answer.json'))\nroot=ET.parse('radar.svg').getroot()\nids={e.get('id') for e in root.iter()}\nassert 'own-ship' in ids\nassert all('target-'+str(t['mmsi']) in ids for t in r['targets'])\nb={'own':{'lat':0,'lon':0,'cog':0,'sog':0},'cpa_danger_nm':0.5,'cpa_caution_nm':1,'tcpa_max_min':60,'targets':[{'mmsi':2,'lat':1/60,'lon':0,'cog':180,'sog':1},{'mmsi':1,'lat':0,'lon':0,'cog':0,'sog':0},{'mmsi':3,'lat':None}]}\na,_=solve(b)\nassert a['targets'][0]['tcpa_min']==60\nassert a['targets'][0]['cpa_nm']==0\nassert a['alerts']==[1,2]\nassert a['skipped']==[3]\nprint('Verified sample output, SVG IDs, closing approach, equal velocities, filtering, and alert ordering.')\nPY\"", "aggregated_output": "", "exit_code": null, "status": "in_progress"}}
{"type": "item.completed", "item": {"id": "item_5", "type": "command_execution", "command": "/bin/bash -lc \"python3 solve.py input.json answer.json && python3 - <<'PY'\nimport json\nimport xml.etree.ElementTree as ET\nfrom solve import solve\nr=json.load(open('answer.json'))\nroot=ET.parse('radar.svg').getroot()\nids={e.get('id') for e in root.iter()}\nassert 'own-ship' in ids\nassert all('target-'+str(t['mmsi']) in ids for t in r['targets'])\nb={'own':{'lat':0,'lon':0,'cog':0,'sog':0},'cpa_danger_nm':0.5,'cpa_caution_nm':1,'tcpa_max_min':60,'targets':[{'mmsi':2,'lat':1/60,'lon':0,'cog':180,'sog':1},{'mmsi':1,'lat':0,'lon':0,'cog':0,'sog':0},{'mmsi':3,'lat':None}]}\na,_=solve(b)\nassert a['targets'][0]['tcpa_min']==60\nassert a['targets'][0]['cpa_nm']==0\nassert a['alerts']==[1,2]\nassert a['skipped']==[3]\nprint('Verified sample output, SVG IDs, closing approach, equal velocities, filtering, and alert ordering.')\nPY\"", "aggregated_output": "Verified sample output, SVG IDs, closing approach, equal velocities, filtering, and alert ordering.\n", "exit_code": 0, "status": "completed"}}
{"type": "item.completed", "item": {"id": "item_6", "type": "agent_message", "text": "Implemented [solve.py](solve.py) using only the Python standard library. Generated `answer.json` and `radar.svg` and verified calculations, filtering, alert ordering, and required SVG IDs.\n\nRun with:\n\n```bash\npython3 solve.py input.json answer.json\n```"}}
{"type": "turn.completed", "usage": {"input_tokens": 86684, "cached_input_tokens": 77696, "cache_write_input_tokens": 0, "output_tokens": 2043, "reasoning_output_tokens": 0}}