Skip to content

Qwen3 Coder 480B Turbo (Venice) on Implement a spec exactly, with nothing extra

QQwen3 Coder 480B Turbo (Venice) · venice-mini-agent v1.2 · cloud · 2026-10-15 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Cost
$0.0023
Wall time
22s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

Implement SPEC.md exactly. Then stop. Do not add anything the spec does not ask for.

Task prompt given to the agent (headless, offline, synthetic fixture inputs): Implement SPEC.md exactly. Then stop. Do not add anything the spec does not ask for. Inspired by https://x.com/mave99a/status/2096480295692460432 (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost is real Venice spend (paid from DIEM credit), as reported by Venice.

Success criteria: All automated rubric checks pass: hidden test test_flow; hidden test test_overwrite; under 60 non-blank lines; no classes; no extra files; stdlib only (no non-stdlib imports).

Setup notes: Harness: venice-mini-agent v1.2 via Venice OpenAI-compatible API. Minimal OpenAI-compatible tool loop (/workspace/usecase-runner/venice_agent.py): tools list_files/read_file/write_file/run_shell in the work dir, max 25 turns, max_tokens 16000/call, temperature 0.2, Venice system prompt and web search off, $0.08 per-run cost cap. Identical for all Venice models. Cost source: Venice /api/v1/models model_spec.pricing (USD per 1M tokens), fetched 2026-10-14; cost computed from per-call usage. Paid from Bob's DIEM daily allowance. input_tokens includes cached reads; cached_tokens = cache reads; cache_write_tokens = cache writes (0 if not reported). Single run, no retries; scored automatically (6/6 checks). Runner notes: Adapted: 24h Codex run scaled to a minimal-spec task scored on spec adherence and over-engineering limits. Single run, no retries. Same rubric/score.py as Claude runs. Web disabled.

Metrics

Cost
$0.0023
Wall time
22s
Human step-ins
0
Turns
12
Model usage
ModelInput tokOutput tokCost
QQwen3 Coder 480B Turbo (Venice)venice/qwen3-coder-480b-a35b-instruct-turbo15,581962$0.0023

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Qwen3 Coder 480B Turbo (Venice) (venice/qwen3-coder-480b-a35b-instruct-turbo) via venice-mini-agent v1.2, cloud.
  3. It counts as a success only if it meets every success criterion: All automated rubric checks pass: hidden test test_flow; hidden test test_overwrite; under 60 non-blank lines; no classes; no extra files; stdlib only (no non-stdlib imports).
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{"turn": 1, "response": {"service_tier": "default", "id": "chatcmpl-ReMpZh2k0xHcINUddlI4FQon", "object": "chat.completion", "created": 1792022425, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "I'll implement the specification exactly as requested. First, let me check what files are available to understand the project structure.\n\n", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_0a2d6d1b12a90ed0", "type": "function", "function": {"name": "list_files", "arguments": "{\"path\": \".\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 531, "completion_tokens": 44, "total_tokens": 575, "prompt_tokens_details": {"cached_tokens": 480}, "cache_read_input_tokens": 480}, "cost": {"usd": 0, "diem": 0.00010305000000000001}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 1, "tool": "list_files", "args": {"path": "."}, "result": "./SPEC.md"}
{"turn": 2, "response": {"service_tier": "default", "id": "chatcmpl-RxbPKiu7xiXbQoFRfkVuyOti", "object": "chat.completion", "created": 1792022426, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Let me read the specification file to understand what needs to be implemented.\n\n", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_cd40687e410bb11a", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\": \"SPEC.md\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 592, "completion_tokens": 36, "total_tokens": 628, "prompt_tokens_details": {"cached_tokens": 544}, "cache_read_input_tokens": 544}, "cost": {"usd": 0, "diem": 9.256e-05}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 2, "tool": "read_file", "args": {"path": "SPEC.md"}, "result": "# kv.py spec\nA tiny key-value store CLI. Single file kv.py, Python 3 stdlib only, data persisted in kv.json in the current directory.\nCommands:\n- python3 kv.py set KEY VALUE   -> prints OK\n- python3 kv.py get KEY         -> prints VALUE, or prints \"NOT FOUND\" and exits 1\n- python3 kv.py del KEY         -> prints OK, or \"NOT FOUND\" and exits 1\n- python3 kv.py list            -> prints keys sorted, one per line\nKeep it minimal: no classes, no extra features, no extra files, under 60 non-blank lines.\n"}
{"turn": 3, "response": {"service_tier": "default", "id": "chatcmpl-RiUzuakMIfEWHMGV0ctxNrQQ", "object": "chat.completion", "created": 1792022427, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Now I'll implement the key-value store according to the specification. I'll create a single file `kv.py` that uses the standard library only and persists data in `kv.json`.\n\n", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_0a3b105778fcfb8e", "type": "function", "function": {"name": "write_file", "arguments": "{\"path\": \"kv.py\", \"content\": \"#!/usr/bin/env python3\\nimport sys\\nimport json\\nimport os\\n\\nDB_FILE = 'kv.json'\\n\\ndef load_db():\\n    if os.path.exists(DB_FILE):\\n        with open(DB_FILE, 'r') as f:\\n            return json.load(f)\\n    return {}\\n\\ndef save_db(db):\\n    with open(DB_FILE, 'w') as f:\\n        json.dump(db, f)\\n\\ndef cmd_set(key, value):\\n    db = load_db()\\n    db[key] = value\\n    save_db(db)\\n    print('OK')\\n\\ndef cmd_get(key):\\n    db = load_db()\\n    if key in db:\\n        print(db[key])\\n    else:\\n        print('NOT FOUND')\\n        sys.exit(1)\\n\\ndef cmd_del(key):\\n    db = load_db()\\n    if key in db:\\n        del db[key]\\n        save_db(db)\\n        print('OK')\\n    else:\\n        print('NOT FOUND')\\n        sys.exit(1)\\n\\ndef cmd_list():\\n    db = load_db()\\n    for key in sorted(db.keys()):\\n        print(key)\\n\\nif __name__ == '__main__':\\n    if len(sys.argv) < 2:\\n        print('Usage: kv.py [set|get|del|list] ...')\\n        sys.exit(1)\\n    \\n    cmd = sys.argv[1]\\n    \\n    if cmd == 'set' and len(sys.argv) == 4:\\n        cmd_set(sys.argv[2], sys.argv[3])\\n    elif cmd == 'get' and len(sys.argv) == 3:\\n        cmd_get(sys.argv[2])\\n    elif cmd == 'del' and len(sys.argv) == 3:\\n        cmd_del(sys.argv[2])\\n    elif cmd == 'list' and len(sys.argv) == 2:\\n        cmd_list()\\n    else:\\n        print('Invalid usage')\\n        sys.exit(1)\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 775, "completion_tokens": 448, "total_tokens": 1223, "prompt_tokens_details": {"cached_tokens": 608}, "cache_read_input_tokens": 608}, "cost": {"usd": 0, "diem": 0.00075477}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 3, "tool": "write_file", "args": {"path": "kv.py", "content": "#!/usr/bin/env python3\nimport sys\nimport json\nimport os\n\nDB_FILE = 'kv.json'\n\ndef load_db():\n    if os.path.exists(DB_FILE):\n        with open(DB_FILE, 'r') as f:\n            return json.load(f)\n    return {}\n\ndef save_db(db):\n    with open(DB_FILE, 'w') as f:\n        json.dump(db, f)\n\ndef cmd_set(key, value):\n    db = load_db()\n    db[key] = value\n    save_db(db)\n    print('OK')\n\ndef cmd_get(key):\n    db = load_db()\n    if key in db:\n        print(db[key])\n    else:\n        print('NOT FOUND')\n        sys.exit(1)\n\ndef cmd_del(key):\n    db = load_db()\n    if key in db:\n        del db[key]\n        save_db(db)\n        print('OK')\n    else:\n        print('NOT FOUND')\n        sys.exit(1)\n\ndef cmd_list():\n    db = load_db()\n    for key in sorted(db.keys()):\n        print(key)\n\nif __name__ == '__main__':\n    if len(sys.argv) < 2:\n        print('Usage: kv.py [set|get|del|list] ...')\n        sys.exit(1)\n    \n    cmd = sys.argv[1]\n    \n    if cmd == 'set' and len(sys.argv) == 4:\n        cmd_set(sys.argv[2], sys.argv[3])\n    elif cmd == 'get' and len(sys.argv) == 3:\n        cmd_get(sys.argv[2])\n    elif cmd == 'del' and len(sys.argv) == 3:\n        cmd_del(sys.argv[2])\n    elif cmd == 'list' and len(sys.argv) == 2:\n        cmd_list()\n    else:\n        print('Invalid usage')\n        sys.exit(1)"}, "result": "wrote 1302 chars"}
{"turn": 4, "response": {"service_tier": "default", "id": "chatcmpl-RdVIHjC7z383f7v6AA13YJyG", "object": "chat.completion", "created": 1792022433, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Let me check if the implementation meets the specification requirements. The spec asks for under 60 non-blank lines, so let me count the actual lines in my implementation.\n\n", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_e3c8887a062b992d", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"cat kv.py | grep -v \\\"^$\\\" | wc -l\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1245, "completion_tokens": 70, "total_tokens": 1315, "prompt_tokens_details": {"cached_tokens": 1200}, "cache_read_input_tokens": 1200}, "cost": {"usd": 0, "diem": 0.00016875}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 4, "tool": "run_shell", "args": {"command": "cat kv.py | grep -v \"^$\" | wc -l"}, "result": "exit=0\nstdout:\n56\n\nstderr:\n"}
{"turn": 5, "response": {"service_tier": "default", "id": "chatcmpl-R44YGsji2vGcbVGTgDBAnt9O", "object": "chat.completion", "created": 1792022435, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Perfect! The implementation has 56 non-blank lines, which is under the 60-line requirement. Let me test the implementation to make sure it works correctly according to the specification.\n\n", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_52281cb45883498b", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py set name John\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1339, "completion_tokens": 68, "total_tokens": 1407, "prompt_tokens_details": {"cached_tokens": 1296}, "cache_read_input_tokens": 1296}, "cost": {"usd": 0, "diem": 0.00016889}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 5, "tool": "run_shell", "args": {"command": "python3 kv.py set name John"}, "result": "exit=0\nstdout:\nOK\n\nstderr:\n"}
{"turn": 6, "response": {"service_tier": "default", "id": "chatcmpl-R7rp7iT8gtHapFEeeRyWAyBq", "object": "chat.completion", "created": 1792022437, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_b61ddfd805045364", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py get name\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1430, "completion_tokens": 28, "total_tokens": 1458, "prompt_tokens_details": {"cached_tokens": 1376}, "cache_read_input_tokens": 1376}, "cost": {"usd": 0, "diem": 0.00011594}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 6, "tool": "run_shell", "args": {"command": "python3 kv.py get name"}, "result": "exit=0\nstdout:\nJohn\n\nstderr:\n"}
{"turn": 7, "response": {"service_tier": "default", "id": "chatcmpl-R9TjQAmCz0gxF7VkiSMmRsAn", "object": "chat.completion", "created": 1792022438, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_3789d372025c5559", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py set age 30\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1481, "completion_tokens": 31, "total_tokens": 1512, "prompt_tokens_details": {"cached_tokens": 1440}, "cache_read_input_tokens": 1440}, "cost": {"usd": 0, "diem": 0.00011845}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 7, "tool": "run_shell", "args": {"command": "python3 kv.py set age 30"}, "result": "exit=0\nstdout:\nOK\n\nstderr:\n"}
{"turn": 8, "response": {"service_tier": "default", "id": "chatcmpl-RNsFtYiQJ9EItQmL1CcFH6jY", "object": "chat.completion", "created": 1792022439, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_e31177dceaee2225", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py list\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1535, "completion_tokens": 27, "total_tokens": 1562, "prompt_tokens_details": {"cached_tokens": 1488}, "cache_read_input_tokens": 1488}, "cost": {"usd": 0, "diem": 0.00011647}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 8, "tool": "run_shell", "args": {"command": "python3 kv.py list"}, "result": "exit=0\nstdout:\nage\nname\n\nstderr:\n"}
{"turn": 9, "response": {"service_tier": "default", "id": "chatcmpl-R931Fu6S1fQMFevGzZim92hm", "object": "chat.completion", "created": 1792022441, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_193400aa3e441a86", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py del age\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1587, "completion_tokens": 28, "total_tokens": 1615, "prompt_tokens_details": {"cached_tokens": 1536}, "cache_read_input_tokens": 1536}, "cost": {"usd": 0, "diem": 0.00012129}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 9, "tool": "run_shell", "args": {"command": "python3 kv.py del age"}, "result": "exit=0\nstdout:\nOK\n\nstderr:\n"}
{"turn": 10, "response": {"service_tier": "default", "id": "chatcmpl-Rsz38TVyVmU2LeNgo9MCCMSf", "object": "chat.completion", "created": 1792022442, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_66c5f13b144cbd66", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py list\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1638, "completion_tokens": 27, "total_tokens": 1665, "prompt_tokens_details": {"cached_tokens": 1584}, "cache_read_input_tokens": 1584}, "cost": {"usd": 0, "diem": 0.00012276}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 10, "tool": "run_shell", "args": {"command": "python3 kv.py list"}, "result": "exit=0\nstdout:\nname\n\nstderr:\n"}
{"turn": 11, "response": {"service_tier": "default", "id": "chatcmpl-Rrd2cEk7KD9nN8sqnDf8Auch", "object": "chat.completion", "created": 1792022443, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "", "reasoning_content": null, "name": null, "tool_calls": [{"id": "call_b9484b90d2e29aa8", "type": "function", "function": {"name": "run_shell", "arguments": "{\"command\": \"python3 kv.py get nonexistent\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 1688, "completion_tokens": 28, "total_tokens": 1716, "prompt_tokens_details": {"cached_tokens": 1648}, "cache_read_input_tokens": 1648}, "cost": {"usd": 0, "diem": 0.00012192}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 11, "tool": "run_shell", "args": {"command": "python3 kv.py get nonexistent"}, "result": "exit=1\nstdout:\nNOT FOUND\n\nstderr:\n"}
{"turn": 12, "response": {"service_tier": "default", "id": "chatcmpl-RulvrwrOugdnSRJ39HNGrzYI", "object": "chat.completion", "created": 1792022444, "model": "qwen3-coder-480b-a35b-instruct-turbo", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Excellent! All the functionality works as specified:\n\n1. `set KEY VALUE` prints \"OK\"\n2. `get KEY` prints the value or \"NOT FOUND\" and exits with code 1\n3. `del KEY` prints \"OK\" or \"NOT FOUND\" and exits with code 1\n4. `list` prints keys sorted, one per line\n5. Data is persisted in kv.json\n6. Implementation is under 60 non-blank lines (56 lines)\n7. Uses only Python 3 standard library\n8. Single file implementation\n\nThe implementation fully satisfies the specification requirements.", "reasoning_content": null, "name": null, "tool_calls": []}, "finish_reason": "stop", "logprobs": null}], "usage": {"prompt_tokens": 1740, "completion_tokens": 127, "total_tokens": 1867, "prompt_tokens_details": {"cached_tokens": 1696}, "cache_read_input_tokens": 1696}, "cost": {"usd": 0, "diem": 0.00027373999999999996}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}

Get new runs and ranking changes weekly