Grok 4.3 (Venice) on Agent bake-off: find a premium flight home

XGrok 4.3 (Venice) · venice-mini-agent v1.2 · cloud · 2026-10-15 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Cost: $0.01
Wall time: 8s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

Mock flight search (no real booking). flights.json lists available flights. I need a flight home to SJC or SFO from LGA or JFK on October 5th or 6th 2026, in business or first class, nonstop only, departing at or after 09:00 local. Choose the cheapest flight meeting all constraints. Write choice.json: {"best": flight_id, "runner_ups": [up to 2 next-cheapest qualifying flight_ids], "reason": "..."}.

Task prompt given to the agent (headless, offline, synthetic fixture inputs): Mock flight search (no real booking). flights.json lists available flights. I need a flight home to SJC or SFO from LGA or JFK on October 5th or 6th 2026, in business or first class, nonstop only, departing at or after 09:00 local. Choose the cheapest flight meeting all constraints. Write choice.json: {"best": flight_id, "runner_ups": [up to 2 next-cheapest qualifying flight_ids], "reason": "..."}. Inspired by https://x.com/twid/status/2101789185598247390 (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost is real Venice spend (paid from DIEM credit), as reported by Venice.

Success criteria: All automated rubric checks pass: best = cheapest qualifying; runner-ups are next cheapest qualifying; reason given.

Setup notes: Harness: venice-mini-agent v1.2 via Venice OpenAI-compatible API. Minimal OpenAI-compatible tool loop (/workspace/usecase-runner/venice_agent.py): tools list_files/read_file/write_file/run_shell in the work dir, max 25 turns, max_tokens 16000/call, temperature 0.2, Venice system prompt and web search off, $0.08 per-run cost cap. Identical for all Venice models. Cost source: Venice /api/v1/models model_spec.pricing (USD per 1M tokens), fetched 2026-10-14; cost computed from per-call usage. Paid from Bob's DIEM daily allowance. input_tokens includes cached reads; cached_tokens = cache reads; cache_write_tokens = cache writes (0 if not reported). Single run, no retries; scored automatically (3/3 checks). Runner notes: Adapted: live flight search replaced by mock flights.json with one correct answer; no booking. Single run, no retries. Same rubric/score.py as Claude runs. Web disabled.

Metrics

Cost: $0.01
Wall time: 8s
Human step-ins: 0
Turns: 3

Model usage
Model	Input tok	Output tok	Cost
XGrok 4.3 (Venice)venice/grok-4-3	10,817	795	$0.01

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Grok 4.3 (Venice) (venice/grok-4-3) via venice-mini-agent v1.2, cloud.
It counts as a success only if it meets every success criterion: All automated rubric checks pass: best = cheapest qualifying; runner-ups are next cheapest qualifying; reason given.
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

{"turn": 1, "response": {"id": "0bf7316f-f7c8-9505-8bd3-6b05d0417ddd", "object": "chat.completion", "created": 1792023480, "model": "grok-4-3", "choices": [{"index": 0, "message": {"role": "assistant", "content": null, "refusal": null, "reasoning_content": "Beginning a mock flight search.\n\n[encrypted reasoning omitted, 645 chars]", "reasoning_encrypted": true, "reasoning_details": [{"type": "reasoning.encrypted", "format": "xai-responses-v1", "index": 0, "id": "rs_0bf7316f-f7c8-9505-8bd3-6b05d0417ddd", "data": "dGdL0DTd9Xj57nSn9nSI4nIX4NOqwA2aCXQF0q47IQI0clAST2tD+Fs57eRJrQCJMhuFYLYcFMP29JurNRRP2SRXD+ymzcSaQJh9NQhNMw7goBPygtTwsj6QIciCCV7yil62kQ6YvWOFF6JdMxHisjo4JtspXk/t/yRoT+m7LVVgLc7A0lGv16MVM26IzBTybcjuCv9d7vUgWn2SERn9G8Vv4IyxqEcIYXnUoE8NFBTtGFWpvcdw3+HyMRvThvJJWf4mvS+segIrPi7caJ+si3rUR84khPKoAl0gUfmxQY6UNkqrae0HGAPlf2FE2qV005hUjL7XvfySkuyZe+87ncbUFLVAGei12eCE5kjJJaTPyZR2dNq7pyiy0fLv0PdUFfapKbbz2+ynUnc91CCS6Fe5ijGhMj+O9ZKWyd3ATEcjiSn6RYMK0JSyUv5zbwzAwTJ4hQ9+QbcyEwGo41MVLZJkCcC/B+0ppk79xIanS/riIuWpGFfFFPSLx3mA7furRkC9SSzY/4NfY70Cwjyrw/uQF5XwChdCUrcpMqaXPH3b4CX1OlT9Fr0mnE+GJc+B4NNgQgt2lwOLDI7XWybw9YadhpbvvqohxjACp0Ca/q+lqg"}], "tool_calls": [{"id": "call-5e834751-1298-443c-a57c-e609e2caf7b9-0", "type": "function", "function": {"name": "read_file", "arguments": "{\"path\":\"flights.json\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 584, "completion_tokens": 120, "total_tokens": 704, "prompt_tokens_details": {"cached_tokens": 128}, "cache_read_input_tokens": 128, "completion_tokens_details": {"reasoning_tokens": 108}}, "cost": {"usd": 0, "diem": 0.00101656}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 1, "tool": "read_file", "args": {"path": "flights.json"}, "result": "[\n  {\n    \"id\": \"F001\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"JFK\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"economy\",\n    \"stops\": 1,\n    \"depart_local\": \"09:30\",\n    \"duration_min\": 493,\n    \"price_usd\": 323,\n    \"airline\": \"Alpha Air\"\n  },\n  {\n    \"id\": \"F002\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"JFK\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"premium_economy\",\n    \"stops\": 0,\n    \"depart_local\": \"09:00\",\n    \"duration_min\": 370,\n    \"price_usd\": 985,\n    \"airline\": \"Delta Mock\"\n  },\n  {\n    \"id\": \"F003\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"JFK\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"business\",\n    \"stops\": 0,\n    \"depart_local\": \"16:30\",\n    \"duration_min\": 372,\n    \"price_usd\": 1687,\n    \"airline\": \"Alpha Air\"\n  },\n  {\n    \"id\": \"F004\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"JFK\",\n    \"dest\": \"OAK\",\n    \"cabin\": \"business\",\n    \"stops\": 1,\n    \"depart_local\": \"13:00\",\n    \"duration_min\": 445,\n    \"price_usd\": 1346,\n    \"airline\": \"Alpha Air\"\n  },\n  {\n    \"id\": \"F005\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"JFK\",\n    \"dest\": \"OAK\",\n    \"cabin\": \"first\",\n    \"stops\": 1,\n    \"depart_local\": \"15:30\",\n    \"duration_min\": 508,\n    \"price_usd\": 2712,\n    \"airline\": \"Bay Jet\"\n  },\n  {\n    \"id\": \"F006\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"LGA\",\n    \"dest\": \"SFO\",\n    \"cabin\": \"business\",\n    \"stops\": 0,\n    \"depart_local\": \"13:00\",\n    \"duration_min\": 337,\n    \"price_usd\": 1263,\n    \"airline\": \"Coastal\"\n  },\n  {\n    \"id\": \"F007\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"LGA\",\n    \"dest\": \"SFO\",\n    \"cabin\": \"first\",\n    \"stops\": 1,\n    \"depart_local\": \"16:30\",\n    \"duration_min\": 439,\n    \"price_usd\": 3103,\n    \"airline\": \"Coastal\"\n  },\n  {\n    \"id\": \"F008\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"LGA\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"premium_economy\",\n    \"stops\": 0,\n    \"depart_local\": \"18:00\",\n    \"duration_min\": 418,\n    \"price_usd\": 628,\n    \"airline\": \"Delta Mock\"\n  },\n  {\n    \"id\": \"F009\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"LGA\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"business\",\n    \"stops\": 0,\n    \"depart_local\": \"07:00\",\n    \"duration_min\": 385,\n    \"price_usd\": 1426,\n    \"airline\": \"Bay Jet\"\n  },\n  {\n    \"id\": \"F010\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"LGA\",\n    \"dest\": \"OAK\",\n    \"cabin\": \"economy\",\n    \"stops\": 0,\n    \"depart_local\": \"06:30\",\n    \"duration_min\": 416,\n    \"price_usd\": 897,\n    \"airline\": \"Delta Mock\"\n  },\n  {\n    \"id\": \"F011\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"EWR\",\n    \"dest\": \"SFO\",\n    \"cabin\": \"economy\",\n    \"stops\": 1,\n    \"depart_local\": \"11:00\",\n    \"duration_min\": 478,\n    \"price_usd\": 720,\n    \"airline\": \"Coastal\"\n  },\n  {\n    \"id\": \"F012\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"EWR\",\n    \"dest\": \"SFO\",\n    \"cabin\": \"premium_economy\",\n    \"stops\": 1,\n    \"depart_local\": \"09:00\",\n    \"duration_min\": 462,\n    \"price_usd\": 763,\n    \"airline\": \"Alpha Air\"\n  },\n  {\n    \"id\": \"F013\",\n    \"date\": \"2026-10-05\",\n    \"origin\": \"EWR\",\n    \"dest\": \"SJC\",\n    \"cabin\": \"business\",\n    \"stops\": 1,\n    \"depart_lo"}
{"turn": 2, "response": {"id": "b4df40da-837f-9f7b-af58-46eeb71b316f", "object": "chat.completion", "created": 1792023481, "model": "grok-4-3", "choices": [{"index": 0, "message": {"role": "assistant", "content": null, "refusal": null, "reasoning_content": "I need to find the cheapest nonstop flight from LGA or JFK to SJC or SFO on October 5th or 6th, 2026. It must be business or first class and depart at or after 09:00 local time.\n\nThree qualifying nonstop flights were found: F003 from JFK to SJC at 16:30, F006 from LGA to SFO at 13:00, and F017 from JFK to SFO at 10:00. The cheapest is F006 at $1263.\n\n[encrypted reasoning omitted, 1615 chars]", "reasoning_encrypted": true, "reasoning_details": [{"type": "reasoning.encrypted", "format": "xai-responses-v1", "index": 0, "id": "rs_b4df40da-837f-9f7b-af58-46eeb71b316f", "data": "aE8fWH/T9shQ6Od99og+jaS3RNHMmGm4QBSw8S7/YL/ZregshUNqPImLiQqKjNau0QDPBmk1hOWHxW1Nkye/dh2CMBWEf8m/J9QM/XHlVpEr32HQyGHeeOCkk9Hkp4dYlhWp2tCGowH+/dZTWnsWDL3xQsS9A35CMhOtDv2/MeHonPJZ6ZQe8AnXOMBo6yE+ZCaf3wma+C9Tz7uSHySUVf5CDpILeXWrQgiDHQsTn94yHALAjxIDQ1YkKRp7revOqZ7LPNCiCmujorVZwQuFW5dKaRlVkO362SU9U+reKmDHCty0VbLZd6/nwh2ScLaYqRIBhU5LDGrmi4m6AeLNZRMp3LTdUk3w6GwVMjvSnuig3stCtS4BoDmfyc33o51ivdZhnD8stNxO53u7Vvrn3K4PW3wA0BYaG/qD+fD53LhXBew8B31zTr24mRIvgnok9wF/Ph594q+wdi+q//P4FLBUdnjC4gLhxjmDOVJRMbom8BtUBRET7azRQmr6jBttN9sT8tvuY8wAJRR4g5B4rO+c+Ybc+zS0/T2jEwzeeB6CYoR7TuAPI9PuEpSxv/FDZs5mR/xc/Tk5XU9LP8lXpXxBhSbBio+LuzTl6FtMqWFwEx/MAxiw/W3bJl8EtbPjmKMQeTAMJ5h33+sFPjEA0c345lRuLQ6CQPngHAQ4/DAq6Jp4s+SOIwT3Kiq1MrV/ilFHlNq+ewFkL/mo/GJrtCRl4WWKu6UB8443kTx9PGwYGg6xS1feZtymDUPTcApmxnfCx839GzqvDt6pZ6PF+WDB8/76aItWiQiKCweA9i4+jdZPqeCLqIfT6wpVRrch3R+f0yJ/1G4eH6IveV9+NIserwadYNH+03sbXpa6gxb7lVI9vMuhjwpSnN8MDu+0LwAI5BD5LSIlQkLZNZutDR11LOAUbh+PRSxDwKDrxfJFnm7OMGdT1Eez2Y9IChSdtceMkQclRW4izqqGWULQQKaq7AykwZPFt9NVDvVeYiThOK5JR4bOUWJtWpYk5T2Cg6jSjOXpsSIe80OyDRAuUmpCyYoGH4m06pcgJoh6FKqp/1F8hDBxh9vkhv6/bw5/n6T6NmBPuvAsQNxQA8UL0Ww26bvgAqzFBYt3fh5Fqgm5JijIR+N1jRT66/OVJ8dUE/VyjhhmHAT0VeBlb3C/OXOO4kndMCnAkqQ+kAdEIowSFnCDA4Mu3CldwJZo+XRnjSx0dYybOekjzni30Jpi1RnNm57UTUKRcwDaPP4LLz9Eu2OZkXX5hl5hTzUb2BoZVWIEvTyOOZ6NFIr4sNFqdFbRszwSNvLRHjwWN6Iba75YVZ0lBvlLZSmis9Wl2HguF59apYxwZvGWv/2y//g7u1oYD7kVzXMKsDEBxVV1rxkCQqV0SZxBsIm58cwOBDZlI8DXJPYZ7CpxQjo1c8aK0tT+Bc9iSQ3k80/r6otAoXtDFjPhy6wWZunSu62VNfxUMuL6Kd4rl+Ib51tSB9zEpGQS5RL7ZK+HicNt2OuEzgjBVfalUQCKJlTl6F1MYuH/DCJShAdK/tzRtnAAnj7XFOcmtquh8y/GEpXKuRyViK05CtDcqB0ZQE++"}], "tool_calls": [{"id": "call-9822b481-2f8d-4a3f-8cf6-c2741fd4d015-1", "type": "function", "function": {"name": "write_file", "arguments": "{\"path\":\"choice.json\",\"content\":\"{\\\"best\\\": \\\"F006\\\", \\\"runner_ups\\\": [\\\"F025\\\", \\\"F017\\\"], \\\"reason\\\": \\\"Cheapest qualifying: LGA-SFO business nonstop 13:00 on 2026-10-05 for $1263. Next: LGA-SFO 21:00 $1395, JFK-SFO 10:00 $1532.\\\"}\"}"}}]}, "finish_reason": "tool_calls", "logprobs": null}], "usage": {"prompt_tokens": 5066, "completion_tokens": 517, "total_tokens": 5583, "prompt_tokens_details": {"cached_tokens": 128}, "cache_read_input_tokens": 128, "completion_tokens_details": {"reasoning_tokens": 426}}, "cost": {"usd": 0, "diem": 0.008504510000000002}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}
{"turn": 2, "tool": "write_file", "args": {"path": "choice.json", "content": "{\"best\": \"F006\", \"runner_ups\": [\"F025\", \"F017\"], \"reason\": \"Cheapest qualifying: LGA-SFO business nonstop 13:00 on 2026-10-05 for $1263. Next: LGA-SFO 21:00 $1395, JFK-SFO 10:00 $1532.\"}"}, "result": "wrote 186 chars"}
{"turn": 3, "response": {"id": "b56f80ac-0136-9f41-b0d3-ed69c335b6cd", "object": "chat.completion", "created": 1792023486, "model": "grok-4-3", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Wrote choice.json with cheapest qualifying flight (F006) + 2 runner-ups.", "refusal": null, "reasoning_content": "I filtered flights.json for qualifying nonstop flights.\n\n[encrypted reasoning omitted, 585 chars]", "reasoning_encrypted": true, "reasoning_details": [{"type": "reasoning.encrypted", "format": "xai-responses-v1", "index": 0, "id": "rs_b56f80ac-0136-9f41-b0d3-ed69c335b6cd", "data": "DrtYOXUHPZUq2aMGBv874d4pJPWBDpOk5tDmTlYJ2Husa7mqoMgZxKGdrAqBF07XvEEs9v6wLWprN5HV7sejZPm5ccUSKoXhKTboESD4HdKXQQi1iVjBZXIu8h43Pt4Qk1lefULQmhjc/zsTWK9EFZX9mmeuZ1xcjWr+yHRqMH4QALipSse7jxch3EaoLpXzFWAA5Rumamyl3nBTO4Gs6zw7dn1nK+E4OGAwQWcSqjnJgsrrAmEtWcuTKrJDW5GI9z9ZgGKdj4TH3eNWIewJWq4qFg25gB4Fa5uB7EbfGgVjJJ1c2gBINhLWk1778wWOutsUeB682/OSlTiAgNxKepEiws9mCOAqzQ1/2DpDONUqPOskiUvea3NMofdXHRgNTpWqYq0snG06I4XbuUAgYK/wKpy1C46iv+YaUJL1lukC3ORCZrX1IZVo3fb+e9idK0pNKCM58kBsEKcBs/+rCy15hsdWBRXbJ8POI/AkUC+O2VH1Ow36dqjX7P/FWil/fxKl9sPvJcndWmZaiX2IO+h1LVcsT1WELcRQDhSI2mYQiEyxFQ"}]}, "finish_reason": "stop", "logprobs": null}], "usage": {"prompt_tokens": 5167, "completion_tokens": 158, "total_tokens": 5325, "prompt_tokens_details": {"cached_tokens": 4992}, "cache_read_input_tokens": 4992, "completion_tokens_details": {"reasoning_tokens": 140}}, "cost": {"usd": 0, "diem": 0.0018438}, "venice_parameters": {"strip_thinking_response": false, "disable_thinking": false, "enable_e2ee": true, "enable_web_search": "off", "enable_web_scraping": false, "enable_web_citations": false, "include_search_results_in_stream": false, "include_venice_system_prompt": false, "enable_x_search": false, "web_search_citations": [], "return_search_results_as_documents": false}}}