Agent bake-off: find a premium flight home
Exact prompt
Mock flight search (no real booking). flights.json lists available flights. I need a flight home to SJC or SFO from LGA or JFK on October 5th or 6th 2026, in business or first class, nonstop only, departing at or after 09:00 local. Choose the cheapest flight meeting all constraints. Write choice.json: {"best": flight_id, "runner_ups": [up to 2 next-cheapest qualifying flight_ids], "reason": "..."}.
Task prompt given to the agent (headless, offline, synthetic fixture inputs): Mock flight search (no real booking). flights.json lists available flights. I need a flight home to SJC or SFO from LGA or JFK on October 5th or 6th 2026, in business or first class, nonstop only, departing at or after 09:00 local. Choose the cheapest flight meeting all constraints. Write choice.json: {"best": flight_id, "runner_ups": [up to 2 next-cheapest qualifying flight_ids], "reason": "..."}. Inspired by https://x.com/twid/status/2101789185598247390 (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost shown is a list-price API-equivalent estimate (run on a Claude Pro subscription, not billed).
Success criteria: All automated rubric checks pass: best = cheapest qualifying; runner-ups are next cheapest qualifying; reason given.
Best model for this task so far: DeepSeek V4.1 Flash (Venice) (success, $0.0033, 8s). Best AI model for Travel & Bookings →
All runs (7)
86% pass rate across 7 eligible runs: 6 pass, 0 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Haiku 5.5default · thinking on | $0.0040 | 9s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| XGrok 4.3 (Venice) | $0.01 | 8s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-15 | |
| QQwen3 Coder 480B Turbo (Venice) | $0.0029 | 11s | 0 | Fully autonomous | fail | Trust 80% | 2026-10-15 | |
| DDeepSeek V4.1 Flash (Venice) | $0.0033 | 8s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-15 | |
| GGemma 4 31B Instruct (Venice) | $0.0034 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 | |
| AClaude Opus 5.5 | $0.11 | 15s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 | |
| AClaude Sonnet 5.5 | $0.04 | 10s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.