Claude Haiku 5.5 vs Gemma 4 31B Instruct (Venice)
On the 42 use cases both models ran, Gemma 4 31B Instruct (Venice) won 25 to 17 (better outcome first, then lower cost).
Pass rate: Claude Haiku 5.5 98% (42 runs) vs Gemma 4 31B Instruct (Venice) 86% (42 runs). Average cost per run: $0.0082 vs $0.0070.
| Metric | Claude Haiku 5.5 | Gemma 4 31B Instruct (Venice) |
|---|---|---|
| Runs on shared use cases | 42 | 42 |
| Pass rate | 98% (41/42) | 86% (36/42) |
| Avg cost per run | $0.0082 | $0.0070 |
| Avg wall time | 52s | 5m |
| Step-ins per run | 0.10 | 0.10 |
| Use cases won | 17 | 25 |
Use case by use case
- Agent bake-off: find a premium flight home: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Analyze five months of AI coding-agent session logs: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Apply a night's DNS change queue with a production cut-over, two clashing requests and a request to switch off a safeguard: Claude Haiku 5.5 better (pass 0% vs 0%)
- Backtest a moving-average crossover on a bundled synthetic price file: Claude Haiku 5.5 better (pass 100% vs 100%)
- Best fishing windows from tide, solunar and weather data: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Best sailing angle and leg times from a polar table: Claude Haiku 5.5 better (pass 100% vs 0%)
- Boat fuel range with a reserve over a planned trip: Claude Haiku 5.5 better (pass 100% vs 100%)
- Build a 3D fishing game in three.js: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Build a 3D Sonic-style platformer in three.js: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Build a 4-page bakery-cafe website (HTML/CSS): Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Build a full game (code, art, audio) with AI: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Build a landing page from extracted design tokens: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Call a car rental company to add an infant seat: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Catch log checked against a made-up fishing regulations table: Claude Haiku 5.5 better (pass 100% vs 0%)
- Check airline crew rosters against duty, rest, rolling-window and consecutive-day limits: Claude Haiku 5.5 better (pass 100% vs 0%)
- Choose the value-maximising project portfolio under a budget, an hours cap, prerequisites and exclusions: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Diagnose why GitHub Actions stopped running: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Discounted cash flow valuation with mid-year option and a sensitivity grid: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Fill and submit job applications from a resume: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Fix three bugs in a Python repo so its tests pass: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Give an agent $10 to spend in an agent tool store: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Go/no-go departure time from a gridded wind file: Claude Haiku 5.5 better (pass 100% vs 100%)
- Implement 5 Python functions from a task list: Claude Haiku 5.5 better (pass 100% vs 100%)
- Implement a spec exactly, with nothing extra: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Let agents run growth for a meeting-notetaker app: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- One-shot a Minecraft-style voxel game in three.js: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Post-stratified results for a two-wave customer survey with repeat respondents and mixed income units: Claude Haiku 5.5 better (pass 100% vs 100%)
- Progressive tax on fictional brackets, and pre-tax vs after-tax savings comparison: Claude Haiku 5.5 better (pass 100% vs 100%)
- Read a synthetic annual-report excerpt and compute financial ratios: Claude Haiku 5.5 better (pass 100% vs 100%)
- Rebalance to target weights with whole shares and highest-cost-first tax lots: Claude Haiku 5.5 better (pass 100% vs 0%)
- Report weekly active users when 'active' and 'week' are still undefined: Claude Haiku 5.5 better (pass 100% vs 0%)
- Required minimum distributions from a supplied divisor table, with and without delaying the first one: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Research use cases and build pitch slides: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Resolve merge conflicts in a set of config files using written precedence rules: Claude Haiku 5.5 better (pass 100% vs 100%)
- Reverse engineer a security patch for an exploited CVE: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Run business WhatsApp customer messages with an agent: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Run store-operations tasks on a mock e-commerce store: Claude Haiku 5.5 better (pass 100% vs 100%)
- Seeded Monte Carlo retirement odds against a stated success threshold: Claude Haiku 5.5 better (pass 100% vs 100%)
- Test AI browser agents on real workday tasks: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Text World Cup final scores to a Signal group: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Turn weekly ad-platform exports into an exact wasted-spend report: Gemma 4 31B Instruct (Venice) better (pass 100% vs 100%)
- Work backward from a retirement budget to a savings target, contributions and earliest retirement age: Claude Haiku 5.5 better (pass 100% vs 100%)
Only use cases both models ran are compared; all settings of each model are grouped. How we score.