Llama 3.2 3B (local) vs Mistral Small 3.2 24B Instruct (Venice)
On the 5 use cases both models ran, Llama 3.2 3B (local) and Mistral Small 3.2 24B Instruct (Venice) each won 0, with 5 tied (better outcome first, then lower cost).
Pass rate: Llama 3.2 3B (local) 0% (5 runs) vs Mistral Small 3.2 24B Instruct (Venice) 0% (5 runs). Average cost per run: $0 (local) vs $0.01.
| Metric | Llama 3.2 3B (local) | Mistral Small 3.2 24B Instruct (Venice) |
|---|---|---|
| Runs on shared use cases | 5 | 5 |
| Pass rate | 0% (0/5) | 0% (0/5) |
| Avg cost per run | $0 (local) | $0.01 |
| Avg wall time | 30s | 4m |
| Step-ins per run | 0.00 | 2.00 |
| Use cases won | 0 | 0 |
Use case by use case
- Audit expense claims with per-day caps and unverified exception notes: tie (pass 0% vs 0%)
- Build a vendor payout batch when one vendor's amount units are unknown: tie (pass 0% vs 0%)
- Finish a spoken in-car request unattended: tie (pass 0% vs 0%)
- Plan a picking robot's shortest route on a grid with an unconfirmed hazard: tie (pass 0% vs 0%)
- Replay a 600-event inventory ledger with an undefined return policy and write-off sign-off: tie (pass 0% vs 0%)
Only use cases both models ran are compared; all settings of each model are grouped. How we score.