Analyze five months of AI coding-agent session logs
Exact prompt
sessions.csv logs 5 months of AI coding-agent sessions. Write answers.json with exactly these keys: "total_sessions" (int), "total_prs" (int), "total_api_equiv_usd" (number rounded to 2 decimals), "busiest_month_by_sessions" (string like "2026-07"), "median_minutes" (number), "usd_per_pr_by_tool" (object tool -> total api_equiv_usd / total prs_opened for that tool, rounded to 2 decimals). Then write REPORT.md with a 5-bullet summary.
Task prompt given to the agent (headless, offline, synthetic fixture inputs): sessions.csv logs 5 months of AI coding-agent sessions. Write answers.json with exactly these keys: "total_sessions" (int), "total_prs" (int), "total_api_equiv_usd" (number rounded to 2 decimals), "busiest_month_by_sessions" (string like "2026-07"), "median_minutes" (number), "usd_per_pr_by_tool" (object tool -> total api_equiv_usd / total prs_opened for that tool, rounded to 2 decimals). Then write REPORT.md with a 5-bullet summary. Inspired by https://x.com/quinn_leng/status/2085828592127336541 (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost shown is a list-price API-equivalent estimate (run on a Claude Pro subscription, not billed).
Success criteria: All automated rubric checks pass: total_sessions correct; total_prs correct; total_api_equiv_usd correct; busiest_month_by_sessions correct; median_minutes correct; usd_per_pr_by_tool correct; REPORT.md written.
Best model for this task so far: DeepSeek V4.1 Flash (Venice) (success, $0.0028, 12s). Best AI model for Software & DevOps →
All runs (8)
88% pass rate across 8 eligible runs: 7 pass, 0 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Haiku 5.5default · thinking on | $0.0061 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| ZGLM 5.3 (Venice) | $0.07 | 29s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| XGrok 4.3 (Venice) | $0.02 | 10s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-15 | |
| QQwen3 Coder 480B Turbo (Venice) | $0.0061 | 16s | 0 | Fully autonomous | fail | Trust 80% | 2026-10-15 | |
| DDeepSeek V4.1 Flash (Venice) | $0.0028 | 12s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 | |
| GGemma 4 31B Instruct (Venice) | $0.0057 | 40s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 | |
| AClaude Opus 5.5 | $0.12 | 21s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 | |
| AClaude Sonnet 5.5 | $0.05 | 14s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-14 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.