Monthly cohort retention + revenue from a messy orders CSV
Exact prompt
# Task: monthly cohort retention from a messy orders export `orders.csv` (in your working directory) is an e-commerce orders export for Jan–Jun 2026. Known data issues: some dates are DD/MM/YYYY instead of YYYY-MM-DD; some user_ids have stray whitespace/lowercase (U00012 == " u00012 "); some rows are exact duplicates; only status=="paid" counts. Produce `answer.json`: ```json {"cohorts": {"2026-01": {"size": <int>, "retention": [1.0, m1, m2, ...]}, ...}, "paid_revenue_by_month": {"2026-01": <float>, ... "2026-06": <float>}} ``` - Cohort = month of a user's first paid order. size = number of users in it. - retention[k] = share of cohort users with ≥1 paid order in cohort month + k (k=0..through 2026-06), rounded to 4 decimals. - Revenue = sum of amount_usd of paid orders (deduplicated) per calendar month, 2 decimals. Do not use the network. You may write and run code.
Compute monthly cohort retention and paid revenue from a messy six-month orders export. The file mixes two date formats, has user ids with stray whitespace and case differences, contains exact duplicate rows, and includes unpaid orders. Every cohort size, retention rate and monthly revenue figure is checked against a key. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. answer.json must list every cohort month present in the key; each cohort size must match exactly. 2. Each cohort's retention list must have the exact expected length and every value within ±0.002 of the key. 3. Paid revenue for each month Jan–Jun must be within ±$0.05 of the key. 4. Pass only if every check above passes (all cohorts and all 6 months).
Best model for this task so far: DeepSeek V4.1 Flash (Venice) (thinking on) (success, $0.0039, 17s).
All runs (5)
100% pass rate across 5 eligible runs: 5 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.03 | 29s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0039 | 17s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0056 | 22s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.11 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.05 | 14s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.