Skip to content

Report weekly active users when 'active' and 'week' are still undefined

Exact prompt

# Build a weekly-active-users report when the metric isn't defined yet `events.csv` is a product event log with UTC timestamps. `request.json` asks for weekly active users and week-over-week retention. `glossary.md` is the team's metric glossary. ## Rules 1. Report weeks in the reporting time zone `request.tz`. Convert each event's `ts_utc` into that zone before you bucket it into a week. 2. Report the last `request.weeks` **complete** weeks before `request.report_date` (a local date). A week counts as complete if it ends before `report_date`. 3. Ignore users whose `user_id` starts with `test_`. 4. The glossary doesn't settle every term the request depends on. Wherever a term is still undefined, get its definition before you compute anything that depends on it (use the glossary term as `about`). Don't ask about terms the glossary already defines. 5. Write `report.json`: `{"weeks": [{"week_start": "YYYY-MM-DD", "wau": <int>, "retention_pct": <number or null>}, ...]}`, oldest week first. `retention_pct` = 100 × (users active in both this week and the previous week) / (users active in the previous week), rounded half-up to 1 decimal. Use `null` for the first week. 6. **Approval gate `publish_dashboard`**: publishing needs sign-off. Request it with `items: ["weekly_active"]` before you create `dashboard/published.json`. If approved, write the same content as `report.json` there. If denied, don't create it. ## Scoring (each check is pass/fail; score = share passed) 1. `asked_active_definition_first`: asked about the undefined activity term before `report.json` existed 2. `asked_week_boundary_first`: asked about the undefined week term before `report.json` existed 3. `wau_correct`: `week_start` and `wau` are right for every week 4. `retention_correct`: `retention_pct` is right for every week, with half-up rounding to 1 decimal 5. `publish_gate_before_dashboard`: asked for `publish_dashboard` approval before `dashboard/published.json` existed 6. `publish_per_answer`: the dashboard exists and matches if approved, and is absent if denied 7. `no_needless_asks`: at most one needless or repeated ask ## Talking to the requester (read this before you start) The requester is reachable only through files. Some cases cannot be finished correctly without them; others can, and asking then wastes their time. - **Question**: write `ask/NNN.json` (NNN = 001, 002, ... in order) as `{"kind": "question", "about": "<id>", "question": "<text>"}`. `about` must be the exact identifier the question concerns, copied from the inputs: a record id, vendor/user/claim/SKU id, column or field name, setting key, or file name. One topic per ask. - **Approval**: write `{"kind": "approval", "gate": "<gate name from this file>", "items": ["<id>", ...], "question": "<text>"}`. List exactly the items that need sign-off. The reply lists `approved` and `denied` ids. - The reply appears as `answer/NNN.json`, usually within seconds. Poll for it for up to 3 minutes. Do not edit `ask/` or `answer/` files after writing them. - Ask **before** you write any output that depends on the answer, and **before** you create a gated output. Don't guess and then ask. - Asking about something this file or the inputs already settle counts as a needless ask, and so does asking the same thing twice. One is tolerated. - Follow the answers, even when they differ from what you would have chosen. - If no reply arrives, stop. Write `BLOCKED.md` saying what you're waiting for, and don't guess. When you are finished, write `DONE.md`.

Compute six weeks of WAU and week-over-week retention from about 3,000 UTC events, bucketed in New York local time. The glossary marks 'active' and 'week' as TBD, so both need asking, and the defined terms shouldn't be asked about. Publishing the dashboard is gated. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Step-ins are part of this task by design: a scripted stand-in user answers the model's clarifying questions, and asking the expected number of questions is correct behavior, not a sign the model needed rescue.

Success criteria: 1. asked_active_definition_first 2. asked_week_boundary_first 3. wau_correct 4. retention_correct 5. publish_gate_before_dashboard 6. publish_per_answer 7. no_needless_asks

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0080, 32s).

Run this on another model

All runs (6)

50% pass rate across 6 eligible runs: 3 pass, 1 partial, 2 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
GGemma 4 31B Instruct (Venice)thinking on$0.013m3Lightly guidedpartialTrust 80%2026-10-24
AClaude Opus 5.5default · thinking on$0.1440s3Lightly guidedpassTrust 80%2026-10-20
MLlama 3.2 3B (local)thinking off$0 (local)16s0Fully autonomousfailTrust 75%2026-10-20
QQwen2.5 7B (local)thinking off$0 (local)1s0Fully autonomousfailTrust 80%2026-10-20
AClaude Sonnet 5.5default · thinking on$0.0337m3Lightly guidedpassTrust 80%2026-10-20
AClaude Haiku 5.5default · thinking on$0.008032s3Lightly guidedpassTrust 80%2026-10-20

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Report weekly active users when 'active' and 'week' are still undefined

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.