Sort a household's texts and emails into scams and genuine messages before anything is paid, without crying wolf
Exact prompt
# Sort a household's texts and emails into scams and genuine messages before anything is paid, without crying wolf Inputs: `inbox.json` (a list of messages: id, channel, sender, sender_name, received, subject, body), `my-accounts.json` (the organisations the household really deals with: name, domains, phones, and the account, order or parcel references it holds with each), `contacts.csv` (name, phone, email) and `rules.csv` (rule_id, rule). Every person, organisation, domain and number is synthetic: domains end in `.example` and phone numbers are 555-01xx test numbers. These rules are the household's; where `neighbour-advice.txt` disagrees, the rules win. Rules: - Claimed sender: a message claims to be from the organisation or contact named in its sender_name; if sender_name is blank, from the one named in the body's first sentence (if any). An address or link belongs to an organisation when its domain is one of the organisation's domains or a subdomain of one (`pay.x.example` belongs to `x.example`; `x.example.pay-now.example` does not). - Scams: a message is a scam when it breaks any rule in rules.csv: R1 it claims to be from an organisation in my-accounts.json or a contact in contacts.csv but its sender is not one of their domains, phones or email addresses; R2 it claims to be from an organisation in my-accounts.json and contains a link that does not belong to that organisation; R3 it asks you to send, tell, reply with or enter a password, PIN or one-time code (a message that only gives you a code, or warns you never to share one, does not); R4 it asks you to pay with gift cards or crypto, or to transfer money to an account it gives; R5 it claims to be from an organisation in my-accounts.json and refers to an account, order or parcel reference that organisation doesn't have in my-accounts.json; R6 it claims to be from an organisation that is not in my-accounts.json and asks you to pay or to open a link. Every other message is genuine. - Scoring: missed scams and false alarms are scored separately, so marking everything a scam fails as surely as marking nothing. - Reasons: each scam's rule is the lowest-numbered rule it breaks. - Write `out/screening.json`: `{"verdicts": {"<message_id>": "scam" | "genuine", ...}, "scam_rule": {"<message_id>": "<rule_id>", ...}, "scams_by_rule": {"<rule_id>": <number of scams with that rule>, ...}}`. `verdicts` has one entry per message in inbox.json and `scam_rule` one entry per message you mark as a scam. - Roll-ups: Rules in rules.csv that no scam breaks are included at 0. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Screen a synthetic household inbox of texts and emails against six written scam rules: wrong sender, a link outside the claimed organisation's domains, asking for a code or password, gift-card or transfer payments, unknown references, and unknown organisations asking for money. Look-alike domains and new-number messages sit next to genuine code texts, never-share warnings, FINAL REMINDER bills and gift-card balances. Missed scams and false alarms are scored separately. All domains are reserved .example names and all numbers are 555-01xx test numbers. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: no_missed_scams no_false_alarms scam_rules scams_by_rule
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0063, 23s).
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.12 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 | |
| AClaude Haiku 5.5default · thinking on | $0.0063 | 23s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 | |
| AClaude Sonnet 5.5default · thinking on | $0.06 | 16s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-25 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.