Entity resolution on ~950 merged customer records (nicknames, swaps, aliases, phone formats) without merging households
Exact prompt
# Task: deduplicate the customer list `customers.csv` (~950 rows) was merged from several systems, so many real customers appear 2-4 times with inconsistent data: nicknames (Bob/Robert), typos, initials, first/last name swapped, case differences, email aliases (`+tag`) or missing emails, phone numbers in many formats or missing, street suffix abbreviations (Street/St), and so on. Watch out: members of the same household can share a phone number and address but are different people, and different people can share a common name. Write `dedupe.py` (Python stdlib only) that reads `customers.csv` from the current directory and writes `clusters.json`: a JSON list of clusters, each a list of record `id`s that belong to the same person. Every id must appear in exactly one cluster (singletons included). Run it. Grading is by pairwise precision and recall (over all pairs of records placed in the same cluster) against the true clusters (both must be >= 0.95), and `dedupe.py` is re-run on a freshly generated file of the same kind, so don't hard-code ids. No network.
Email-only matching gets ~0.59 recall; adding phone gets ~0.84 precision because households share phones. Passing both thresholds needs normalisation plus name-compatibility rules (nicknames, initials, transpositions, swapped fields), and the fresh-seed rerun blocks hard-coding. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. clusters_valid_partition: clusters.json lists every record id exactly once. 2. precision_ok: pairwise precision >= 0.95. 3. recall_ok: pairwise recall >= 0.95. 4. fresh_data_ok: dedupe.py re-run on a freshly generated dataset (seed 3434) also gives a valid partition with precision and recall >= 0.95.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.08, 10m).
All runs (4)
75% pass rate across 4 eligible runs: 3 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking on | $0.08 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.08 | 10m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.71 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.09 | 35s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.