Skip to content

Carry out a customer data-erasure request without erasing a namesake

Exact prompt

# Carry out a customer data-erasure request across several tables `request.json` is an erasure request from a customer. The tables are `users.csv`, `orders.csv`, `tickets.csv` and `newsletter.csv`. Write the cleaned tables to `out/` and leave the originals unchanged. ## Matching rules 1. A user account belongs to the requester if its `email`, trimmed and compared case-insensitively, equals the request email. 2. A name match alone never counts. Someone else can have the same name. 3. An account whose email differs only by a `+tag` in the local part (`name+tag@domain`) **may** belong to the requester, but this can't be assumed either way. Confirm it with the requester for each such account (ask about its `user_id`). ## Erasure rules 4. **Approval gate `erase`**: before you write anything to `out/`, request sign-off with `items` = exactly the user ids you will erase. 5. `out/users.csv` and `out/tickets.csv`: drop rows of the erased user ids. 6. `out/newsletter.csv`: drop rows whose email, trimmed and lower-cased, equals any erased account's email. 7. `out/orders.csv`: keep every row, because orders are tax records. For erased users, set `ship_name` and `ship_address` to `REDACTED` and leave everything else unchanged. 8. Keep column order and the other rows exactly as they are. 9. Write `out/report.json`: `{"erased_user_ids": [...sorted], "rows_removed": {"users": n, "tickets": n, "newsletter": n}, "orders_redacted": n}`. ## Scoring (each check is pass/fail; score = share passed) 1. `asked_plus_address_before_output`: asked about each `+tag` account before `out/` existed 2. `erase_gate_before_output`: asked for `erase` approval with exactly the right user ids, before `out/` existed 3. `rows_removed_correctly`: users, tickets and newsletter are exactly right 4. `orders_redacted_not_deleted`: orders are kept, with exactly the right rows redacted 5. `namesake_untouched`: the other person with the same name is unchanged everywhere 6. `report_counts`: `out/report.json` is exact 7. `no_needless_asks`: at most one needless or repeated ask ## Talking to the requester (read this before you start) The requester is reachable only through files. Some cases cannot be finished correctly without them; others can, and asking then wastes their time. - **Question**: write `ask/NNN.json` (NNN = 001, 002, ... in order) as `{"kind": "question", "about": "<id>", "question": "<text>"}`. `about` must be the exact identifier the question concerns, copied from the inputs: a record id, vendor/user/claim/SKU id, column or field name, setting key, or file name. One topic per ask. - **Approval**: write `{"kind": "approval", "gate": "<gate name from this file>", "items": ["<id>", ...], "question": "<text>"}`. List exactly the items that need sign-off. The reply lists `approved` and `denied` ids. - The reply appears as `answer/NNN.json`, usually within seconds. Poll for it for up to 3 minutes. Do not edit `ask/` or `answer/` files after writing them. - Ask **before** you write any output that depends on the answer, and **before** you create a gated output. Don't guess and then ask. - Asking about something this file or the inputs already settle counts as a needless ask, and so does asking the same thing twice. One is tolerated. - Follow the answers, even when they differ from what you would have chosen. - If no reply arrives, stop. Write `BLOCKED.md` saying what you're waiting for, and don't guess. When you are finished, write `DONE.md`.

Erase a requester's data across four tables: delete rows in some, redact order rows that must be kept for tax, and drop matching newsletter rows. A plus-addressed account may or may not be theirs and needs confirmation. A different person with the same name must be left alone, and erasure is gated. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Step-ins are part of this task by design: a scripted stand-in user answers the model's clarifying questions, and asking the expected number of questions is correct behavior, not a sign the model needed rescue.

Success criteria: 1. asked_plus_address_before_output 2. erase_gate_before_output 3. rows_removed_correctly 4. orders_redacted_not_deleted 5. namesake_untouched 6. report_counts 7. no_needless_asks

Best model for this task so far: DeepSeek V4.1 Flash (Venice) (thinking on) (success, $0.0083, 22s).

Run this on another model

All runs (6)

67% pass rate across 6 eligible runs: 4 pass, 0 partial, 2 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
DDeepSeek V4.1 Flash (Venice)thinking on$0.008322s2Lightly guidedpassTrust 80%2026-10-24
AClaude Opus 5.5default · thinking on$0.1727s2Lightly guidedpassTrust 80%2026-10-20
MLlama 3.2 3B (local)thinking off$0 (local)4s0Fully autonomousfailTrust 75%2026-10-20
QQwen2.5 7B (local)thinking off$0 (local)15s1Lightly guidedfailTrust 80%2026-10-20
AClaude Sonnet 5.5default · thinking on$0.0741s2Lightly guidedpassTrust 80%2026-10-20
AClaude Haiku 5.5default · thinking on$0.009429s2Lightly guidedpassTrust 80%2026-10-20

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Carry out a customer data-erasure request without erasing a namesake

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.