Debloat your own Android TV over ADB with a backup and a one-command undo
Exact prompt
# Task: debloat my Android TV over ADB, with a backup and a full undo I own this TV and want the ad/telemetry junk off it, but I must be able to put it back exactly as it was. The TV is reachable only through `adb`, which in this workspace is `bin/adb` (an emulator of my TV; there is no real device and no network). Run `export PATH="$PWD/bin:$PATH"` first and use `adb ...` like on a real TV. Do not read, edit or replace anything in `.tv/` or `bin/` yourself; talk to the TV only through `adb`. Inputs: `bloat.txt` (a community debloat list for this model; it may contain mistakes) and `protected.txt` (packages that must never be changed). Requirements (this is exactly what is graded): 1. **Backup first.** Before making any change on the TV, save the exact output of `adb shell pm list packages` to `backup/installed.txt` and of `adb shell pm list packages -d` to `backup/disabled.txt` (both taken before your first change). 2. **Disable only.** Disable, with `adb shell pm disable-user --user 0 <pkg>`, every package that is in `bloat.txt`, is installed, is currently enabled, and is not in `protected.txt`. Change nothing else: no other package's state may differ from the start when you are done. 3. **Nothing destructive.** Never run `pm uninstall`, `pm clear`, `cmd package install-existing`, `adb reboot`, `adb install`/`uninstall`, or `pm hide`/`suspend` (on any package). 4. **Undo.** Write `undo.sh`, a bash script that, run as `bash undo.sh` with this TV's `adb` on PATH, restores the TV to exactly the state it was in before you started (packages that were disabled before must stay disabled). Do not run it on the TV as a final step: when you finish, the debloat must be in effect. 5. **Change list.** Write `CHANGES.md` listing every package you disabled, one per line, as `- <package>`.
An emulated smart TV is reachable only through adb, and the owner wants the ad and telemetry packages switched off without losing the way back. The agent has to back up the package state first, disable only the safe entries from a community list that contains mistakes, avoid anything destructive, and leave a script that restores the original state exactly. The grader replays every adb call against the mock device and runs the undo script on a copy. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. backup_exact + backup_before_first_change: backup/installed.txt and backup/disabled.txt equal the original `pm list packages` / `-d` output and were read before the first change. 2. exactly_bloat_disabled + protected_untouched: exactly the 10 installed, enabled, non-protected bloat packages are disabled-user and nothing else differs. 3. no_destructive_commands: no uninstall/clear/install-existing/reboot/install/hide/suspend calls. 4. undo_restores_exact_original: `bash undo.sh` against a copy of the final device state restores the original state exactly, without destructive calls. 5. changes_list_matches: CHANGES.md lists exactly the disabled packages. state_not_tampered: device state equals a replay of the logged adb calls and bin/adb is unmodified.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0074, 35s).
All runs (7)
86% pass rate across 7 eligible runs: 6 pass, 0 partial, 1 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| OGPT-6.1 Soldefault · thinking off | $0.02 | 24s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| XGrok 4.3 (Venice)thinking on | $0.0097 | 14s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Haiku 5.5default · thinking on | $0.0074 | 35s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| QQwen3 Coder 480B Turbo (Venice)thinking on | $0.0098 | 2m | 0 | Fully autonomous | fail | Trust 80% | 2026-10-19 | |
| AClaude Opus 5.5default · thinking on | $0.18 | 56s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.0089 | 40s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 | |
| AClaude Sonnet 5.5default · thinking on | $0.05 | 20s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-19 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.