Close a path-traversal hole (encodings, backslashes, NUL, symlinks) in a stdlib static file server
Exact prompt
# Task: close the path-traversal hole in the docs server `server.py` serves files from `public/`. A security report says it can be made to return files outside `public/` (for example `config.env` next to it). Fix it. Requirements: - No request may ever return the contents of a file outside `public/` - whatever the encoding (`..`, percent-encoding incl. double encoding, backslashes, absolute paths, NUL bytes) and including via symlinks inside `public/` that point outside it. Such requests get 404 (or 400/403) and never a 5xx or a crash. - Legitimate files keep working exactly as before: `/` and `/docs/readme.txt`, percent-encoded names like `/docs/my%20file.txt`, query strings ignored, correct Content-Type. - Directories without index.html return 404. Symlinks inside `public/` that point to files *inside* `public/` still work. - Keep it stdlib only, same command line (`python3 server.py [port]`, env `E05_PORT`, default 18501, bind 127.0.0.1). Write a short `SECURITY_FIX.md` (what was wrong, what you changed). No network beyond localhost.
Naive '..' string filters fail on double encoding, backslashes, absolute paths and especially symlinks pointing out of the web root; over-strict fixes break encoded names or in-root symlinks. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. blocks_traversal: none of 16 attack requests returns the random canary planted in config.env and in a file outside the workdir. 2. attack_status_4xx: every attack request gets a 4xx status (no 5xx, no dropped connection). 3. serves_index: / returns index.html as text/html. 4. serves_file_query: /docs/readme.txt?x=1 returns the file as text/plain. 5. serves_encoded_name: /docs/my%20file.txt works. 6. inside_symlink_ok: a symlink inside public/ pointing to a file inside public/ is served. 7. dir_404: a directory without index.html and a missing file return 404. 8. server_survived: the server process is still running after all requests. 9. notes_present: SECURITY_FIX.md exists and is not empty.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.01, 51s).
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Haiku 5.5default · thinking on | $0.01 | 51s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5default · thinking on | $0.20 | 47s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 22s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.