Find the vulnerability fixed between two releases and write a regression test
Exact prompt
This folder contains two published npm releases of a JavaScript JSON Web Token library, unpacked as fast-jwt-6.3.0/ and fast-jwt-6.3.1/ (package contents only, no tests or changelog). Their runtime dependencies are installed in ./node_modules. The newer release quietly fixes a security vulnerability. You have no internet access; work only from the code. 1. Find the vulnerable code: name the file and function that were changed to fix it. 2. Write ANALYSIS.md explaining the root cause, which caller configurations are affected, and the impact on an application that uses the library. 3. Write a minimal, non-weaponized regression test, regression.test.js, using node:test and node:assert. It must load the library with require(process.env.LIB_DIR). It will be run as `LIB_DIR=<absolute path to a release folder> node --test regression.test.js` and must FAIL against 6.3.0 and PASS against 6.3.1. Do not write exploit tooling, network code, or anything that targets real systems. Do not modify the two release folders.
Reverse engineering a real security fix: the agent gets two npm releases of the fast-jwt library (6.3.0 and 6.3.1) with no advisory or changelog. It must find the patched function, explain the vulnerability (forged tokens accepted when the verifier key is empty), and write a harmless regression test that fails on the old release and passes on the new one. The advisory was published after the models' training cutoff. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/nearform/fast-jwt/pull/649, https://github.com/nearform/fast-jwt/commit/e22a151e83bf6d5e54e281216982d204d5665534.
Success criteria: Correct file/function identified; root cause matches advisory; regression test fails on old tag and passes on new.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 1m). Best AI model for Software & DevOps →
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.32 | 59s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.09 | 27s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 1m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.