Match a project's dependencies against a batch of security advisories and rank what to patch first
Exact prompt
# Match a project's dependencies against new security advisories and rank what to patch first `project/lockfile.json` lists the project's installed packages (`name`, `version`, `direct` true for a direct dependency, false for a transitive one) and its `ecosystem`. `advisories/` holds a batch of security advisories, one JSON file each: `id`, `ecosystem`, `package`, `affected` (a list of version ranges), `fixed_in` (the first fixed version, or null if there is no fix yet), `cvss`, `exploited` and `withdrawn`. All data is synthetic. ## Matching rules - An advisory applies only if its `ecosystem` equals the lockfile's and it is not withdrawn. - Package names match after normalising both: lower-case, and treat any run of `-`, `_` or `.` as a single `-`. - Versions are dot-separated whole numbers compared part by part as numbers (so 1.10.0 is newer than 1.9.0); a missing part counts as 0. - Each `affected` entry is a comma-separated list of conditions that must all hold (`>=`, `>`, `<=`, `<`, `==`); the installed version is affected if any entry holds. ## Ranking - A package is affected if at least one advisory applies to it and its installed version is affected. - Priority = the highest, over its matching advisories, of cvss plus 2.0 if that advisory is marked exploited. - upgrade_to = the highest `fixed_in` among its matching advisories, or null if any matching advisory has no fix. - Order: priority highest first, then direct dependencies before transitive ones, then name ascending. ## Output Write `out/patch_plan.json`: `{"affected_packages": [<sorted lockfile names>], "packages": {"<lockfile name>": {"current": "<version>", "advisories": [<sorted advisory ids>], "priority": <number>, "upgrade_to": "<version>"|null}}, "order": [<lockfile names in patch order>]}` Use each package's name exactly as written in the lockfile. ## Checks (partial credit = share passed) 1. `affected_set`: the affected packages are exactly those with an advisory that applies and an affected installed version. 2. `advisory_ids`: each affected package lists exactly the advisories that apply to its installed version. 3. `priorities`: each priority is the highest cvss plus 2.0 if that advisory is marked exploited. 4. `upgrade_targets`: each upgrade_to is the highest fixed_in among its matching advisories, or null if any has no fix. 5. `patch_order`: the order is priority highest first, then direct dependencies before transitive ones, then name ascending. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Check a project's lockfile against a batch of synthetic security advisories using version-range matching, then rank the affected packages by priority. Versions must be compared as numbers, upper bounds are exclusive, package names are normalised, and withdrawn or other-ecosystem advisories don't count. Each affected package gets its advisories, a priority with a bonus for known exploitation, and the version that fixes all of them. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: affected_set advisory_ids priorities upgrade_targets patch_order
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0056, 18s).
All runs (3)
100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Opus 5.5default · thinking on | $0.16 | 24s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Haiku 5.5default · thinking on | $0.0056 | 18s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 | |
| AClaude Sonnet 5.5default · thinking off | $0.05 | 9s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-22 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.