Skip to content

Match a project's dependencies against a batch of security advisories and rank what to patch first

Exact prompt

# Match a project's dependencies against new security advisories and rank what to patch first `project/lockfile.json` lists the project's installed packages (`name`, `version`, `direct` true for a direct dependency, false for a transitive one) and its `ecosystem`. `advisories/` holds a batch of security advisories, one JSON file each: `id`, `ecosystem`, `package`, `affected` (a list of version ranges), `fixed_in` (the first fixed version, or null if there is no fix yet), `cvss`, `exploited` and `withdrawn`. All data is synthetic. ## Matching rules - An advisory applies only if its `ecosystem` equals the lockfile's and it is not withdrawn. - Package names match after normalising both: lower-case, and treat any run of `-`, `_` or `.` as a single `-`. - Versions are dot-separated whole numbers compared part by part as numbers (so 1.10.0 is newer than 1.9.0); a missing part counts as 0. - Each `affected` entry is a comma-separated list of conditions that must all hold (`>=`, `>`, `<=`, `<`, `==`); the installed version is affected if any entry holds. ## Ranking - A package is affected if at least one advisory applies to it and its installed version is affected. - Priority = the highest, over its matching advisories, of cvss plus 2.0 if that advisory is marked exploited. - upgrade_to = the highest `fixed_in` among its matching advisories, or null if any matching advisory has no fix. - Order: priority highest first, then direct dependencies before transitive ones, then name ascending. ## Output Write `out/patch_plan.json`: `{"affected_packages": [<sorted lockfile names>], "packages": {"<lockfile name>": {"current": "<version>", "advisories": [<sorted advisory ids>], "priority": <number>, "upgrade_to": "<version>"|null}}, "order": [<lockfile names in patch order>]}` Use each package's name exactly as written in the lockfile. ## Checks (partial credit = share passed) 1. `affected_set`: the affected packages are exactly those with an advisory that applies and an affected installed version. 2. `advisory_ids`: each affected package lists exactly the advisories that apply to its installed version. 3. `priorities`: each priority is the highest cvss plus 2.0 if that advisory is marked exploited. 4. `upgrade_targets`: each upgrade_to is the highest fixed_in among its matching advisories, or null if any has no fix. 5. `patch_order`: the order is priority highest first, then direct dependencies before transitive ones, then name ascending. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Check a project's lockfile against a batch of synthetic security advisories using version-range matching, then rank the affected packages by priority. Versions must be compared as numbers, upper bounds are exclusive, package names are normalised, and withdrawn or other-ecosystem advisories don't count. Each affected package gets its advisories, a priority with a bonus for known exploitation, and the version that fixes all of them. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: affected_set advisory_ids priorities upgrade_targets patch_order

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0056, 18s).

Run this on another model

All runs (3)

100% pass rate across 3 eligible runs: 3 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Opus 5.5default · thinking on$0.1624s0Fully autonomouspassTrust 80%2026-10-22
AClaude Haiku 5.5default · thinking on$0.005618s0Fully autonomouspassTrust 80%2026-10-22
AClaude Sonnet 5.5default · thinking off$0.059s0Fully autonomouspassTrust 80%2026-10-22

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Match a project's dependencies against a batch of security advisories and rank what to patch first

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.