Skip to content

Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Find the vulnerability fixed between two releases and write a regression test

AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.32
Wall time
59s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

This folder contains two published npm releases of a JavaScript JSON Web Token library, unpacked as fast-jwt-6.3.0/ and fast-jwt-6.3.1/ (package contents only, no tests or changelog). Their runtime dependencies are installed in ./node_modules. The newer release quietly fixes a security vulnerability. You have no internet access; work only from the code. 1. Find the vulnerable code: name the file and function that were changed to fix it. 2. Write ANALYSIS.md explaining the root cause, which caller configurations are affected, and the impact on an application that uses the library. 3. Write a minimal, non-weaponized regression test, regression.test.js, using node:test and node:assert. It must load the library with require(process.env.LIB_DIR). It will be run as `LIB_DIR=<absolute path to a release folder> node --test regression.test.js` and must FAIL against 6.3.0 and PASS against 6.3.1. Do not write exploit tooling, network code, or anything that targets real systems. Do not modify the two release folders.

Reverse engineering a real security fix: the agent gets two npm releases of the fast-jwt library (6.3.0 and 6.3.1) with no advisory or changelog. It must find the patched function, explain the vulnerability (forged tokens accepted when the verifier key is empty), and write a harmless regression test that fails on the old release and passes on the new one. The advisory was published after the models' training cutoff. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Upstream fix being reproduced: https://github.com/nearform/fast-jwt/pull/649, https://github.com/nearform/fast-jwt/commit/e22a151e83bf6d5e54e281216982d204d5665534.

Success criteria: Correct file/function identified; root cause matches advisory; regression test fails on old tag and passes on new.

Setup notes: Real CVE: nearform/fast-jwt GHSA-8wpc-h4q6-8fxv (CVE-2026-107720), fixed in 6.3.1 (merged 2026-07-30, after the models' Jun 2026 cutoff). Agent got the npm package contents (package.json, README, LICENSE, src/) of 6.3.0 and 6.3.1 via git archive, with deps in node_modules; no tests, changelog or advisory. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Regression test run by the harness on pristine copies of each release. Rubric is keyword-based for the analysis plus executed test results (batch2/p08/run_p08.py). Claude Code's own reported cost for claude-opus-5.5: $0.316906. Score: 8/8. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.32
Wall time
59s
Human step-ins
0
Turns
8
Model usage
ModelInput tokOutput tokCost
AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on210,9895,643$0.32

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: Correct file/function identified; root cause matches advisory; regression test fails on old tag and passes on new.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 57462,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.31690619999999997,
  "usage": {
   "input_tokens": 16,
   "cache_creation_input_tokens": 20742,
   "cache_read_input_tokens": 190231,
   "output_tokens": 5643,
   "output_tokens_details": {
    "thinking_tokens": 599
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 20742,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 807,
     "cache_read_input_tokens": 30310,
     "cache_creation_input_tokens": 2578,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 2578
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 16,
    "outputTokens": 5643,
    "cacheReadInputTokens": 190231,
    "cacheCreationInputTokens": 20742,
    "webSearchRequests": 0,
    "costUSD": 0.31690619999999997,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 599,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 8,
  "subtype": "success",
  "api_error_status": null,
  "result": "I've written `ANALYSIS.md` and `regression.test.js`. The test fails against 6.3.0 (2 of 3 tests fail) and passes against 6.3.1 (3 of 3). I didn't touch the two release folders.\n\n**1. The vulnerable code:** the key-validation block in `createVerifier` in `src/verifier.js`.\n\n**2. Root cause:** in 6.3.0, `createVerifier` only checks the key's type. An empty string `''` passes as a string and `null` passes as an object. Because both are falsy, the verifier skips the code that matches the key to the allowed algorithms and rejects empty secrets. The caller's `algorithms` list (e.g. `['HS256']`) stays in force anyway.\n\nWhen a token is checked, `verifyToken` sees no key and no signature, so it doesn't complain. `validateAlgorithmAndSignature` only checks a signature if one is present. The result: a token with `alg: HS256` in its header and an empty signature (`header.payload.`) is accepted as genuine.\n\n**Affected configurations:** a direct (non-function) key that is `''` or `null`, combined with an explicit `algorithms` list containing anything other than `none`. Mixed lists like `['none', 'HS256']` are affected too. This typically happens through a mistake like `key: process.env.JWT_SECRET || ''`. These are not affected:\n- callers that pass a function as the key\n- callers that leave `algorithms` unset\n- callers whose key is `undefined`\n\n**Impact:** a full authentication bypass. Anyone can create a token with any claims (user ID, roles, audience, expiry) and the library returns it as verified. The 6.3.1 fix refuses an empty or `null` key at creation time unless `none` is the only allowed algorithm. It also refuses a real key when `none` is the only allowed algorithm.\n\n**3. Regression test:** it builds a verifier with `key: ''` and another with `key: null`, both with `algorithms: ['HS256']`. It then checks that an unsigned `HS256` token is rejected, either when the verifier is created or when the token is checked. A third test confirms the intended setup (only `none` allowed, no key) still works. It contains no network code and doesn't target any real system.\n\nOne more change in 6.3.1: in 6.3.0 the error for a key of the wrong type had `code: undefined` because it used a misspelled error-code name. 6.3.1 corrects it.",
  "ttft_ms": 2334,
  "type": "result",
  "duration_ms": 58686,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1541,
  "time_to_request_ms": 217,
  "first_content_frame_ms": 1541,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly