Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Reverse engineer a security patch for an exploited CVE

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.0052
Wall time
25s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

patch.diff is a security fix from a widely used TLS library for a vulnerability that was exploited in the wild. Analyze only the diff (no web). Write ANALYSIS.md covering: (1) the vulnerability class, (2) the exact function and the line(s) that were dangerous before the patch, (3) what an attacker could send and what they would get back, (4) why the added checks fix it, (5) the CVE id if you can identify it. Do not write exploit code.

Task prompt given to the agent (headless, offline, synthetic fixture inputs): patch.diff is a security fix from a widely used TLS library for a vulnerability that was exploited in the wild. Analyze only the diff (no web). Write ANALYSIS.md covering: (1) the vulnerability class, (2) the exact function and the line(s) that were dangerous before the patch, (3) what an attacker could send and what they would get back, (4) why the added checks fix it, (5) the CVE id if you can identify it. Do not write exploit code. Inspired by https://x.com/KevTheHermit/status/2094160406935638494 (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost shown is a list-price API-equivalent estimate (run on a Claude Pro subscription, not billed).

Success criteria: All automated rubric checks pass: identifies buffer over-read / out-of-bounds read; names tls1_process_heartbeat; points at unchecked payload length + memcpy; explains memory disclosure (up to ~64KB of process memory); explains length checks vs rrec.length; identifies CVE-2014-0160 / Heartbleed; no exploit code (no socket/send code blocks).

Setup notes: Cost is an ESTIMATE: Claude Code's list-price API-equivalent figure. The run used a Claude Pro subscription, so nothing was billed. input_tokens = all input incl. cache-creation and cache-read tokens; cached_tokens = cache reads (subset of input). Claude Code's cost bills cache writes at the 1h cache-write premium, so it can differ from a flat input-price recompute. Web disabled; tools Read/Write/Edit/Glob/Grep/Bash; single run; scored automatically (7/7 checks).

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.0052
Wall time
25s
Human step-ins
0
Turns
4
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on85,2164,589$0.0052

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: All automated rubric checks pass: identifies buffer over-read / out-of-bounds read; names tls1_process_heartbeat; points at unchecked payload length + memcpy; explains memory disclosure (up to ~64KB of process memory); explains length checks vs rrec.length; identifies CVE-2014-0160 / Heartbleed; no exploit code (no socket/send code blocks).
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
{
 "duration_api_ms": 24083,
 "stop_reason": "end_turn",
 "session_id": "[REDACTED]",
 "total_cost_usd": 0.00524232,
 "usage": {
  "input_tokens": 8,
  "cache_creation_input_tokens": 11026,
  "cache_read_input_tokens": 74182,
  "output_tokens": 4589,
  "output_tokens_details": {
   "thinking_tokens": 1620
  },
  "server_tool_use": {
   "web_search_requests": 0,
   "web_fetch_requests": 0
  },
  "service_tier": "standard",
  "cache_creation": {
   "ephemeral_1h_input_tokens": 11026,
   "ephemeral_5m_input_tokens": 0
  },
  "inference_geo": "global",
  "iterations": [
   {
    "input_tokens": 2,
    "output_tokens": 653,
    "cache_read_input_tokens": 23289,
    "cache_creation_input_tokens": 276,
    "cache_creation": {
     "ephemeral_5m_input_tokens": 0,
     "ephemeral_1h_input_tokens": 276
    },
    "type": "message"
   }
  ],
  "speed": "standard",
  "fallback_credit": null
 },
 "modelUsage": {
  "claude-haiku-5-5": {
   "inputTokens": 8,
   "outputTokens": 4589,
   "cacheReadInputTokens": 74182,
   "cacheCreationInputTokens": 11026,
   "webSearchRequests": 0,
   "costUSD": 0.00524232,
   "contextWindow": 1000000,
   "maxOutputTokens": 128000,
   "thinkingTokens": 1620,
   "canonicalModel": "claude-haiku-5-5",
   "provider": "firstParty",
   "costBasis": "list"
  }
 },
 "permission_denials": [],
 "terminal_reason": "completed",
 "fast_mode_state": "off",
 "fast_mode_disabled_reason": "sdk_opt_in_required",
 "subagent_stats": {
  "spawned": 0,
  "requested": {
   "background": 0,
   "foreground": 0,
   "unset": 0
  },
  "started_in_background": 0,
  "max_depth": 0,
  "spawned_by_subagents": 0,
  "completed": 0,
  "failed": 0,
  "killed": {
   "parent": 0,
   "user": 0,
   "system": 0
  },
  "refused": {
   "depth_limit": 0,
   "concurrency_limit": 0,
   "budget": 0
  },
  "by_type": {}
 },
 "safety_stops": 0,
 "is_error": false,
 "num_turns": 4,
 "subtype": "success",
 "api_error_status": null,
 "result": "I wrote `ANALYSIS.md` in the work directory. The patch is the Heartbleed fix in OpenSSL's heartbeat handling (`ssl/t1_lib.c`, `tls1_process_heartbeat`), which is CVE-2014-0160. I did not use the web, so the CVE ID comes from my knowledge of this public fix and is not checked against a reference. No exploit code is included.\n\nSummary of the analysis:\n\n1. **Vulnerability class:** out-of-bounds read from an attacker-controlled length field, causing information disclosure. The receiver trusts the 16-bit payload length without checking it against the bytes actually received.\n2. **Dangerous code before the patch:** `hbtype = *p++` and `n2s(p, payload)` ran before any length check. The response path then did `memcpy(bp, pl, payload)`, where `payload` is attacker-controlled and `pl` points into the received record.\n3. **Attacker input and output:** a heartbeat record that claims a larger payload than it contains. The server echoes back the claimed number of bytes, so the excess comes from adjacent process memory. Each request leaks up to about 64 KB, and repeated requests leak more. This is a read, and the diff shows no code-execution path.\n4. **Why the fix works:** two checks, `1 + 2 + 16 > rrec.length` before reading the header, and `1 + 2 + payload + 16 > rrec.length` after reading the length. Together they ensure the copy stays inside the received record. Malformed messages are silently discarded, as RFC 6520 specifies.\n5. **CVE:** CVE-2014-0160 (Heartbleed), affecting OpenSSL 1.0.1 through 1.0.1f and fixed in 1.0.1g. The version range is from memory and was not verified.\n\nThe diff is partial: its hunk headers have no line numbers, and the record framing and response allocation are outside the shown lines. The document flags those points as inferred.",
 "ttft_ms": 1414,
 "type": "result",
 "duration_ms": 24421,
 "uuid": "[REDACTED]",
 "ttft_stream_ms": 1069,
 "time_to_request_ms": 225,
 "first_content_frame_ms": 1069,
 "queued_turn_count": 0,
 "result_index": 0
}

Get new runs and ranking changes weekly