Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Which tunnel peer gets this packet? Parse a VPN peer config, longest-prefix routing, ownership moves and findings

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.07
Wall time: 22s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Which tunnel peer gets this packet? Our home-lab VPN box uses WireGuard-style configs (sample: `home.conf`). `wgroute.py` is meant to answer "which peer would a packet to IP X be sent to?" and to list config problems, but the draft is wrong in many ways. Rewrite `wgroute.py` (Python 3 stdlib only; `ipaddress` is fine) so that `WGConfig(text)` exposes `.peers()`, `.route(ip)` and `.findings` exactly as below. ## Parsing 1. Lines are numbered from 1. `#` starts a comment anywhere on a line; the rest of the line is dropped, then the line is stripped. Empty lines are skipped. 2. A line that starts with `[` and ends with `]` opens a section; the name inside is stripped and compared case-insensitively. Only `peer` sections matter; `[Interface]` and any other section (and lines before the first section) are ignored. 3. Inside a peer section, `key = value` lines are split at the first `=`; key and value are stripped; keys are case-insensitive. Lines without `=` and unknown keys are ignored. `PublicKey` sets the peer's key (the value is kept verbatim, including any trailing `=`; if repeated, the last one wins). `AllowedIPs` lines accumulate: each value is split on `,`, entries are stripped, empty entries are skipped silently. 4. A peer section is processed when the next section header or the end of the text is reached, in file order: - no PublicKey -> finding `("missing_key", header_line)`, the section is skipped; - a PublicKey already used by an earlier processed peer -> finding `("duplicate_peer", key, header_line)`, the whole section is skipped; - otherwise each AllowedIPs entry in order: it is an IPv4/IPv6 address or CIDR; a bare address means /32 or /128; host bits are cleared (`10.0.0.9/24` -> `10.0.0.0/24`). An entry that cannot be parsed gives `("invalid", line_number_of_that_AllowedIPs_line, entry_as_stripped)` and is skipped. 5. **Ownership.** Each normalised prefix belongs to at most one peer. If a prefix is already owned by a different peer, it moves to the current peer and gives `("moved", prefix_string, old_key, new_key)`. Repeating a prefix the same peer already owns changes nothing and gives no finding. 6. `.findings` is the list of these tuples in the order they were produced. Prefix strings use Python's `str(ipaddress.ip_network(...))` form. ## Queries 7. `.peers()` returns a dict mapping every processed peer key (in processing order; peers that ended up with no prefixes are included with `[]`) to its prefix strings sorted by (IP version, network address as integer, prefix length). 8. `.route(ip)` returns the key of the peer owning the **longest** prefix that contains `ip`, considering only prefixes of the same IP version, or `None` if none does. Surrounding whitespace in `ip` is ignored. An IPv4-mapped IPv6 address (`::ffff:a.b.c.d`) is looked up as the IPv4 address `a.b.c.d`. An invalid address raises `ValueError`. 9. **Speed.** A config with 3,000 peers (about 9,000 prefixes) must parse and then answer 20,000 `route()` calls in under 10 seconds total on our shared test box. ## Scoring Hidden check groups (partial credit = fraction passed; all must pass): `parse_sections_comments` (1-3), `skip_rules_findings` (4, 6), `normalise_and_invalid` (4), `ownership_moves` (5, 6), `peers_listing` (7), `longest_prefix` (8), `mapped_and_errors` (8), `randomized_agreement` (300 random configs of small random prefixes compared on peers(), findings and 50 routes each), `speed` (9). On `home.conf` the correct `.route("10.8.0.2")` is `"PHONEkey="`.

Exact findings order (missing key, duplicate peer, invalid entry with line number, prefix moved between peers), host-bit normalisation, IPv4-mapped IPv6 lookups and a 3,000-peer speed check that rules out linear scans. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. parse_sections_comments 2. skip_rules_findings 3. normalise_and_invalid 4. ownership_moves 5. peers_listing 6. longest_prefix 7. mapped_and_errors 8. randomized_agreement (300 configs) 9. speed (3,000 peers, 20,000 lookups < 10 s)

Setup notes: Use Case Scout batch 8 case i02-vpn-peer-routing (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring: 2.10. Claude Code's own reported cost for claude-sonnet-5.5: $0.072275. Score: 9/9. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.07
Wall time: 22s
Human step-ins: 0
Turns: 3

Model usage
Model	Input tok	Output tok	Cost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on	61,427	2,819	$0.07

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. parse_sections_comments 2. skip_rules_findings 3. normalise_and_invalid 4. ownership_moves 5. peers_listing 6. longest_prefix 7. mapped_and_errors 8. randomized_agreement (300 configs) 9. speed (3,000 peers, 20,000 lookups < 10 s)
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 20715,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.07227549999999999,
  "usage": {
   "input_tokens": 6,
   "cache_creation_input_tokens": 9726,
   "cache_read_input_tokens": 51695,
   "output_tokens": 2819,
   "output_tokens_details": {
    "thinking_tokens": 785
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 9726,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 604,
     "cache_read_input_tokens": 19939,
     "cache_creation_input_tokens": 2326,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 2326
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 6,
    "outputTokens": 2819,
    "cacheReadInputTokens": 51695,
    "cacheCreationInputTokens": 9726,
    "webSearchRequests": 0,
    "costUSD": 0.07227549999999999,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 785,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 3,
  "subtype": "success",
  "api_error_status": null,
  "result": "I rewrote `wgroute.py` to follow the spec. I only ran it on `home.conf`; I didn't test it against anything else, and I didn't time the 3,000-peer case.\n\nOn `home.conf`:\n- **Routing:** `.route(\"[REDACTED:ipv4]\")` returns `\"PHONEkey=\"`, as expected.\n- **Findings:** it reports `('moved', '[REDACTED:ipv4]/32', 'LAPTOPkey=', 'PHONEkey=')` and `('invalid', 18, 'bogus/99')`.\n- **Peers:** `LAPTOPkey=` is left with only `fd42::2/128`, and `NASkey=` gets `[REDACTED:ipv4]/0` and `[REDACTED:ipv4]/24`.\n- **Mapped addresses:** `::ffff:[REDACTED:ipv4]` (with surrounding spaces) routes as IPv4 `[REDACTED:ipv4]`.\n\nFor speed, `route()` does one dictionary lookup per prefix length in use (at most 33 for IPv4, 129 for IPv6). It doesn't scan every prefix, so 20,000 calls on 9,000 prefixes should be quick.\n\nThe spec left a few cases open, so I made these calls:\n- **Empty `PublicKey`:** `PublicKey =` with nothing after it counts as a missing key.\n- **Netmask form:** an entry like `[REDACTED:ipv4]/[REDACTED:ipv4]` is reported as invalid. Python's `ip_network` would accept it, but the spec says \"address or CIDR\".\n- **Moved prefixes:** a prefix that moves to another peer is removed from the old peer's list, but the old peer stays in `.peers()` even if it ends up empty.",
  "ttft_ms": 1834,
  "type": "result",
  "duration_ms": 21068,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1401,
  "time_to_request_ms": 257,
  "first_content_frame_ms": 1401,
  "queued_turn_count": 0,
  "result_index": 0
 }
]