Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Fix five behavioural bugs in an LRU cache with per-entry TTL

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-17 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.05
Wall time: 10s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: fix the TTL cache `ttlcache.py` is an LRU cache with per-entry TTL. Users report: recently-read keys get evicted, expired keys are still counted in `len()`/`in`, entries that expire exactly at their deadline are still returned, overwriting a key does not refresh its LRU position, and `purge()` crashes. Intended behaviour: - `get` on a live key marks it most-recently-used; when the cache exceeds `maxsize` the **least**-recently-used entry is evicted. - An entry is expired when `clock() >= expires_at` (an entry with ttl=10 set at t=0 is gone at t=10). Expired entries are removed when touched by `get`, and are never visible via `len()` or `in`. - `set` on an existing key replaces the value, resets its TTL and makes it most-recently-used. - `ttl` of 0 or negative means "do not store" (and removes any existing entry for that key). - `purge()` removes all expired entries and returns how many were removed. - `hits`/`misses` count `get` outcomes. Keep the public API (constructor signature, method names). Add your own tests if you like. No network.

Bugs are spread over eviction order, LRU refresh on get/overwrite, an exact expiry boundary, expired-entry visibility, zero TTL and a dict-mutation crash; hidden tests use an injected clock. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. All 11 hidden unittest cases pass against the agent's ttlcache.py (60 s timeout): LRU eviction of least-recently-used, get/overwrite refresh recency, expiry at clock() >= expires_at, expired entries invisible to len/in, overwrite resets TTL, ttl<=0 not stored, purge() returns count without crashing, hits/misses counted, constructor API kept.

Setup notes: Use Case Scout batch 5 case e01-ttl-cache-bugs (offline fixture). Prepared with batch5/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-sonnet-5.5: $0.046728. Score: 11/11. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.05
Wall time: 10s
Human step-ins: 0
Turns: 3

Model usage
Model	Input tok	Output tok	Cost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on	56,838	1,307	$0.05

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. All 11 hidden unittest cases pass against the agent's ttlcache.py (60 s timeout): LRU eviction of least-recently-used, get/overwrite refresh recency, expiry at clock() >= expires_at, expired entries invisible to len/in, overwrite resets TTL, ttl<=0 not stored, purge() returns count without crashing, hits/misses counted, constructor API kept.
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 9332,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.046728200000000004,
  "usage": {
   "input_tokens": 6,
   "cache_creation_input_tokens": 7170,
   "cache_read_input_tokens": 49662,
   "output_tokens": 1307,
   "output_tokens_details": {
    "thinking_tokens": 23
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 7170,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 438,
     "cache_read_input_tokens": 18891,
     "cache_creation_input_tokens": 818,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 818
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 6,
    "outputTokens": 1307,
    "cacheReadInputTokens": 49662,
    "cacheCreationInputTokens": 7170,
    "webSearchRequests": 0,
    "costUSD": 0.046728200000000004,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 23,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 3,
  "subtype": "success",
  "api_error_status": null,
  "result": "I fixed all five reported problems in `ttlcache.py`, and a quick script I ran against the new code passed. I didn't add a test file.\n\n- **Eviction:** the cache now drops the least-recently-used entry. Before, it dropped the newest one. When it goes over `maxsize`, it clears expired entries first, so a live entry isn't evicted while a dead one is still stored.\n- **Reads:** `get` on a live key now makes it most-recently-used. An entry expires when `clock() >= expires_at`, and `get` deletes it when it finds it expired.\n- **`len()` and `in`:** both now ignore expired entries. `len()` purges before counting, and `in` removes an expired key it finds.\n- **`set`:**\n  - Overwriting a key replaces the value, resets its TTL and moves it to the most-recent position.\n  - A `ttl` of 0 or less stores nothing and removes any existing entry for that key.\n- **`purge()`:** it no longer crashes. The old version deleted from the dict while looping over it. It now collects the expired keys first and returns how many it removed.\n\nThe check script covered LRU order after a `get`, expiry exactly at the deadline, a TTL reset on overwrite, the `purge()` count and `ttl=0`. It didn't check `hits`/`misses`, which I left as they were.",
  "ttft_ms": 1795,
  "type": "result",
  "duration_ms": 9693,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1166,
  "time_to_request_ms": 265,
  "first_content_frame_ms": 1167,
  "queued_turn_count": 0,
  "result_index": 0
 }
]