Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Make a TCP key-value server follow its wire protocol (pipelining, split reads, binary payloads, expiry, close rules)

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-17 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.03
Wall time
2m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Task: make the MKV server follow its protocol `server.py` implements the key-value protocol in `PROTOCOL.md`, but it only works when every command arrives in its own TCP packet and only one client connects at a time. Rewrite/fix `server.py` so it implements PROTOCOL.md exactly: pipelining, arbitrarily split/merged reads, binary payloads (including `\r\n` inside), concurrent clients sharing the store, expiry, all error replies and the connection-closing rules. A misbehaving client must never crash the server or affect other clients. Python standard library only (threads, selectors or asyncio are all fine). Keep the command line (`python3 server.py [port]`, env `F03_PORT`, default 18601, bind 127.0.0.1). No network beyond localhost.

The starting server assumes one command per recv and one client at a time; a correct fix needs a per-connection buffer, length-prefixed payload framing that tolerates CRLF inside payloads, per-client threads with a shared locked store, lazy expiry and exact error/close semantics. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. basic_commands: PING/SET/GET/DEL/INCR exact replies incl. empty payload, case-insensitive commands, case-sensitive keys, negative INCR. 2. error_replies: unknown command, wrong args, bad length and non-integer INCR give the exact errors and keep the connection open. 3. pipelining: 601 pipelined commands in one write get 601 in-order replies. 4. split_reads_binary: a SET with a payload containing CRLF/NUL/0xFF sent one byte at a time round-trips exactly. 5. concurrent_clients: a client stalled mid-payload does not block others; 20 clients x 50 INCR on one key gives 1000. 6. expiry: EXPIRE semantics incl. fractional seconds, :0 for missing keys, SET clearing expiry. 7. connection_closing_rules: line too long, bad payload terminator, payload > 1 MiB and QUIT each send the right reply and close the connection. 8. survives_bad_clients: after junk bytes and clients disconnecting mid-command, the server still answers PING.

Setup notes: Use Case Scout batch 6 case f03-kv-protocol-server (offline fixture). Prepared with batch6/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring: 1.93. Claude Code's own reported cost for claude-haiku-5.5: $0.027244. Score: 8/8 checks. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.03
Wall time
2m
Human step-ins
0
Turns
14
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on569,10028,019$0.03

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. basic_commands: PING/SET/GET/DEL/INCR exact replies incl. empty payload, case-insensitive commands, case-sensitive keys, negative INCR. 2. error_replies: unknown command, wrong args, bad length and non-integer INCR give the exact errors and keep the connection open. 3. pipelining: 601 pipelined commands in one write get 601 in-order replies. 4. split_reads_binary: a SET with a payload containing CRLF/NUL/0xFF sent one byte at a time round-trips exactly. 5. concurrent_clients: a client stalled mid-payload does not block others; 20 clients x 50 INCR on one key gives 1000. 6. expiry: EXPIRE semantics incl. fractional seconds, :0 for missing keys, SET clearing expiry. 7. connection_closing_rules: line too long, bad payload terminator, payload > 1 MiB and QUIT each send the right reply and close the connection. 8. survives_bad_clients: after junk bytes and clients disconnecting mid-command, the server still answers PING.
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 132722,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.027243550000000002,
  "usage": {
   "input_tokens": 28,
   "cache_creation_input_tokens": 39687,
   "cache_read_input_tokens": 529385,
   "output_tokens": 28019,
   "output_tokens_details": {
    "thinking_tokens": 18171
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 39687,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 980,
     "cache_read_input_tokens": 51595,
     "cache_creation_input_tokens": 631,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 631
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 28,
    "outputTokens": 28019,
    "cacheReadInputTokens": 529385,
    "cacheCreationInputTokens": 39687,
    "webSearchRequests": 0,
    "costUSD": 0.027243550000000002,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 18171,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 14,
  "subtype": "success",
  "api_error_status": null,
  "result": "I rewrote `server.py` on asyncio so it follows PROTOCOL.md. A local test script that I kept outside the work directory passes 23 of 23 checks, covering the points you listed.\n\n**What changed**\n- **Concurrency:** one event loop serves all clients, and each command runs without awaiting, so the shared store needs no locks. A stalled or garbage client only affects its own connection.\n- **Parsing:** input is read as a stream, so commands can be split or merged arbitrarily and pipelined. Commands are case-insensitive. Payloads are read as exactly `<len>` bytes plus a CRLF check, so `\\r\\n` inside a payload is fine.\n- **Errors and closing:** wrong args, unknown commands, too-large lengths, overlong lines and bad payloads all return the exact replies in the spec. Lines over 1024 bytes are rejected as soon as they pass that length, even before their CRLF arrives. Before closing, the server flushes the final reply and absorbs pending input for up to 1 second, so the client doesn't see a TCP reset that drops the reply.\n- **Expiry:** deadlines use a monotonic clock. Expired keys are dropped on access. `SET` clears expiry; `INCR` keeps it.\n- **Command line:** unchanged, `python3 server.py [port]`, then `F03_PORT`, then 18601, bound to 127.0.0.1. I checked the env var and the default port.\n\n**Judgment calls where the spec is silent**\n- `EXPIRE` with a non-numeric time returns `-ERR wrong args`. Negative or zero times are accepted; the key then expires immediately and the reply is `:1`.\n- An empty line gets `-ERR unknown command`, because each command must get one reply.\n- Expired keys are only removed when touched. There is no background sweep, so untouched expired keys stay in memory.\n\n**Known limitation**\n- `INCR` on a value near the 1 MB limit parses a huge integer on the event loop. That can stall every client for some time. I didn't add a guard, since any cap would change the \"base-10 integer\" behaviour the spec requires.\n\nThe work directory is not a git repo, so I did not commit anything. The test script is at `/tmp/mkvtest/test_mkv.py` if you want to rerun it.",
  "ttft_ms": 1711,
  "type": "result",
  "duration_ms": 144035,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1661,
  "time_to_request_ms": 265,
  "first_content_frame_ms": 1661,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly