Skip to content

Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Classify 12 faulty sensors (frozen/°F/spike/gap/drift) with fault start hours

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.08
Wall time
23s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Task: find faulty sensors `readings.csv` has hourly temperature readings (sensor_id, hour 0–167, temp_c) from 40 indoor sensors over one week. Normal sensors follow a daily cycle around 18–24 °C with small noise. Some sensors developed exactly one fault each, starting at some hour. Fault types: `frozen` (value stops changing), `unit_fahrenheit` (starts reporting °F), `spike` (a single absurd reading), `offline_gap` (a run of missing hours), `drift` (steady upward drift). Write `faults.json`: `{"S07": {"type": "drift", "start_hour": 55}, ...}` listing ONLY faulty sensors. No network.

Find the faulty sensors among 40 indoor temperature sensors over one week of hourly readings, and name each fault type and the hour it started. Faults include frozen values, a switch to Fahrenheit, single spikes, offline gaps and slow drift, and flagging a healthy but noisy sensor counts against the answer. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. Fault type correct for at least 11 of the 12 faulty sensors. 2. Zero false positives: no healthy sensor may appear in faults.json. 3. start_hour (integer) within ±3 hours of the true start for at least 9 sensors (±8 hours allowed for drift faults).

Setup notes: Use Case Scout batch 3 case c04-sensor-anomalies (offline fixture). Prepared with batch3/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR/faults.json KEYDIR/key.json`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-sonnet-5.5: $0.075966. Score: pass. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.08
Wall time
23s
Human step-ins
0
Turns
5
Model usage
ModelInput tokOutput tokCost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on102,0172,570$0.08

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. Fault type correct for at least 11 of the 12 faulty sensors. 2. Zero false positives: no healthy sensor may appear in faults.json. 3. start_hour (integer) within ±3 hours of the true start for at least 9 sensors (±8 hours allowed for drift faults).
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 20708,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.07596589999999999,
  "usage": {
   "input_tokens": 10,
   "cache_creation_input_tokens": 10268,
   "cache_read_input_tokens": 91739,
   "output_tokens": 2570,
   "output_tokens_details": {
    "thinking_tokens": 452
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 10268,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 608,
     "cache_read_input_tokens": 22085,
     "cache_creation_input_tokens": 722,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 722
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 10,
    "outputTokens": 2570,
    "cacheReadInputTokens": 91739,
    "cacheCreationInputTokens": 10268,
    "webSearchRequests": 0,
    "costUSD": 0.07596589999999999,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 452,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 5,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `faults.json` with 12 faulty sensors out of the 40. I haven't checked it against any ground truth. The start hours for the drift sensors are estimates.\n\n| Sensor | Fault | Start hour | How I spotted it |\n|---|---|---|---|\n| S04 | frozen | 117 | Same value repeated from here to the end of the week |\n| S06 | frozen | 67 | Same value repeated from here to the end of the week |\n| S07 | frozen | 29 | Same value repeated from here to the end of the week |\n| S22 | unit_fahrenheit | 47 | Readings jump to Fahrenheit-range values (max 73.9) |\n| S26 | unit_fahrenheit | 69 | Readings jump to Fahrenheit-range values (max 81.1) |\n| S38 | unit_fahrenheit | 116 | Readings jump to Fahrenheit-range values (max 80.3) |\n| S01 | spike | 115 | A single reading of \u221219.5 |\n| S09 | spike | 113 | A single reading of \u221223.0 |\n| S34 | offline_gap | 36 | 10 missing hours, starting here |\n| S39 | offline_gap | 66 | 10 missing hours, starting here |\n| S31 | drift | 62 | Steady rise of roughly 0.08 \u00b0C per hour, reaching about +8 \u00b0C by the end |\n| S40 | drift | 57 | Steady rise of roughly 0.1 \u00b0C per hour, reaching about +8 \u00b0C by the end |\n\n- **Drift start hours:** I got these by fitting a straight ramp to each sensor's difference from its own first-day pattern, so they could be off by several hours.\n- **Frozen start hours:** I used the last hour before the value first repeats. If the intended start is the first repeated hour, each frozen start is one higher.",
  "ttft_ms": 1798,
  "type": "result",
  "duration_ms": 22049,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1081,
  "time_to_request_ms": 214,
  "first_content_frame_ms": 1082,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly