Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Classify 12 faulty sensors (frozen/°F/spike/gap/drift) with fault start hours

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.01
Wall time: 53s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Task: find faulty sensors `readings.csv` has hourly temperature readings (sensor_id, hour 0–167, temp_c) from 40 indoor sensors over one week. Normal sensors follow a daily cycle around 18–24 °C with small noise. Some sensors developed exactly one fault each, starting at some hour. Fault types: `frozen` (value stops changing), `unit_fahrenheit` (starts reporting °F), `spike` (a single absurd reading), `offline_gap` (a run of missing hours), `drift` (steady upward drift). Write `faults.json`: `{"S07": {"type": "drift", "start_hour": 55}, ...}` listing ONLY faulty sensors. No network.

Find the faulty sensors among 40 indoor temperature sensors over one week of hourly readings, and name each fault type and the hour it started. Faults include frozen values, a switch to Fahrenheit, single spikes, offline gaps and slow drift, and flagging a healthy but noisy sensor counts against the answer. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. Fault type correct for at least 11 of the 12 faulty sensors. 2. Zero false positives: no healthy sensor may appear in faults.json. 3. start_hour (integer) within ±3 hours of the true start for at least 9 sensors (±8 hours allowed for drift faults).

Setup notes: Use Case Scout batch 3 case c04-sensor-anomalies (offline fixture). Prepared with batch3/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR/faults.json KEYDIR/key.json`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. Claude Code's own reported cost for claude-haiku-5.5: $0.014459. Score: pass. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.01
Wall time: 53s
Human step-ins: 0
Turns: 16

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	285,185	10,056	$0.01

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. Fault type correct for at least 11 of the 12 faulty sensors. 2. Zero false positives: no healthy sensor may appear in faults.json. 3. start_hour (integer) within ±3 hours of the true start for at least 9 sensors (±8 hours allowed for drift faults).
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 51611,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.014458700000000001,
  "usage": {
   "input_tokens": 18,
   "cache_creation_input_tokens": 34617,
   "cache_read_input_tokens": 250550,
   "output_tokens": 10056,
   "output_tokens_details": {
    "thinking_tokens": 4186
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 34617,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 757,
     "cache_read_input_tokens": 45575,
     "cache_creation_input_tokens": 1581,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 1581
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 18,
    "outputTokens": 10056,
    "cacheReadInputTokens": 250550,
    "cacheCreationInputTokens": 34617,
    "webSearchRequests": 0,
    "costUSD": 0.014458700000000001,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 4186,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 16,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `faults.json` with 12 faulty sensors. All five fault types are represented, and the file parses as valid JSON.\n\n```json\n{\n  \"S01\": {\"type\": \"spike\", \"start_hour\": 115},\n  \"S04\": {\"type\": \"frozen\", \"start_hour\": 117},\n  \"S06\": {\"type\": \"frozen\", \"start_hour\": 67},\n  \"S07\": {\"type\": \"frozen\", \"start_hour\": 41},\n  \"S09\": {\"type\": \"spike\", \"start_hour\": 113},\n  \"S22\": {\"type\": \"unit_fahrenheit\", \"start_hour\": 47},\n  \"S26\": {\"type\": \"unit_fahrenheit\", \"start_hour\": 69},\n  \"S31\": {\"type\": \"drift\", \"start_hour\": 65},\n  \"S34\": {\"type\": \"offline_gap\", \"start_hour\": 36},\n  \"S38\": {\"type\": \"unit_fahrenheit\", \"start_hour\": 116},\n  \"S39\": {\"type\": \"offline_gap\", \"start_hour\": 66},\n  \"S40\": {\"type\": \"drift\", \"start_hour\": 59}\n}\n```\n\nHow each was identified:\n- **frozen:** the value repeats exactly for a long run (S07 from hour 41, S06 from 67, S04 from 117).\n- **unit_fahrenheit:** values jump to 65\u201380, which is the Celsius equivalent of about 18\u201327 \u00b0C. The first Fahrenheit hour is the first jump (S22 at 47, S26 at 69, S38 at 116).\n- **spike:** one reading far outside the normal range (S01 at hour 115 reads \u221219.51; S09 at 113 reads \u221222.99).\n- **offline_gap:** 10 hours missing (S34 hours 36\u201345, S39 hours 66\u201375). The file has 6,700 rows, which is 20 short of 40 \u00d7 168.\n- **drift:** residuals against the per-hour median of all sensors rise steadily. The start hours come from a piecewise-linear fit, so they are estimates. S31 \u2248 65 and S40 \u2248 59, with slopes near 0.08 \u00b0C/hour.\n\nThe fault-free sensors showed only small, stable offsets from the median, and none had a residual that looked like a fault. The analysis scripts are in `/tmp/analysis/`, not in the work directory.",
  "ttft_ms": 1698,
  "type": "result",
  "duration_ms": 52240,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1403,
  "time_to_request_ms": 219,
  "first_content_frame_ms": 1403,
  "queued_turn_count": 0,
  "result_index": 0
 }
]