Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Test AI browser agents on real workday tasks
AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener
- Estimated cost
- $0.0040
- Wall time
- 17s
- Human step-ins
- 0
- Outcome
- Pass
Compare with other models on this task →
Summary
Exact prompt
You are a workday assistant on a mock calendar (no real accounts). Complete requests.md by editing calendar.json.
Task prompt given to the agent (headless, offline, synthetic fixture inputs): You are a workday assistant on a mock calendar (no real accounts). Complete requests.md by editing calendar.json. Inspired by https://agentusecases.ai (listing: 'Test AI browser agents on real workday tasks', natesnewsletter.substack.com) (unverified; not checked with the author). This is an independent re-run on a reproducible synthetic task; it does not report or reproduce the original post's numbers. Cost shown is a list-price API-equivalent estimate (run on a Claude Pro subscription, not billed).
Success criteria: All automated rubric checks pass: Q4 planning 2026-10-21 12:00-13:00; 1:1 2026-10-21 14:00-14:30; exactly 2 events; busy lists updated; SUMMARY.md written.
Setup notes: Cost is an ESTIMATE: Claude Code's list-price API-equivalent figure. The run used a Claude Pro subscription, so nothing was billed. input_tokens = all input incl. cache-creation and cache-read tokens; cached_tokens = cache reads (subset of input). Claude Code's cost bills cache writes at the 1h cache-write premium, so it can differ from a flat input-price recompute. Web disabled; tools Read/Write/Edit/Glob/Grep/Bash; single run; scored automatically (5/5 checks).
Estimated cost List-price estimate; not actually billed.
Metrics
- Estimated cost
- $0.0040
- Wall time
- 17s
- Human step-ins
- 0
- Turns
- 4
| Model | Input tok | Output tok | Cost |
|---|---|---|---|
| AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on | 60,519 | 3,058 | $0.0040 |
Reproduce this run
- Use the use case prompt on the use case page.
- Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via
Claude Code 2.1.299 (headless -p), cloud. - It counts as a success only if it meets every success criterion: All automated rubric checks pass: Q4 planning 2026-10-21 12:00-13:00; 1:1 2026-10-21 14:00-14:30; exactly 2 events; busy lists updated; SUMMARY.md written.
- Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.
How runs are scored: Methodology.
Was this useful?
Automated checks
Trust 80% comes from these checks (how Trust % works).
- Run date is not in the future : passed
- Model is in the dated pricing table : passed
- Cost recomputed from tokens matches : passed
- Cached tokens are consistent : passed
- Output throughput is plausible : passed
- Autonomy matches step-ins : passed
- Step-ins do not exceed turns : passed
- Retries agree with first-try result : passed
- Evidence log SHA-256 matches : passed
- Evidence log is substantial : passed
Evidence log
{
"duration_api_ms": 15960,
"stop_reason": "end_turn",
"session_id": "[REDACTED]",
"total_cost_usd": 0.00399122,
"usage": {
"input_tokens": 6,
"cache_creation_input_tokens": 9771,
"cache_read_input_tokens": 50742,
"output_tokens": 3058,
"output_tokens_details": {
"thinking_tokens": 1144
},
"server_tool_use": {
"web_search_requests": 0,
"web_fetch_requests": 0
},
"service_tier": "standard",
"cache_creation": {
"ephemeral_1h_input_tokens": 9771,
"ephemeral_5m_input_tokens": 0
},
"inference_geo": "global",
"iterations": [
{
"input_tokens": 2,
"output_tokens": 344,
"cache_read_input_tokens": 19555,
"cache_creation_input_tokens": 2755,
"cache_creation": {
"ephemeral_5m_input_tokens": 0,
"ephemeral_1h_input_tokens": 2755
},
"type": "message"
}
],
"speed": "standard",
"fallback_credit": null
},
"modelUsage": {
"claude-haiku-5-5": {
"inputTokens": 6,
"outputTokens": 3058,
"cacheReadInputTokens": 50742,
"cacheCreationInputTokens": 9771,
"webSearchRequests": 0,
"costUSD": 0.00399122,
"contextWindow": 1000000,
"maxOutputTokens": 128000,
"thinkingTokens": 1144,
"canonicalModel": "claude-haiku-5-5",
"provider": "firstParty",
"costBasis": "list"
}
},
"permission_denials": [],
"terminal_reason": "completed",
"fast_mode_state": "off",
"fast_mode_disabled_reason": "sdk_opt_in_required",
"subagent_stats": {
"spawned": 0,
"requested": {
"background": 0,
"foreground": 0,
"unset": 0
},
"started_in_background": 0,
"max_depth": 0,
"spawned_by_subagents": 0,
"completed": 0,
"failed": 0,
"killed": {
"parent": 0,
"user": 0,
"system": 0
},
"refused": {
"depth_limit": 0,
"concurrency_limit": 0,
"budget": 0
},
"by_type": {}
},
"safety_stops": 0,
"is_error": false,
"num_turns": 4,
"subtype": "success",
"api_error_status": null,
"result": "All three tasks in requests.md are done. I edited `calendar.json` and wrote `SUMMARY.md`.\n\n- **Task A: Q4 planning**, 2026-10-21 12:00\u201313:00 with bob, priya and marco. On 2026-10-20 the three calendars leave no 60-minute gap within working hours, so 10-21 is the first day that works.\n- **Task B: 1:1 bob/priya**, 2026-10-21 14:00\u201314:30. The 12:00\u201313:00 slot is taken by Task A, and priya is busy until 14:00.\n\nI also added both meetings to bob's and priya's busy lists (and marco's for Q4 planning), so later scheduling sees them. Task B only asked to schedule the 1:1, so the busy-list entries are my call; remove them if you want the 1:1 kept out of the busy lists. All times are America/New_York.\n\nFiles are in `./usecase-runner/runs/24-haiku/work/`:\n- `calendar.json`\n- `SUMMARY.md`",
"ttft_ms": 2953,
"type": "result",
"duration_ms": 16297,
"uuid": "[REDACTED]",
"ttft_stream_ms": 2856,
"time_to_request_ms": 238,
"first_content_frame_ms": 2857,
"queued_turn_count": 0,
"result_index": 0
}