Skip to content

Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Optimal single-machine job sequence (weighted tardiness with precedence)

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.01
Wall time
11m
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Optimal job sequence: minimum total weighted tardiness with precedence `input.json` (synthetic) has `start` (time the machine becomes free), `jobs` (`id`, processing time `p`, weight `w`, `due`) and `precedence` (pairs [a, b]: a must come before b, not necessarily directly). One machine, no idle time, no preemption. Write `solve.py` (Python 3 standard library only); `python solve.py input.json answer.json`. Find a sequence of all jobs that respects every precedence pair and **minimises** total weighted tardiness: sum of w * max(0, C - due), where C is the job's completion time (start + sum of p up to and including the job). Any optimal sequence is accepted. Fresh instances have **20 and 21 jobs**; each run has a 90-second limit. ## answer.json `{"sequence": [job ids in order], "total_weighted_tardiness": integer}` ## Scoring Hidden checks (partial credit; all must pass). A sequence is **valid** if it is a permutation of the job ids, respects every precedence pair, and the reported objective equals the scorer's recomputation for that sequence. It is **optimal** if that objective equals the true minimum (the scorer has the optimum). `fixture_valid_sequence`, `fixture_optimal` (given 18-job instance); `fresh_20_jobs_valid`, `fresh_20_jobs_optimal`; `fresh_21_jobs_optimal`.

Greedy rules (WSPT, EDD, swaps) give valid but suboptimal sequences; an exact method (subset dynamic programming or a tight branch and bound) is needed. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. fixture_valid_sequence 2. fixture_optimal 3. fresh_20_jobs_valid 4. fresh_20_jobs_optimal 5. fresh_21_jobs_optimal

Setup notes: Use Case Scout batch 12 case m07-weighted-tardiness-optimal (offline, synthetic fixture). Prepared with batch12/prepare.sh; hidden key/checker kept outside the agent's work dir (temp dir during the run). Prompt = TASK.md verbatim. Scored with the case's scorer: `python score.py WORKDIR KEYDIR` (python3, run from the case dir). Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring 0.88 (8 cores). Claude Code's own reported cost for claude-haiku-5.5: $0.012089. Score: 5/5. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.01
Wall time
11m
Human step-ins
0
Turns
16
Model usage
ModelInput tokOutput tokCost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on423,1308,097$0.01

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. fixture_valid_sequence 2. fixture_optimal 3. fresh_20_jobs_valid 4. fresh_20_jobs_optimal 5. fresh_21_jobs_optimal
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 49138,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.012089140000000002,
  "usage": {
   "input_tokens": 32,
   "cache_creation_input_tokens": 20034,
   "cache_read_input_tokens": 403064,
   "output_tokens": 8097,
   "output_tokens_details": {
    "thinking_tokens": 2529
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 20034,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 532,
     "cache_read_input_tokens": 32224,
     "cache_creation_input_tokens": 349,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 349
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 32,
    "outputTokens": 8097,
    "cacheReadInputTokens": 403064,
    "cacheCreationInputTokens": 20034,
    "webSearchRequests": 0,
    "costUSD": 0.012089140000000002,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 2529,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 16,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `solve.py` and it produces `answer.json` for the fixture: sequence of 18 jobs with total weighted tardiness **462**. The solver is an exact subset DP, so the result is the true optimum, not a heuristic.\n\n**How it works**\n- A set of jobs scheduled first has a fixed completion time (`start + sum of p`), so appending job j costs `w_j * max(0, C - d_j)` regardless of the order inside the set.\n- Jobs are only appended once all their predecessors (transitive closure of the precedence pairs) are already in the set, which enforces precedence.\n- The optimal sequence is recovered from parent pointers. `main()` asserts that the output is a permutation, respects every precedence pair, and that the recomputed objective matches the reported value.\n\n**Verification**\n- **Brute force:** on 300 random instances with 2\u20138 jobs, random start times, and random precedence sets, the DP matched the minimum over all permutations (0 mismatches). Each returned sequence was also checked for precedence and objective.\n- **Timing:** a 21-job instance with no precedence (the worst case, 2\u00b2\u00b9 states) takes about 4.7 s. A 21-job instance with 15 precedence pairs takes 0.3 s. Both are well under the 90 s limit.\n- The fixture runs in about 0.16 s.\n\n**Files:** `./usecase-runner/runs/m07-claude-haiku/work/solve.py` and `answer.json`. The test script lives in `/tmp/chk`, outside the work directory. The directory is not a git repository, so I did not commit anything.",
  "ttft_ms": 3735,
  "type": "result",
  "duration_ms": 683652,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1429,
  "time_to_request_ms": 256,
  "first_content_frame_ms": 1429,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly