Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Resolve package versions with a specified backtracking procedure (caret and OR constraints, yanked versions, numeric ordering)

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.01
Wall time: 1m
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Resolve package versions with backtracking Our offline package mirror needs to pick one version of every package a project needs. `resolve.py` is a greedy draft that picks the newest version of everything and ignores constraints. Rewrite `resolve(index, root)` (Python 3 stdlib only) to follow this spec exactly. Example data: `index.json`, `root.json`; for them the answer is `{"esc": "0.3.1", "http": "2.6.0", "tls": "1.9.0", "tmpl": "1.5.0", "web": "2.0.0"}`. ## Data - `index`: `{package: {version_string: info}}`, `info` is `null`/`{}` or has optional `"deps": {package: constraint}` and `"yanked": true|false`. - `root`: `{package: constraint}`. - Versions are 1-3 dot-separated non-negative integers; missing parts are 0 (`1.2` = `1.2.0`) and comparison is numeric (`1.10.0 > 1.9.0`). Returned versions use the exact strings from the index. (No index has two keys of the same package with equal numeric value.) ## Constraints 1. A constraint is one or more alternatives separated by `||`; it is satisfied if **any** alternative is. An alternative is terms separated by `,`; it is satisfied if **all** terms are. Spaces around terms and between an operator and its version are allowed. 2. Terms: `==V`, `!=V`, `>=V`, `<=V`, `>V`, `<V`, `^V`, and `*` (or an empty term), which matches everything. `^V` means `>=V` and `<` the next version that increases the first non-zero part of V: `^1.4` -> `<2.0.0`, `^0.3.1` -> `<0.4.0`, `^0.0.5` -> `<0.0.6`, `^0` / `^0.0.0` -> `<0.0.1`. 3. Anything else is invalid and raises `ValueError`. Root constraints are checked before any resolving starts. ## Algorithm (the result must equal what this procedure returns) 4. State: a set of **decisions** (package -> version) and, for every package, the list of constraints currently imposed on it (from `root` and from the deps of decided versions). 5. Step: among packages that have at least one constraint and no decision, take the **alphabetically smallest**. If there is none, the decisions are the answer. If that package is not in the index, this step fails. 6. Candidates are its versions from **highest to lowest** that satisfy **every** constraint on it. A **yanked** version is skipped unless one of the constraints on that package contains a term `==V` with V numerically equal to that version (in any alternative). 7. Trying a candidate: impose its deps' constraints. The candidate fails immediately if a dep package is not in the index, or if a dep's constraint is not satisfied by an existing decision for that dep. Otherwise record the decision and continue with step 5. If everything below fails, undo this candidate (its decision and the constraints it added) and try the next candidate. 8. If all candidates fail, this step fails and the previous decision is undone (backtracking). If the very first step fails, raise `ResolutionError` (keep that class in the module). 9. Return a dict of decisions with keys in sorted order (packages never constrained are not included). 10. **Speed:** each hidden index (up to 60 packages x 12 versions, needing up to a few thousand candidate tries) must resolve within 10 seconds on our shared box. ## Scoring Hidden check groups (partial credit = fraction passed; all must pass): `operators_numeric` (1-2, numeric comparison), `caret_and_or` (1-2), `invalid_constraints` (3), `yanked_rules` (6), `backtracking` (7-8), `missing_and_unsat` (5, 7, 8), `order_determinism` (5-6), `fixture_answer`, `randomized_agreement` (300 random small indexes: same result or both `ResolutionError`), `deep_backtracking` (10).

Result must equal a fully specified DFS (alphabetical decision order, highest-first, immediate failure on decided conflicts, yanked-unless-pinned); caret rules for 0.x and OR alternatives, 300 random indexes and deeper 60-package indexes. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)

Setup notes: Use Case Scout batch 8 case i06-dependency-resolver (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring: 2.09. Claude Code's own reported cost for claude-haiku-5.5: $0.013052. Score: 10/10. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.01
Wall time: 1m
Human step-ins: 0
Turns: 7

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	195,945	13,841	$0.01

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 63721,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.0130519,
  "usage": {
   "input_tokens": 14,
   "cache_creation_input_tokens": 21951,
   "cache_read_input_tokens": 173980,
   "output_tokens": 13841,
   "output_tokens_details": {
    "thinking_tokens": 9311
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 21951,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 657,
     "cache_read_input_tokens": 34056,
     "cache_creation_input_tokens": 434,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 434
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 14,
    "outputTokens": 13841,
    "cacheReadInputTokens": 173980,
    "cacheCreationInputTokens": 21951,
    "webSearchRequests": 0,
    "costUSD": 0.0130519,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 9311,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 7,
  "subtype": "success",
  "api_error_status": null,
  "result": "I rewrote `resolve.py` to follow the TASK.md spec: depth-first search with backtracking, plus constraint parsing and yank handling. On the example data it returns `{\"esc\": \"0.3.1\", \"http\": \"2.6.0\", \"tls\": \"1.9.0\", \"tmpl\": \"1.5.0\", \"web\": \"2.0.0\"}`, which matches the expected answer.\n\nThe file is `./usecase-runner/runs/i06-claude-haiku/work/resolve.py`.\n\nWhat I checked:\n- **Fixture:** the example data gives the expected answer.\n- **Unit cases:** numeric comparison (`1.10.0 > 1.9.0`, `1.2` == `1.2.0`), the caret rules (`^1.4`, `^0.3.1`, `^0.0.5`, `^0`), `||` and `,` combinations, empty and `*` constraints, yanked versions with and without a matching `==V`, and invalid constraints raising `ValueError`. All passed.\n- **Random small indexes:** 300 generated cases ran without error. I did not compare them against an independent reference, so that check only shows the code doesn't crash.\n- **Timing:** a 60-package, 12-version chain resolved in about 1 ms. I did not benchmark a pathological case that forces exponential search, so the 10-second budget on the hidden deep-backtracking indexes is unverified.\n\nChoices I made where the spec was silent:\n- **Self-dependency:** a candidate's own deps are checked against that candidate's version, so a package that depends on itself with an unsatisfied constraint is rejected.\n- **Dependency constraint parsing:** dep constraints are parsed when a candidate is tried, not up front. An invalid dep constraint therefore raises `ValueError` only if the search reaches it. Root constraints are validated before any resolving.\n- **Entry point:** I kept `ResolutionError` and added a small `__main__` that takes the index and root JSON paths as arguments.",
  "ttft_ms": 1252,
  "type": "result",
  "duration_ms": 64222,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1057,
  "time_to_request_ms": 261,
  "first_content_frame_ms": 1057,
  "queued_turn_count": 0,
  "result_index": 0
 }
]