Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Resolve package versions with a specified backtracking procedure (caret and OR constraints, yanked versions, numeric ordering)

AClaude Opus 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.32
Wall time: 3m
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt

# Resolve package versions with backtracking Our offline package mirror needs to pick one version of every package a project needs. `resolve.py` is a greedy draft that picks the newest version of everything and ignores constraints. Rewrite `resolve(index, root)` (Python 3 stdlib only) to follow this spec exactly. Example data: `index.json`, `root.json`; for them the answer is `{"esc": "0.3.1", "http": "2.6.0", "tls": "1.9.0", "tmpl": "1.5.0", "web": "2.0.0"}`. ## Data - `index`: `{package: {version_string: info}}`, `info` is `null`/`{}` or has optional `"deps": {package: constraint}` and `"yanked": true|false`. - `root`: `{package: constraint}`. - Versions are 1-3 dot-separated non-negative integers; missing parts are 0 (`1.2` = `1.2.0`) and comparison is numeric (`1.10.0 > 1.9.0`). Returned versions use the exact strings from the index. (No index has two keys of the same package with equal numeric value.) ## Constraints 1. A constraint is one or more alternatives separated by `||`; it is satisfied if **any** alternative is. An alternative is terms separated by `,`; it is satisfied if **all** terms are. Spaces around terms and between an operator and its version are allowed. 2. Terms: `==V`, `!=V`, `>=V`, `<=V`, `>V`, `<V`, `^V`, and `*` (or an empty term), which matches everything. `^V` means `>=V` and `<` the next version that increases the first non-zero part of V: `^1.4` -> `<2.0.0`, `^0.3.1` -> `<0.4.0`, `^0.0.5` -> `<0.0.6`, `^0` / `^0.0.0` -> `<0.0.1`. 3. Anything else is invalid and raises `ValueError`. Root constraints are checked before any resolving starts. ## Algorithm (the result must equal what this procedure returns) 4. State: a set of **decisions** (package -> version) and, for every package, the list of constraints currently imposed on it (from `root` and from the deps of decided versions). 5. Step: among packages that have at least one constraint and no decision, take the **alphabetically smallest**. If there is none, the decisions are the answer. If that package is not in the index, this step fails. 6. Candidates are its versions from **highest to lowest** that satisfy **every** constraint on it. A **yanked** version is skipped unless one of the constraints on that package contains a term `==V` with V numerically equal to that version (in any alternative). 7. Trying a candidate: impose its deps' constraints. The candidate fails immediately if a dep package is not in the index, or if a dep's constraint is not satisfied by an existing decision for that dep. Otherwise record the decision and continue with step 5. If everything below fails, undo this candidate (its decision and the constraints it added) and try the next candidate. 8. If all candidates fail, this step fails and the previous decision is undone (backtracking). If the very first step fails, raise `ResolutionError` (keep that class in the module). 9. Return a dict of decisions with keys in sorted order (packages never constrained are not included). 10. **Speed:** each hidden index (up to 60 packages x 12 versions, needing up to a few thousand candidate tries) must resolve within 10 seconds on our shared box. ## Scoring Hidden check groups (partial credit = fraction passed; all must pass): `operators_numeric` (1-2, numeric comparison), `caret_and_or` (1-2), `invalid_constraints` (3), `yanked_rules` (6), `backtracking` (7-8), `missing_and_unsat` (5, 7, 8), `order_determinism` (5-6), `fixture_answer`, `randomized_agreement` (300 random small indexes: same result or both `ResolutionError`), `deep_backtracking` (10).

Result must equal a fully specified DFS (alphabetical decision order, highest-first, immediate failure on decided conflicts, yanked-unless-pinned); caret rules for 0.x and OR alternatives, 300 random indexes and deeper 60-package indexes. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)

Setup notes: Use Case Scout batch 8 case i06-dependency-resolver (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring: 2.71. Claude Code's own reported cost for claude-opus-5.5: $0.323503. Score: 10/10. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.32
Wall time: 3m
Human step-ins: 0
Turns: 9

Model usage
Model	Input tok	Output tok	Cost
AClaude Opus 5.5anthropic/claude-opus-5.5effort default (no --effort flag) · thinking on	223,665	7,261	$0.32

Reproduce this run

Use the use case prompt on the use case page.
Same setup: Claude Opus 5.5 (anthropic/claude-opus-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 84220,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.32350280000000003,
  "usage": {
   "input_tokens": 18,
   "cache_creation_input_tokens": 17113,
   "cache_read_input_tokens": 206534,
   "output_tokens": 7261,
   "output_tokens_details": {
    "thinking_tokens": 2797
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 17113,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 747,
     "cache_read_input_tokens": 28198,
     "cache_creation_input_tokens": 1061,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 1061
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 18,
    "outputTokens": 7261,
    "cacheReadInputTokens": 206534,
    "cacheCreationInputTokens": 17113,
    "webSearchRequests": 0,
    "costUSD": 0.32350280000000003,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 2797,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 9,
  "subtype": "success",
  "api_error_status": null,
  "result": "I rewrote `resolve.py` to follow the spec's backtracking procedure, and on the example data it returns exactly the expected answer: `{\"esc\": \"0.3.1\", \"http\": \"2.6.0\", \"tls\": \"1.9.0\", \"tmpl\": \"1.5.0\", \"web\": \"2.0.0\"}`.\n\n**What I tested:**\n- **Constraints:** all the `^` examples from the spec match, as do numeric comparison (`1.10 > 1.9`), `||` alternatives, and empty or `*` terms.\n- **Invalid input:** things like `=1`, `~1`, `>=[REDACTED:ipv4]`, `>=` and a bare `1.0` all raise `ValueError`, and a bad root constraint raises before resolving starts.\n- **Yanked versions:** a yanked version is skipped unless there's a matching `==` pin. The match is numeric, so `==1` covers a version listed as `1.0`.\n- **Failures:** a root package that isn't in the index raises `ResolutionError`.\n- **Speed:** on a random 60-package \u00d7 12-version index, about 4,300 dependency checks (a few thousand candidate tries) took 0.05 seconds, so the 10-second limit has a lot of room.\n\n**Limits on speed:** my first random stress test didn't finish and I stopped it. Some random indexes need exponentially many tries, far more than the \"few thousand\" the spec promises. This resolver can't shortcut those cases, because the result has to match the spec's procedure exactly. An index that needs vastly more tries than that will be slow.\n\n**Choices where the spec is ambiguous:**\n- **Self-dependencies:** if a version depends on its own package, that constraint is not checked against the version itself, since the decision isn't recorded until after the dependency checks.\n- **When dependency constraints are checked:** a dependency's constraint is checked for validity only when that version is tried, so a bad one raises `ValueError` mid-resolve rather than up front.\n- **Spaces:** only plain spaces are accepted around terms, not tabs.\n\n`ResolutionError` is still in the module. I also added a small command line: `python3 resolve.py [index.json] [root.json]` prints the result.",
  "ttft_ms": 3863,
  "type": "result",
  "duration_ms": 204849,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 2965,
  "time_to_request_ms": 286,
  "first_content_frame_ms": 2965,
  "queued_turn_count": 0,
  "result_index": 0
 }
]