Skip to content

Claude Sonnet 5.5 (effort default (no --effort flag) · thinking on) on Resolve package versions with a specified backtracking procedure (caret and OR constraints, yanked versions, numeric ordering)

AClaude Sonnet 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-20 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first try
Estimated cost
$0.08
Wall time
28s
Human step-ins
0
Outcome
Pass

Compare with other models on this task →

Summary

Exact prompt

# Resolve package versions with backtracking Our offline package mirror needs to pick one version of every package a project needs. `resolve.py` is a greedy draft that picks the newest version of everything and ignores constraints. Rewrite `resolve(index, root)` (Python 3 stdlib only) to follow this spec exactly. Example data: `index.json`, `root.json`; for them the answer is `{"esc": "0.3.1", "http": "2.6.0", "tls": "1.9.0", "tmpl": "1.5.0", "web": "2.0.0"}`. ## Data - `index`: `{package: {version_string: info}}`, `info` is `null`/`{}` or has optional `"deps": {package: constraint}` and `"yanked": true|false`. - `root`: `{package: constraint}`. - Versions are 1-3 dot-separated non-negative integers; missing parts are 0 (`1.2` = `1.2.0`) and comparison is numeric (`1.10.0 > 1.9.0`). Returned versions use the exact strings from the index. (No index has two keys of the same package with equal numeric value.) ## Constraints 1. A constraint is one or more alternatives separated by `||`; it is satisfied if **any** alternative is. An alternative is terms separated by `,`; it is satisfied if **all** terms are. Spaces around terms and between an operator and its version are allowed. 2. Terms: `==V`, `!=V`, `>=V`, `<=V`, `>V`, `<V`, `^V`, and `*` (or an empty term), which matches everything. `^V` means `>=V` and `<` the next version that increases the first non-zero part of V: `^1.4` -> `<2.0.0`, `^0.3.1` -> `<0.4.0`, `^0.0.5` -> `<0.0.6`, `^0` / `^0.0.0` -> `<0.0.1`. 3. Anything else is invalid and raises `ValueError`. Root constraints are checked before any resolving starts. ## Algorithm (the result must equal what this procedure returns) 4. State: a set of **decisions** (package -> version) and, for every package, the list of constraints currently imposed on it (from `root` and from the deps of decided versions). 5. Step: among packages that have at least one constraint and no decision, take the **alphabetically smallest**. If there is none, the decisions are the answer. If that package is not in the index, this step fails. 6. Candidates are its versions from **highest to lowest** that satisfy **every** constraint on it. A **yanked** version is skipped unless one of the constraints on that package contains a term `==V` with V numerically equal to that version (in any alternative). 7. Trying a candidate: impose its deps' constraints. The candidate fails immediately if a dep package is not in the index, or if a dep's constraint is not satisfied by an existing decision for that dep. Otherwise record the decision and continue with step 5. If everything below fails, undo this candidate (its decision and the constraints it added) and try the next candidate. 8. If all candidates fail, this step fails and the previous decision is undone (backtracking). If the very first step fails, raise `ResolutionError` (keep that class in the module). 9. Return a dict of decisions with keys in sorted order (packages never constrained are not included). 10. **Speed:** each hidden index (up to 60 packages x 12 versions, needing up to a few thousand candidate tries) must resolve within 10 seconds on our shared box. ## Scoring Hidden check groups (partial credit = fraction passed; all must pass): `operators_numeric` (1-2, numeric comparison), `caret_and_or` (1-2), `invalid_constraints` (3), `yanked_rules` (6), `backtracking` (7-8), `missing_and_unsat` (5, 7, 8), `order_determinism` (5-6), `fixture_answer`, `randomized_agreement` (300 random small indexes: same result or both `ResolutionError`), `deep_backtracking` (10).

Result must equal a fully specified DFS (alphabetical decision order, highest-first, immediate failure on decided conflicts, yanked-unless-pinned); caret rules for 0.x and OR alternatives, 300 random indexes and deeper 60-package indexes. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)

Setup notes: Use Case Scout batch 8 case i06-dependency-resolver (offline fixture). Prepared with batch8/prepare.sh; hidden key/tests kept outside the agent's work dir. Prompt = TASK.md verbatim. Scored with the scout's scorer: `python score.py WORKDIR KEYDIR`. Tools: Read/Write/Edit/Glob/Grep/Bash, web disabled. Single run, no retries. 1-min load average at scoring: 2.12. Claude Code's own reported cost for claude-sonnet-5.5: $0.075434. Score: 10/10. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost
$0.08
Wall time
28s
Human step-ins
0
Turns
3
Model usage
ModelInput tokOutput tokCost
AClaude Sonnet 5.5anthropic/claude-sonnet-5.5effort default (no --effort flag) · thinking on61,4643,092$0.08

Reproduce this run

  1. Use the use case prompt on the use case page.
  2. Same setup: Claude Sonnet 5.5 (anthropic/claude-sonnet-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
  3. It counts as a success only if it meets every success criterion: 1. operators_numeric 2. caret_and_or 3. invalid_constraints 4. yanked_rules 5. backtracking 6. missing_and_unsat 7. order_determinism 8. fixture_answer 9. randomized_agreement (300 indexes) 10. deep_backtracking (< 10 s each)
  4. Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Evidence log
[
 {
  "duration_api_ms": 27343,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.0754343,
  "usage": {
   "input_tokens": 6,
   "cache_creation_input_tokens": 9835,
   "cache_read_input_tokens": 51623,
   "output_tokens": 3092,
   "output_tokens_details": {
    "thinking_tokens": 1071
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 9835,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 603,
     "cache_read_input_tokens": 19901,
     "cache_creation_input_tokens": 2473,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 2473
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-sonnet-5-5": {
    "inputTokens": 6,
    "outputTokens": 3092,
    "cacheReadInputTokens": 51623,
    "cacheCreationInputTokens": 9835,
    "webSearchRequests": 0,
    "costUSD": 0.0754343,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 1071,
    "canonicalModel": "claude-sonnet-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 3,
  "subtype": "success",
  "api_error_status": null,
  "result": "I rewrote `resolve(index, root)` in `resolve.py` as a backtracking resolver that follows your spec. It returns `{\"esc\": \"0.3.1\", \"http\": \"2.6.0\", \"tls\": \"1.9.0\", \"tmpl\": \"1.5.0\", \"web\": \"2.0.0\"}` on `index.json` and `root.json`, which is the expected answer.\n\nThat fixture run is the only test I ran. I did not test the invalid-constraint cases, the yanked rules, unsatisfiable inputs or deep backtracking. I also did not time a 60-package, 12-version index, so the 10-second limit is unchecked. It is plain depth-first backtracking with no extra pruning.\n\nI made two judgment calls where the spec was silent:\n- **Index constraints:** a dep constraint stored in the index is parsed only when it is first used. A bad one raises `ValueError` at that point, not up front. Root constraints are all checked before resolving starts.\n- **Self-dependency:** if a version lists its own package as a dep, the dep check runs before that version's decision is recorded. The constraint is therefore imposed but never checked against it.\n\n`ResolutionError` is still in the module. I also added a small `__main__` block, so `python3 -I resolve.py index.json root.json` prints the result as JSON.",
  "ttft_ms": 1879,
  "type": "result",
  "duration_ms": 27685,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1373,
  "time_to_request_ms": 241,
  "first_content_frame_ms": 1374,
  "queued_turn_count": 0,
  "result_index": 0
 }
]

Get new runs and ranking changes weekly