Claude Opus 5.5 (effort default (no --effort flag) · thinking on) on Call six A/B tests correctly through traffic-split mismatches, burn-in effects, under-filled arms and guardrail breaches

[
 {
  "duration_api_ms": 21140,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.1468306,
  "usage": {
   "input_tokens": 8,
   "cache_creation_input_tokens": 10692,
   "cache_read_input_tokens": 70813,
   "output_tokens": 2355,
   "output_tokens_details": {
    "thinking_tokens": 197
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 10692,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 683,
     "cache_read_input_tokens": 20897,
     "cache_creation_input_tokens": 1941,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 1941
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-opus-5-5": {
    "inputTokens": 8,
    "outputTokens": 2355,
    "cacheReadInputTokens": 70813,
    "cacheCreationInputTokens": 10692,
    "webSearchRequests": 0,
    "costUSD": 0.1468306,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 197,
    "canonicalModel": "claude-opus-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 4,
  "subtype": "success",
  "api_error_status": null,
  "result": "I wrote `out/decisions.json` with a call for all six experiments. Each one was judged on the days after its burn-in only (the first 3 or 7 days were dropped).\n\n| Experiment | Decision | Lift | Why |\n|---|---|---|---|\n| EXP-onboard-1 | inconclusive | +6.20% | z = 1.58, not significant |\n| EXP-checkout-2 | inconclusive | \u22124.94% | z = \u22121.71, not significant |\n| EXP-pricing-3 | **no-ship** | +8.10% | Conversion win (z = 2.83), but support contacts per user rose 34.8%, well over the 10% allowed |\n| EXP-reco-4 | inconclusive | +10.29% | z = 2.78 would be a win, but each arm has about 23.8k users against 51,464 planned |\n| EXP-onboard-5 | **ship** | +10.76% | z = 3.32, and support contacts per user fell 3.7% |\n| EXP-reco-6 | **invalid** | \u2014 | Users don't match the planned 50/50 split: 21,176 control vs 25,410 treatment (chi-square 384.8, threshold 10.828) |\n\n- **Lift for every valid experiment:** EXP-reco-6 has no lift because it's invalid, as the task specifies. Lifts are rounded half-up to 2 decimals.\n- **EXP-onboard-1's uneven arms:** with its 80/20 split, the treatment arm has about 13.6k users and control about 54.6k. Both are above the 11,972 planned per arm, and the result isn't significant anyway, so it's inconclusive however \"planned users per arm\" is read.\n- **Data check:** the data was clean. Every row's day number matched its date counted from the experiment's start, with no gaps or duplicates.\n\nThe script that produced the file is `/tmp/s/run.py`, outside the work folder because only `out/` may be written to.",
  "ttft_ms": 2142,
  "type": "result",
  "duration_ms": 21572,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 1001,
  "time_to_request_ms": 254,
  "first_content_frame_ms": 1001,
  "queued_turn_count": 0,
  "result_index": 0
 }
]