Skip to content

Spot experiment readouts whose headline metric points the wrong way, and replace it with the right one

Exact prompt

# Check whether each experiment readout uses a misleading metric Inputs: `requests/R1.json` ... `requests/R4.json`. Each is a readout of an A/B test with a `headline_metric` and a `headline_claim`. Conversion readouts give visitors and conversions per segment and variant. Order-value readouts give every order value per variant. Rules: - Conversion readouts: the headline metric is the pooled conversion rate (all conversions / all visitors per variant). The right metric is the segment-standardised rate: each variant's segment rates weighted by the segment's share of all visitors in the test (both variants together). The readout is misleading when the two metrics pick different winners; reason_code "simpson". - Order-value readouts: the headline metric is the mean order value. The right metric is the median order value (for an even count, the average of the two middle values). The readout is misleading when the two metrics pick different winners; reason_code "outlier". - If the readout is not misleading, reason_code is "none" and the right metric is still reported. - Rates are fractions rounded half-up to 4 decimals; medians are rounded half-up to the cent. - winner = the variant ("A" or "B") with the higher value on the right metric. Write `out/review.json`: `{"R1": {"misleading": <bool>, "reason_code": "simpson|outlier|none", "right_metric": "standardised_rate|median_order_value", "A": <value>, "B": <value>, "winner": "A|B"}, ...}` for all four. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.

Review four A/B readouts. Conversion readouts can hide Simpson's paradox behind a skewed segment mix; order-value readouts can be swung by a few huge orders. For each, decide whether the headline is misleading, name the reason, compute the right metric (standardised rate or median) and the true winner. Each seed has at least one misleading and one clean readout. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Folder-restore note: this run's output files under work/out/ were lost in a 1:22 AM ET restore on Oct 21, 2026. The trace, step-in exchanges and output-hash list were checked against the recorded SHA-256 values and match; the hashes of the individual output files inside that list can no longer be recomputed. Measured data unchanged.

Success criteria: misleading_flags reason_codes standardised_rates medians winners

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0055, 18s).

Run this on another model

All runs (4)

75% pass rate across 4 eligible runs: 3 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
QQwen3 Coder 480B Turbo (Venice)thinking on$0.004610s0Fully autonomouspartialTrust 80%2026-10-28
AClaude Opus 5.5default · thinking on$0.1419s0Fully autonomouspassTrust 80%2026-10-21
AClaude Haiku 5.5default · thinking on$0.005518s0Fully autonomouspassTrust 80%2026-10-21
AClaude Sonnet 5.5default · thinking off$0.0511s0Fully autonomouspassTrust 80%2026-10-21

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Spot experiment readouts whose headline metric points the wrong way, and replace it with the right one

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.