Spot experiment readouts whose headline metric points the wrong way, and replace it with the right one
Exact prompt
# Check whether each experiment readout uses a misleading metric Inputs: `requests/R1.json` ... `requests/R4.json`. Each is a readout of an A/B test with a `headline_metric` and a `headline_claim`. Conversion readouts give visitors and conversions per segment and variant. Order-value readouts give every order value per variant. Rules: - Conversion readouts: the headline metric is the pooled conversion rate (all conversions / all visitors per variant). The right metric is the segment-standardised rate: each variant's segment rates weighted by the segment's share of all visitors in the test (both variants together). The readout is misleading when the two metrics pick different winners; reason_code "simpson". - Order-value readouts: the headline metric is the mean order value. The right metric is the median order value (for an even count, the average of the two middle values). The readout is misleading when the two metrics pick different winners; reason_code "outlier". - If the readout is not misleading, reason_code is "none" and the right metric is still reported. - Rates are fractions rounded half-up to 4 decimals; medians are rounded half-up to the cent. - winner = the variant ("A" or "B") with the higher value on the right metric. Write `out/review.json`: `{"R1": {"misleading": <bool>, "reason_code": "simpson|outlier|none", "right_metric": "standardised_rate|median_order_value", "A": <value>, "B": <value>, "winner": "A|B"}, ...}` for all four. ## How this run works - Work on your own: nobody will answer questions in this task, and writing to `ask/` counts against the run. - Write only under `out/`. Create `out/` if it is missing. Inputs are synthetic; don't fetch anything from the network. - Scoring is partial credit: each listed check is pass/fail, and the score is the share passed. Numbers are compared to within 0.005 unless the task says otherwise.
Review four A/B readouts. Conversion readouts can hide Simpson's paradox behind a skewed segment mix; order-value readouts can be swung by a few huge orders. For each, decide whether the headline is misleading, name the reason, compute the right metric (standardised rate or median) and the true winner. Each seed has at least one misleading and one clean readout. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked). Folder-restore note: this run's output files under work/out/ were lost in a 1:22 AM ET restore on Oct 21, 2026. The trace, step-in exchanges and output-hash list were checked against the recorded SHA-256 values and match; the hashes of the individual output files inside that list can no longer be recomputed. Measured data unchanged.
Success criteria: misleading_flags reason_codes standardised_rates medians winners
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0055, 18s).
All runs (4)
75% pass rate across 4 eligible runs: 3 pass, 1 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| QQwen3 Coder 480B Turbo (Venice)thinking on | $0.0046 | 10s | 0 | Fully autonomous | partial | Trust 80% | 2026-10-28 | |
| AClaude Opus 5.5default · thinking on | $0.14 | 19s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 | |
| AClaude Haiku 5.5default · thinking on | $0.0055 | 18s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 | |
| AClaude Sonnet 5.5default · thinking off | $0.05 | 11s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-21 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.