Methodology

A run is one attempt by one model (or model combination) at one use case, with the model, tokens, cost, wall time, human step-ins, turns, retries, setup and an evidence log recorded. A use case is the task: a prompt plus success criteria. Rankings only compare runs to other runs of the same use case.

Success

Each run has an outcome of success, partial or fail, plus whether it succeeded on the first try and how many retries it took.

• A run counts as a success only when it meets 100% of the use case's rubric (its success criteria). Anything less is not a completed task.
• Among runs that complete the task, models are then separated by cost, wall time and human step-ins.

Pass rate & partial runs

A model's pass rate is the share of its eligible runs that met the full rubric. Rankings, the homepage and every use-case page show passes, partials and fails side by side; failed and partial runs are never hidden.

Partial means the run did real work but missed part of the rubric. It is not a success and never counts toward the pass rate. Inside the score it earns half the outcome credit (0.5) instead of 0, so a run that got most of the way is not scored the same as one that produced nothing.

Trend arrows

The ▲/▼ next to a model is its rank change versus 7 days ago. We rebuild the leaderboard as it stood then, using only runs that had been published by that date (and not yet replaced), and compare rank numbers. If a model wasn't ranked back then, it shows “new”; if there is no ranked history at all, no arrow is shown. Trends are never estimated.

Step-ins (interventions)

A step-in is any message after the initial prompt that corrects, clarifies, approves, redirects or nudges the agent. Scripted nudges count too: an automated “continue” sent by a harness is a step-in, the same as a typed one.

Autonomy is derived from step-ins: 0 is fully autonomous, 1–3 is lightly guided, 4 or more is heavily guided. A submission whose claimed autonomy disagrees with its step-in count is flagged.

Cost

• Cost is computed from the run's token counts and the dated rates in data/pricing.json, using the price row in effect on the run date. Input, output, cached-input and cache-write tokens are priced separately.
• Long-context tiers are included: when a model has a higher rate for prompts above a token threshold, it is applied per request using the submission's per-request token counts. If a submission only has totals, base rates are used and the run gets a long_context_unpriced warning.
• Costs are list-price, API-equivalent figures, not necessarily what was billed. Local models cost $0.
• A claimed cost more than 25% away from the computed cost raises a cost_mismatch flag.
• Cost per success = total spend across attempts (cost × (1 + retries)) ÷ outcome value (success 1, partial 0.5). Failed runs have no cost per success.

Score

Every run gets a score from 0 to 100: 100 × the weighted sum of five metrics, each between 0 and 1. The default weights are below; you can change them on the rankings page.

A model's score is the weighted average of its runs' scores. Each run is weighted by its Trust % and by recency, with a half-life of 90 days. Models are grouped by name plus settings (for example “thinking off”), so different settings are ranked separately.

Models with fewer than 5 runs are listed but not given a rank number. When a run is a rerun of another run on a different model, the pair feeds a Bradley-Terry head-to-head rating, blended in at 30% once there are 20 or more pairs.

Confidence interval

The range after each score is a 90% confidence interval: score ± 1.645 × √(weighted variance ÷ effective sample size), clipped to 0–100. A model with an effective sample size of 1 or less has no meaningful interval, so none is shown (“too few runs for a range”); internally it is treated as ±50. On the rankings page the whisker on each score bar spans the interval. When two models' ranges overlap, the data does not clearly separate them.

Trust %

Every run gets a trust score from automated plausibility checks when it is submitted:

• Starts at 50%.
• +20% when the evidence log is substantial (at least 200 characters).
• +10% when no hard flag was raised.
• −5% for each warning (for example, a model missing from the pricing table, or unpriced long-context requests).
• +10% each time someone independently reproduces the run with the same models on the same use case, which also marks it Verified.

Checks include recomputing cost, plausible output throughput, step-ins vs. turns, retries vs. first-try, autonomy vs. step-ins and the evidence log's SHA-256 hash.

Under review

Any hard flag sends a run to Under review. It stays public and is never deleted, but it does not count toward rankings until a reviewer clears every hard flag. Sponsorship never affects rankings.

Funding & conflicts

• UseCaseRank is independently run by Bob Bullock.
• No AI provider has given free credits, discounts or sponsorship for runs.
• Every run is paid for by the site owner, Bob Bullock, from his own accounts and subscriptions.
• There are no sponsorships, affiliate links or paid placements on the site today.
• Rankings are never pay-to-play: no payment changes a score, a weight, a rank, or which runs are included.
• Any future compute or credit contributions will be listed on this page, and they will never affect rankings or which models get run.
• If a paid placement is ever added, it will be clearly labeled as paid and kept separate from the rankings.

Costs shown are list-price, API-equivalent figures computed from token counts and dated prices, not necessarily what was billed. See also Disclosures.