Skip to content

Effort dial sweep on one well-specified feature

Exact prompt

You are working in the notes-api repository (Express 4, Node 20, node:test + supertest, ESLint). A Redis server is running for you at the URL in the REDIS_URL environment variable (db 0 is scratch space you may flush). Ticket API-88: Add a rate-limiter middleware. Create src/middleware/rateLimit.js exporting createRateLimiter(options) that returns Express middleware implementing a per-key token bucket stored in Redis, so that several app instances sharing one Redis share the same limits. Options: - redis: an ioredis client (required) - capacity: bucket size (max burst), integer - refillPerSec: tokens added per second (may be fractional) - keyFn(req): returns the bucket key; default is req.ip - prefix: Redis key prefix, default 'rl:' Behaviour: - Each request consumes one token. If a token is available, call next() and set X-RateLimit-Limit (capacity) and X-RateLimit-Remaining (tokens left, integer, after this request). - If the bucket is empty, respond 429 with JSON {"error":"rate_limited"}, a Retry-After header (whole seconds, rounded up, at least 1) and X-RateLimit-Remaining: 0. Do not call next(). - The check-and-consume must be atomic under concurrent requests, including across app instances. - Bucket keys must not live forever in Redis: give them a TTL. - If Redis fails (error or unavailable), fail open: let the request through and do not crash. Wire it into the app so that createApp({ rateLimit: { redis, capacity, refillPerSec, ... } }) applies the limiter to all /notes routes (not /health). Add thorough tests (at least 8), keep the existing tests passing, and make sure `npm run lint` is clean.

The same feature ticket, a Redis-backed token-bucket rate limiter for an Express API, run at a range of reasoning-effort levels on models of different sizes. Nine hidden acceptance tests check burst limits, refill, atomicity under concurrency, sharing across app instances, key TTLs and fail-open behaviour, alongside lint and the agent's own tests. It shows what extra effort buys in tokens, cost and quality. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: All provided acceptance tests pass; agent-written tests ≥ 8; no lint errors.

Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 2m). Best AI model for Effort dial sweep on one well-specified feature →

Run this on another model

All runs (11)

100% pass rate across 11 eligible runs: 11 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ

Runs for this use case
ModelCostTimeStep-insⓘAutonomyOutcomeTrustⓘDateCompare
AClaude Sonnet 5.5max · thinking on$3.091.2h0Fully autonomouspassTrust 80%2026-10-17
AClaude Opus 5.5max · thinking on$4.831.7h0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5xhigh · thinking on$2.245.0h0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5high · thinking on$0.603m0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5medium · thinking on$0.362m0Fully autonomouspassTrust 80%2026-10-16
AClaude Opus 5.5low · thinking on$0.2152s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5xhigh · thinking on$1.0714m0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5high · thinking on$0.222m0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5medium · thinking on$0.1350s0Fully autonomouspassTrust 80%2026-10-16
AClaude Sonnet 5.5low · thinking on$0.1145s0Fully autonomouspassTrust 80%2026-10-16
AClaude Haiku 5.5default · thinking on$0.022m0Fully autonomouspassTrust 80%2026-10-16

Embed this result

A live badge for this use case. It links back here and updates as new runs are published.

UseCaseRank badge for Effort dial sweep on one well-specified feature

Get new runs and ranking changes weekly

Which models got better or cheaper on real tasks like this one.