Effort dial sweep on one well-specified feature
Exact prompt
You are working in the notes-api repository (Express 4, Node 20, node:test + supertest, ESLint). A Redis server is running for you at the URL in the REDIS_URL environment variable (db 0 is scratch space you may flush). Ticket API-88: Add a rate-limiter middleware. Create src/middleware/rateLimit.js exporting createRateLimiter(options) that returns Express middleware implementing a per-key token bucket stored in Redis, so that several app instances sharing one Redis share the same limits. Options: - redis: an ioredis client (required) - capacity: bucket size (max burst), integer - refillPerSec: tokens added per second (may be fractional) - keyFn(req): returns the bucket key; default is req.ip - prefix: Redis key prefix, default 'rl:' Behaviour: - Each request consumes one token. If a token is available, call next() and set X-RateLimit-Limit (capacity) and X-RateLimit-Remaining (tokens left, integer, after this request). - If the bucket is empty, respond 429 with JSON {"error":"rate_limited"}, a Retry-After header (whole seconds, rounded up, at least 1) and X-RateLimit-Remaining: 0. Do not call next(). - The check-and-consume must be atomic under concurrent requests, including across app instances. - Bucket keys must not live forever in Redis: give them a TTL. - If Redis fails (error or unavailable), fail open: let the request through and do not crash. Wire it into the app so that createApp({ rateLimit: { redis, capacity, refillPerSec, ... } }) applies the limiter to all /notes routes (not /health). Add thorough tests (at least 8), keep the existing tests passing, and make sure `npm run lint` is clean.
The same feature ticket, a Redis-backed token-bucket rate limiter for an Express API, run at a range of reasoning-effort levels on models of different sizes. Nine hidden acceptance tests check burst limits, refill, atomicity under concurrency, sharing across app instances, key TTLs and fail-open behaviour, alongside lint and the agent's own tests. It shows what extra effort buys in tokens, cost and quality. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: All provided acceptance tests pass; agent-written tests ≥ 8; no lint errors.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.02, 2m). Best AI model for Effort dial sweep on one well-specified feature →
All runs (11)
100% pass rate across 11 eligible runs: 11 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| AClaude Sonnet 5.5max · thinking on | $3.09 | 1.2h | 0 | Fully autonomous | pass | Trust 80% | 2026-10-17 | |
| AClaude Opus 5.5max · thinking on | $4.83 | 1.7h | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5xhigh · thinking on | $2.24 | 5.0h | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5high · thinking on | $0.60 | 3m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5medium · thinking on | $0.36 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5low · thinking on | $0.21 | 52s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5xhigh · thinking on | $1.07 | 14m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5high · thinking on | $0.22 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5medium · thinking on | $0.13 | 50s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5low · thinking on | $0.11 | 45s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Haiku 5.5default · thinking on | $0.02 | 2m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.