Implement an API client from a written spec against a flaky local mock server (429/503/412 ETag)
Exact prompt
# Task: write an API client from a spec Read `API_SPEC.md`. Write `client.py` (stdlib only: urllib/http.client/json/time) exposing: ```python class ApiError(Exception): # has .status (int) and .body (dict) class Client: def __init__(self, base_url: str, token: str): ... def list_items(self, limit: int = 20) -> list[dict] # ALL items across all pages, in order def get_item(self, item_id: str) -> dict # raises ApiError(404) if missing def update_item(self, item_id: str, **fields) -> dict # optimistic concurrency: GET for ETag, PATCH with If-Match; on 412 re-fetch and retry (up to 3 times); returns updated item ``` All calls must handle 429 (honour Retry-After) and 503 (retry with backoff) transparently, up to 5 retries; other errors raise ApiError. A mock server is provided: `python server.py 18431` (token `secret-token`) — start it yourself to test, and STOP it when you are done (kill only the PID you started). No network beyond 127.0.0.1.
Retry-After handling, pagination tokens, optimistic concurrency with a forced conflict, typed errors. Mock injects deterministic faults. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).
Success criteria: 1. Scorer starts the mock server on port 18431 (override D05_PORT), kills only that PID afterwards, and runs 6 hidden pytest tests; pytest must exit 0 (120 s timeout). 2. Tests: list_items returns all items across pages in order; get_item works and raises ApiError with status 404 for a missing id; wrong token raises ApiError with status 401; update_item succeeds and increments version; update_item recovers from a forced 412 conflict by re-fetching and retrying; invalid qty raises ApiError 422 with body field 'qty'. 429 and 503 faults are injected throughout and must be handled transparently.
Best model for this task so far: Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) (success, $0.0098, 45s).
All runs (4)
100% pass rate across 4 eligible runs: 4 pass, 0 partial, 0 fail. Partial and failed runs are listed below with everything else.ⓘ
| Model | Cost | Time | Step-insⓘ | Autonomy | Outcome | Trustⓘ | Date | Compare |
|---|---|---|---|---|---|---|---|---|
| DDeepSeek V4.1 Flash (Venice)thinking on | $0.01 | 5m | 0 | Fully autonomous | pass | Trust 80% | 2026-10-20 | |
| AClaude Haiku 5.5default · thinking on | $0.0098 | 45s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Opus 5.5default · thinking on | $0.18 | 37s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 | |
| AClaude Sonnet 5.5default · thinking on | $0.07 | 23s | 0 | Fully autonomous | pass | Trust 80% | 2026-10-16 |
Embed this result
A live badge for this use case. It links back here and updates as new runs are published.
Get new runs and ranking changes weekly
Which models got better or cheaper on real tasks like this one.