Claude Haiku 5.5 (effort default (no --effort flag) · thinking on) on Extract 40 fields from a 125k-token RFP into JSON

AClaude Haiku 5.5default · thinking on · Claude Code 2.1.299 (headless -p) · cloud · 2026-10-16 · by @bobsiopener

Trust 80%Fully autonomoussuccess · first tryCompare

Estimated cost: $0.13
Wall time: 10s
Human step-ins: 0
Outcome: Pass

Compare with other models on this task →

Exact prompt (this run)

The full text of a Request for Proposals (RFP) is provided on standard input. Extract the 40 fields below and reply with ONLY a JSON object (no prose, no code fence) whose keys are exactly these field names. Use YYYY-MM-DD for dates, plain numbers without $ signs or commas for numeric fields, and true/false for booleans. If the document later amends a value (for example in an addendum), report the final amended value. If a field is not stated in the text you were given, use null. Do not use any tools. Fields: - issuing_agency: name of the agency issuing the RFP - rfp_number: RFP/solicitation number - project_title: official project title - procurement_officer_name: name of the single point of contact / procurement officer - procurement_officer_email: procurement officer email - procurement_officer_phone: procurement officer phone - rfp_issue_date: date the RFP was issued (YYYY-MM-DD) - questions_deadline: final deadline for written questions (YYYY-MM-DD) - pre_proposal_conference_date: pre-proposal conference date (YYYY-MM-DD) - pre_proposal_conference_mandatory: is attendance at the pre-proposal conference mandatory (true/false) - proposal_due_date: final proposal due date (YYYY-MM-DD) - proposal_due_time: proposal due time incl. time zone - submission_method: how proposals must be submitted - number_of_hard_copies: number of printed hard copies required (integer, 0 if none) - technical_proposal_page_limit: page limit for the technical proposal (integer) - minimum_font_size_pt: minimum font size in points (number) - proposal_validity_days: days the proposal must remain valid (integer) - contract_term_years: initial contract term in years (integer) - renewal_options_count: number of optional renewal periods (integer) - not_to_exceed_budget_usd: not-to-exceed budget for THIS procurement in USD (number) - bid_bond_percent: bid bond as % of proposed price (number) - performance_bond_percent: performance bond as % of contract value (number) - general_liability_per_occurrence_usd: prime contractor's commercial general liability per-occurrence limit in USD (number) - cyber_liability_usd: cyber/network security liability limit in USD (number) - liquidated_damages_per_day_usd: liquidated damages per calendar day in USD (number) - go_live_date: required production go-live date (YYYY-MM-DD) - uptime_sla_percent: monthly availability SLA in percent (number) - p1_response_time_hours: vendor response time for Priority 1 / critical incidents in hours (number) - data_retention_years: years records must be retained (integer) - eval_weight_technical: evaluation weight (points or %) for technical approach - eval_weight_cost: evaluation weight for cost/price - eval_weight_experience: evaluation weight for experience/qualifications - eval_weight_local_business: evaluation weight for local business preference - min_reference_projects: minimum number of comparable reference projects required (integer) - min_years_in_business: minimum years the proposer must have been in business (integer) - dbe_goal_percent: DBE/disadvantaged business participation goal in % (number) - retainage_percent: retainage withheld from each payment in % (number) - payment_terms_days: days after approved invoice that payment is made (integer) - max_concurrent_users: number of concurrent users the system must support (integer) - asset_records_to_migrate: number of asset records to be migrated (integer) [stdin: rfp.txt, 371122 chars]

Pulling 40 fields out of a roughly 125k-token request-for-proposals document, either in one prompt or in two halves merged afterwards. The document hides three addendum overrides and several near-miss values. It measures accuracy and how the price changes when a single prompt crosses a 100k-token long-context pricing threshold. Original benchmark task written for usecaserank.com (not based on a specific public post). Headless, offline run on a reproducible fixture. Each run's notes say how its cost was measured (estimates are marked).

Success criteria: Field-level accuracy ≥ 95% against a hand-labelled key.

Setup notes: Synthetic 40-field RFP (~371k chars, generated by batch2/p05/gen.py, seed 20261016) with 3 addendum overrides and 4 distractor values. Mode 'whole': whole document in one prompt via stdin. All tools disabled (pure in-context extraction). Scored automatically against answer_key.json (exact match after normalization). The input document was generated synthetically by a script (not a real RFP). Run variant: mode=whole. Claude Code's own reported cost for claude-haiku-5.5: $0.130571. Score: 40/40. Cost is an ESTIMATE: measured tokens x pricing.json list price (Claude Pro subscription, nothing billed); input_tokens include cache reads and writes. Single run. Comparison on this use case: whole document in one prompt (about 132k input tokens) vs chunked (two prompts of about 71k-74k tokens, each under 100k, plus a small merge call by the same model), each on Haiku 5.5 and Sonnet 5.5. Price-cliff context: in pricing.json, a Haiku 5.5 request whose prompt is over 100,000 input tokens is billed at long-context rates (5x: $0.50/M input, $2.50/M output instead of $0.10/$0.50); Sonnet 5.5 has no long-context tier there. So only the Haiku whole-document run crosses the cliff.

Estimated cost List-price estimate; not actually billed.

Metrics

Estimated cost: $0.13
Wall time: 10s
Human step-ins: 0
Turns: 1

Model usage
Model	Input tok	Output tok	Cost
AClaude Haiku 5.5anthropic/claude-haiku-5.5effort default (no --effort flag) · thinking on	132,971	1,663	$0.13

Reproduce this run

Use the exact prompt shown above for this run (the use case page has the shared prompt).
Same setup: Claude Haiku 5.5 (anthropic/claude-haiku-5.5, effort default (no --effort flag) · thinking on) via Claude Code 2.1.299 (headless -p), cloud.
It counts as a success only if it meets every success criterion: Field-level accuracy ≥ 95% against a hand-labelled key.
Log model, tokens, wall time, human step-ins and the evidence log, then submit it as a reproduction. A different model? Submit a head-to-head rerun.

How runs are scored: Methodology.

Was this useful?

⇄ Rerun on another model

Automated checks

Trust 80% comes from these checks (how Trust % works).

Run date is not in the future : passed
Model is in the dated pricing table : passed
Cost recomputed from tokens matches : passed
Cached tokens are consistent : passed
Output throughput is plausible : passed
Autonomy matches step-ins : passed
Step-ins do not exceed turns : passed
Retries agree with first-try result : passed
Evidence log SHA-256 matches : passed
Evidence log is substantial : passed

Evidence log

[
 {
  "duration_api_ms": 9539,
  "stop_reason": "end_turn",
  "session_id": "[REDACTED]",
  "total_cost_usd": 0.1305706,
  "usage": {
   "input_tokens": 2,
   "cache_creation_input_tokens": 126067,
   "cache_read_input_tokens": 6902,
   "output_tokens": 1663,
   "output_tokens_details": {
    "thinking_tokens": 948
   },
   "server_tool_use": {
    "web_search_requests": 0,
    "web_fetch_requests": 0
   },
   "service_tier": "standard",
   "cache_creation": {
    "ephemeral_1h_input_tokens": 126067,
    "ephemeral_5m_input_tokens": 0
   },
   "inference_geo": "global",
   "iterations": [
    {
     "input_tokens": 2,
     "output_tokens": 1663,
     "cache_read_input_tokens": 6902,
     "cache_creation_input_tokens": 126067,
     "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 126067
     },
     "type": "message"
    }
   ],
   "speed": "standard",
   "fallback_credit": null
  },
  "modelUsage": {
   "claude-haiku-5-5": {
    "inputTokens": 2,
    "outputTokens": 1663,
    "cacheReadInputTokens": 6902,
    "cacheCreationInputTokens": 126067,
    "webSearchRequests": 0,
    "costUSD": 0.1305706,
    "contextWindow": 1000000,
    "maxOutputTokens": 128000,
    "thinkingTokens": 948,
    "canonicalModel": "claude-haiku-5-5",
    "provider": "firstParty",
    "costBasis": "list"
   }
  },
  "permission_denials": [],
  "terminal_reason": "completed",
  "fast_mode_state": "off",
  "fast_mode_disabled_reason": "sdk_opt_in_required",
  "subagent_stats": {
   "spawned": 0,
   "requested": {
    "background": 0,
    "foreground": 0,
    "unset": 0
   },
   "started_in_background": 0,
   "max_depth": 0,
   "spawned_by_subagents": 0,
   "completed": 0,
   "failed": 0,
   "killed": {
    "parent": 0,
    "user": 0,
    "system": 0
   },
   "refused": {
    "depth_limit": 0,
    "concurrency_limit": 0,
    "budget": 0
   },
   "by_type": {}
  },
  "safety_stops": 0,
  "is_error": false,
  "num_turns": 1,
  "subtype": "success",
  "api_error_status": null,
  "result": "{\"issuing_agency\": \"Harlow Valley Regional Water Authority\", \"rfp_number\": \"HVRWA-2026-0417\", \"project_title\": \"Enterprise Asset Management and Work Order System Replacement\", \"procurement_officer_name\": \"Delphine Okafor\", \"procurement_officer_email\": \"[REDACTED:email]\", \"procurement_officer_phone\": \"[REDACTED:phone]\", \"rfp_issue_date\": \"2026-11-02\", \"questions_deadline\": \"2026-11-24\", \"pre_proposal_conference_date\": \"2026-11-13\", \"pre_proposal_conference_mandatory\": false, \"proposal_due_date\": \"2026-12-22\", \"proposal_due_time\": \"2:00 PM Eastern\", \"submission_method\": \"Electronic submission via the Authority's eBid portal, plus two (2) printed hard copies delivered to the Procurement Office within two business days after the electronic deadline. Emailed or faxed proposals will not be accepted.\", \"number_of_hard_copies\": 2, \"technical_proposal_page_limit\": 45, \"minimum_font_size_pt\": 11, \"proposal_validity_days\": 180, \"contract_term_years\": 5, \"renewal_options_count\": 2, \"not_to_exceed_budget_usd\": 3850000, \"bid_bond_percent\": 5, \"performance_bond_percent\": 100, \"general_liability_per_occurrence_usd\": 2000000, \"cyber_liability_usd\": 5000000, \"liquidated_damages_per_day_usd\": 1500, \"go_live_date\": \"2028-03-31\", \"uptime_sla_percent\": 99.9, \"p1_response_time_hours\": 2, \"data_retention_years\": 10, \"eval_weight_technical\": 45, \"eval_weight_cost\": 30, \"eval_weight_experience\": 15, \"eval_weight_local_business\": 10, \"min_reference_projects\": 3, \"min_years_in_business\": 7, \"dbe_goal_percent\": 12, \"retainage_percent\": 10, \"payment_terms_days\": 45, \"max_concurrent_users\": 350, \"asset_records_to_migrate\": 182000}",
  "ttft_ms": 7893,
  "type": "result",
  "duration_ms": 9749,
  "uuid": "[REDACTED]",
  "ttft_stream_ms": 4619,
  "time_to_request_ms": 201,
  "first_content_frame_ms": 4619,
  "queued_turn_count": 0,
  "result_index": 0
 }
]