client role · bench v0.3.0 · 430 hermetic trials · swept 2026-07-18
Each trial drops a frontier coding agent in an offline workspace with read/write/edit/bash and a live MCP endpoint, and asks it to get something done over Streamable HTTP —
no SDK, no client library. The runner grades what actually went over the wire: 0 if any MUST
checkpoint fails (completing the task is itself a MUST), otherwise the fraction of SHOULD
checkpoints passed.
mean score across all 430 trials with no spec material in the workspace — priors only. That is the capability claim.
the same matrix with the pinned spec repo on disk — +43 points for reading the spec. Nothing else about the run changes.
Ranked by the no-docs mean score. The track shows the same two numbers as the columns: where a model lands on priors alone, and where it lands with the spec in the workspace.
no docs full docs
| # | Model | No docs | Full docs | Gap | 0 → 100% | Trials | Mays | Nominal cost |
|---|---|---|---|---|---|---|---|---|
| 1 | openai/gpt-5.6-sol:medium | 40% | 84% | +44 | | 86 | 13/14 | $34.29 |
| 2 | anthropic/claude-opus-4-8:medium | 35% | 65% | +30 | | 86 | 13/14 | $47.84 |
| 3 | openai/gpt-5.6-luna:medium | 26% | 70% | +44 | | 86 | 14/14 | $5.81 |
| 4 | openai/gpt-5.6-terra:medium | 26% | 72% | +46 | | 86 | 13/14 | $14.77 |
| 5 | moonshotai/kimi-k2.7-code | 12% | 59% | +47 | | 86 | 13/14 | $17.48 |
One trial per cell (k=1). Cost is nominal — token usage × public API pricing; the actual runs went through subscription tokens. MAY checkpoints never move the score.
Mean score per spec revision. 2025-11-25 is the current stable protocol; 2026-07-28 is the next revision — stateless mode, multi-round tool results, async tasks, the header contract.
| Spec version | No docs | Full docs | Gap | 0 → 100% | Trials |
|---|---|---|---|---|---|
| 2025-11-25 | 49% | 66% | +17 | | 210 |
| 2026-07-28 | 7% | 74% | +67 | | 220 |
With the spec on disk the newer revision scores 74% — higher than the older one. The new surface is learnable; it just isn't memorised.
Scenarios grouped into capability areas. Per-scenario results live on the heatmap.
| Capability area | No docs | Full docs | 0 → 100% | Scenarios |
|---|---|---|---|---|
| Tools, resources & prompts calling a tool, reading a resource, getting a prompt, all three in one turn | 75% | 85% | | 4 |
| Streaming & transport SSE responses, GET streams, no JSON-RPC batching, mid-call stream drops | 72% | 84% | | 5 |
| Lifecycle & session initialize, protocol-version header, session echo and recovery, instructions | 43% | 72% | | 6 |
| Authorization & identity PKCE, DCR, client-ID metadata documents, resource indicators, refresh, enterprise identity | 15% | 48% | | 8 |
| Multi-round tool results 2026-07-28: retry a tool call with inputResponses, echo requestState, fresh ids | 10% | 78% | | 4 |
| Elicitation handing a form or a URL back to the user mid-call, then resuming | 10% | 20% | | 2 |
| Request-header contract 2026-07-28: Mcp-Method, Mcp-Name and annotated Mcp-Param-* headers mirroring the body | 0% | 100% | | 2 |
| Async tasks 2026-07-28: declare the tasks extension, poll a taskId to completion, cancel one | 0% | 95% | | 2 |
| Stateless mode 2026-07-28: no initialize, no session id, per-request _meta instead | 0% | 90% | | 5 |
164 of 430 trials scored below a clean pass. The runner records one reason per trial.
Same models, same scenarios, same harness: 27% on priors alone, 70% with the pinned spec repo in the workspace. Every model gains, and the spread between models (40% down to 12% no-docs) is smaller than the docs effect on any single one of them.
2025-11-25 scores 49% with no docs; 2026-07-28 scores 7%. Not one no-docs trial carried the per-request metadata the newer stateless framing requires, and most of them fell straight back to the initialize-plus-session-id lifecycle they already know.
Request-header contract, Async tasks, Stateless mode — not one trial scored above zero from priors, and every one of them reaches 90% or better once the spec is on disk. These are read-or-fail surfaces, not guess-from-shape ones.
The one area where docs barely help: 10% → 20%. Most trials never declare the elicitation capability at all, so the server never gets to ask; the ones that do declare it accept the request and then never open the URL. No model met that obligation in either condition.
102 of the 164 sub-par trials finished the job and still broke a MUST — the tool call worked, the wire was wrong. The other 62 never finished the job at all — a failed task assertion, a timeout, the turn cap, or a crash.
The biggest area — 8 scenarios, 130 trials, 45% of the nominal spend — and it still only reaches 48% with the spec open. 4 scenario cells were never passed by any model in either condition and most of them are here: refusing an authorization server that does not advertise PKCE, and preferring a client-ID metadata document over dynamic registration.
none) mean score — priors only. Docs-assisted means sit beside it.none = nothing but the prompt. full = the whole pinned
modelcontextprotocol spec repo, vendored into the workspace. The environment is offline in both
cases.
bash runs inside a throwaway container on an
internal network, reaching only the reference MCP server. Assertions come from the runner's own
transport log — never from anything the agent wrote.