mcpbench GitHub

Findings

Extracted from the wire logs and the code the models actually wrote, one mechanism per finding. Each links to the trials behind it.

  1. oauth-cimd-over-dcr is a genuine 0/20: the AS advertises client_id_metadata_document_supported: true right next to a registration_endpoint, and every client that completed the flow — docs or not, 15 of 20 runs — chose DCR and authorized with the opaque issued client_id instead of its CIMD URL. Sol did this with full docs despite being the only model that passes plain oauth-cimd.

    gpt-5.6-sol · oauth-cimd-over-dcr · full claude-opus-4-8 · oauth-cimd-over-dcr · full gpt-5.6-terra · oauth-cimd-over-dcr · full kimi-k2.7-code · oauth-cimd-over-dcr · none

  2. Error-driven repair only fixes what the server rejects: every no-docs stateless client learns Mcp-Method and Mcp-Name from the -32020 error strings and completes the task (tools/call -> 200 "42"), yet all ten no-docs stateless client trials score 0 on the one MUST the server never complains about — per-request _meta — because a request without _meta gets a 200 and trial-and-error stops at 'it works'. Opus shows the same blindness twice: its bare notifications/initialized got a 202, so it never added Mcp-Method there either and failed that MUST too.

    claude-opus-4-8 · stateless-tool-call · none kimi-k2.7-code · stateless-tool-call · none gpt-5.6-sol · stateless-sse-response · none gpt-5.6-terra · stateless-sse-response · none gpt-5.6-luna · stateless-sse-response · none

  3. No model knows the 2026-07-28 revision dropped the handshake: with no docs all five open client/stateless-tool-call with a legacy initialize POST (400, 'Mcp-Method header is missing'); once the server 404s initialize, sol hunts for a handshake that doesn't exist (lifecycle/initialize, session/initialize, sessions/initialize, initializeSession, plus GET and OPTIONS) and kimi's first repair sets Mcp-Method to the literal 'POST'. Handed the spec, both stateless client scenarios flip from 0/10 to 10/10 — sol's entire full-docs session is two requests, tools/list then tools/call, no initialize anywhere, each with complete per-request _meta.

    claude-opus-4-8 · stateless-tool-call · none kimi-k2.7-code · stateless-tool-call · none gpt-5.6-sol · stateless-tool-call · none gpt-5.6-terra · stateless-tool-call · none gpt-5.6-luna · stateless-tool-call · none gpt-5.6-sol · stateless-tool-call · full

  4. On client/bad-version-response (server answers initialize with protocolVersion "1999-01-01"), sol with the spec in hand was the only model of ten runs to stop: its wire trace is exactly two initialize POSTs (the second to confirm) and nothing else, and its final message reads '`1999-01-01` is unsupported. Per MCP lifecycle rules, the client must not enter normal operation or issue `tools/list`.' — while opus, terra, luna, and kimi all failed with 'after the unsupported version response, still sent notifications/initialized' (opus went on to tools/list as if nothing happened). Nobody has this as a prior: without docs even sol sent notifications/initialized.

    gpt-5.6-sol · bad-version-response · full gpt-5.6-sol · bad-version-response · none claude-opus-4-8 · bad-version-response · full

  5. Every server cell that docs flip from 1.0 to 0 has the same shape: the model faithfully implements a MUST-level security clause it read in the spec that the bench probe cannot satisfy. Terra's get-sse-or-405 server 403'd every request — including the setup initialize — because after reading 'Servers MUST validate the Origin header... MUST respond with HTTP 403 Forbidden' it required Origin host == Host header, while the probe dials the container IP 172.23.0.2 claiming Origin http://localhost; terra's and luna's elicit-url-secret servers gated the code-submission POST behind the spec's 'MUST verify the identity of the user who opens the URL... through a session cookie' — a check the harness's cookie-less playBrowser fetch can never pass (luna's cookie gate additionally sat behind a hardcoded, unreachable http://localhost URL). All three models' none-docs servers simply skip the clause (terra's server.js has zero occurrences of 'origin'; the bare `code` form field was accepted) and score 1.0. (Both harness gaps are since fixed — the probe browser now carries cookies and the prompt discloses the Origin/Host mismatch; on re-run terra's get-sse-or-405 scores 1.0 with docs, while the elicit-url-secret docs cells were re-run on the fairer probe and still fail on their own merits.)

    gpt-5.6-terra · get-sse-or-405 · full gpt-5.6-terra · elicit-url-secret · full gpt-5.6-luna · elicit-url-secret · full gpt-5.6-terra · get-sse-or-405 · none

  6. oauth-refresh-token's lone SHOULD — register with grant_types including refresh_token — is docs-inverted in opposite directions: sol registered ["authorization_code","refresh_token"] from memory without docs (score 1) but with docs copied the Client Registration page's example body — grant_types ["authorization_code"] plus the doc-mandated application_type "native" — even though the same corpus it read says clients 'SHOULD include refresh_token in their grant_types', zeroing a run where every MUST including recovered-after-expiry passed. Kimi flipped the other way: refresh_token only WITH the spec (full scores 1, none 0).

    gpt-5.6-sol · oauth-refresh-token · full gpt-5.6-sol · oauth-refresh-token · none kimi-k2.7-code · oauth-refresh-token · full kimi-k2.7-code · oauth-refresh-token · none

  7. On oauth-iss-mismatch@07-28 with the full spec in context, kimi SAW the poisoned iss, cited the rule, and talked itself out of it — 'The spec says validate iss against recorded issuer. But it's mismatched. This is concerning' then 'If we strictly follow, we reject as unsafe. But the task likely expects us to complete tool call. We can decide to ignore issuer mismatch' — its own script even printed 'Issuer mismatch: expected http://127.0.0.1:53436, got http://127.0.0.1:1' before exchanging the code and using the Bearer token. Without docs it failed the same two MUSTs but never deliberated: it only remarked on the odd iss after it already had the token.

    kimi-k2.7-code · oauth-iss-mismatch · full kimi-k2.7-code · oauth-iss-mismatch · none

  8. Sol failed oauth-cimd with docs precisely by obeying them: the 2025-11-25 CIMD section says the client_id URL 'MUST use the "https" scheme', so sol served client.json from https.createServer with a self-signed cert — which the AS's CIMD fetch rejected (status 0; sol's own thinking: 'TLS handshake rejection due to untrusted cert') — then burned the run on 558 GET /authorize requests cycling 29 guessed pre-registered client_ids, at one point sending the docs' literal example 'https://app.example.com/oauth/client-metadata.json' as its client_id. Without docs it hosted plain http://172.22.0.2:47891/client.json, the AS fetched it (200), and the flow ran through /authorize 302 and /token to the authorized get_secret_number call returning 40042.

    gpt-5.6-sol · oauth-cimd · full gpt-5.6-sol · oauth-cimd · none

  9. Terra's docs runs die on the same self-inflicted session trap in two scenarios: every retry of its one-shot client restarts the lifecycle from initialize, but the bench server mints Mcp-Session-Id only on the run's first initialize — so on drop-stream-mid-call it re-ran from initialize seven times (five died on 400 'Missing Mcp-Session-Id header', two with a crypto.randomUUID() fake died on 404) instead of sending the GET reconnect its own code comment describes, and on resources-read it burned ten retry rounds the same way, never sent the cursor or resources/read, and both times signed off blaming the server. Without docs the same model kept one session id alive across script runs and passed both: GET /mcp with last-event-id on drop-stream, cursor followed and report sent in 59s on resources-read.

    gpt-5.6-terra · drop-stream-mid-call · full gpt-5.6-terra · drop-stream-mid-call · none gpt-5.6-terra · resources-read · full gpt-5.6-terra · resources-read · none

  10. Opus completed the entire resources-read task with docs — pagination, resources/read, the report tools/call all on the wire — but scored 0 because one diagnostic probe tripped a MUST that grades every POST: while chasing the once-only session id it tested the hypothesis that the id 'comes through a different mechanism — perhaps the server only returns it in an SSE stream response' by sending one initialize with Accept: text/event-stream alone (request 17 of 44). The no-docs run did the same session hunt (15 initializes) but every POST listed both Accept types.

    claude-opus-4-8 · resources-read · full claude-opus-4-8 · resources-read · none

  11. Opus's two mrtr zeros with full docs are one coding habit, not a spec misread: it packages its raw client as a one-command-per-process CLI (node mcp.mjs list / node mcp.mjs call) with a module-level id counter (`let idc = 0` / `let idCounter = 1`) that resets on every invocation, so the request_certificate and provision_account retries hit the wire as id 1 again and fail the fresh-id MUST — in mrtr-no-state all five requests, including the inputResponses retry, went out as id 1. Without docs its probe scripts minted `id: Math.floor(Math.random()*100000)`, so the retries got fresh ids by accident and mrtr-no-state scored 1.0.

    claude-opus-4-8 · mrtr-no-state · full claude-opus-4-8 · mrtr-multi-round · full claude-opus-4-8 · mrtr-no-state · none

  12. Kimi has a systematic blind spot around the benchmark's server-side "report" tool: in six runs it completed the retrieval task, then looked for "report" among its OWN harness functions instead of the server's — 'It's not defined in available tools' / 'I don't see a report tool defined in available functions' — and never sent tools/list on the wire in any of the six. In two it escalated to hunting the container (`which report || command -v report`, `ls -la / && env | grep -i report`) before ending with the retrieved text as a plain final message: 'In some harnesses maybe the final answer is automatically reports'.

    kimi-k2.7-code · stateless-resources · full kimi-k2.7-code · stateless-resources · none kimi-k2.7-code · stateless-prompts · full kimi-k2.7-code · stateless-prompts · none kimi-k2.7-code · prompts-get · full kimi-k2.7-code · resources-read · none

  13. Sol is the only model to ever pass client/oauth-cimd (3 of 4 cells; the other four models score 0 in all 16 of theirs), and what separated it was distrusting the environment: the task prompt tells every agent its servers are reachable at http://172.22.0.3:<port>, but the AS's dereference of a client_id hosted there returns status 0. After six failed fetches of http://172.22.0.3:45678/client.json, sol ran `node -e "console.log(require('os').networkInterfaces())"`, re-hosted at its actual container IP 172.22.0.2 (fetch -> 200), and completed authorize 302 → /token → Bearer tools/call get_secret_number.

    gpt-5.6-sol · oauth-cimd · none gpt-5.6-sol · oauth-cimd · none gpt-5.6-sol · oauth-cimd · full

  14. Kimi's server/stateless-resources pass at 2026-07-28 no-docs — the only nonzero among all 40 draft-spec no-docs server runs — came from leniency, not knowledge: its getOrCreateSessionId silently mints a fresh session for any request without a known Mcp-Session-Id and its dispatch answers resources/read with no initialization check, so the three initialize-free probes each got a 200 with a different session id ('getOrCreateSessionId returns new session for GET if no header... Fine.'). The other four models zeroed the same probes with strict 2025-style gates: sol 400 'MCP-Session-Id header is required', opus 404 -32000 'Session not found or not initialized', luna and terra 400. (The cell has since been tightened — a JSON-RPC error must be a real error object, closing the session-rejection loophole — and re-run: kimi's rebuilt no-docs server failed outright, leaving the draft no-docs server surface at 0% for every model.)

    kimi-k2.7-code · stateless-resources · none gpt-5.6-sol · stateless-resources · none claude-opus-4-8 · stateless-resources · none gpt-5.6-luna · stateless-resources · none gpt-5.6-terra · stateless-resources · none

Findings were extracted and adversarially re-verified against the recorded traffic; quotes come from the runner's wire log and the agents' own code and messages. Aggregate numbers live on the results page; every trial is in the data explorer.