Methodology

Mean score for coding agents implementing MCP Streamable HTTP from scratch, without an SDK or client library. Any failed MUST, task completion included, scores 0. Otherwise the score is the fraction of SHOULD checkpoints passed.

The trial: An agent gets an offline workspace, four tools (read/write/edit/bash) and a task. Client role: drive a live reference MCP server. Server role: stand up a Streamable HTTP endpoint from scratch and leave it running; the runner probes it after the session.
Score: Every checkpoint asserts one RFC 2119 sentence of the pinned spec. A failed MUST zeroes the trial (finishing the task is itself a MUST); otherwise the score is the fraction of SHOULD checkpoints passed — 1.0 when a scenario has none. MAY checkpoints never move the score; their tally is a ranking tiebreaker.
Spec revisions: Each revision is graded as its own protocol surface with its own scenario set — 2026-07-28 (stateless lifecycle, multi-round tool results, async tasks, the header contract) is not comparable scenario-for-scenario with 2025-11-25, which is why every table splits by revision first.
Docs axis: none = nothing but the prompt: what the model knows. full = the complete pinned modelcontextprotocol spec repo vendored into the workspace: whether it can use the docs. The environment is offline either way and nothing else changes.
Isolation: The agent's bash executes inside a fresh credential-free container on an internal-only Docker network; pi itself stays on the host and no credentials ever enter the container. Assertions come from the runner's own transport observations, never from anything the agent wrote. Before the agent starts, the harness proves the workspace mount works in both directions; infrastructure failures are recorded as infra-error and excluded from every leaderboard.
Calibration: A conformant oracle implementation must score 1.0 on every scenario and a null agent must score 0, re-verified in CI. Scores re-derive from stored checkpoints on every build.

Scenario corpus, reference server, prober and all run results are in the repo. Design decisions live in its CONTEXT.md, including every documented deviation from the spec text.

Every cell is k=1. Want more models, thinking levels, or higher k? Reach out to sponsor tokens, or contribute on GitHub.