Heatmap
Model:effort × scenario mean score (0–100). A trial scores 0 if any MUST checkpoint fails (task success is a
MUST), otherwise the fraction of SHOULD checkpoints passed; MAY checkpoints never affect the score. Defaults to
the headline none (no-docs) condition — priors only; select full to see the
docs-assisted scores. With a single version + docs selected, cells show the mean over repeat trials and link to
trial detail.
| Model | client/bad-version-response | client/composite | client/discover-instructions | client/drop-stream-mid-call | client/elicit-form | client/elicit-url | client/ema-id-jag-flow | client/get-stream | client/header-mirroring | client/initialize-handshake | client/mrtr-basic | client/mrtr-elicit-url | client/mrtr-multi-round | client/mrtr-no-state | client/no-batching | client/oauth-cimd | client/oauth-cimd-over-dcr | client/oauth-dcr-application-type | client/oauth-iss-mismatch | client/oauth-pkce | client/oauth-refresh-token | client/param-headers | client/pkce-unsupported-as | client/prompts-get | client/protocol-version-header | client/resources-read | client/session-echo | client/session-recovery | client/sse-response | client/stalled-tool | client/stateless-composite | client/stateless-prompts | client/stateless-resources | client/stateless-sse-response | client/stateless-tool-call | client/task-cancel | client/task-polling | client/tool-call-basic | server/discover | server/elicit-form-plain | server/elicit-url-secret | server/error-semantics | server/get-sse-or-405 | server/minimal-server | server/notifications-202 | server/origin-validation | server/session-handling | server/stateless-headers | server/stateless-resources | server/stateless-tool-call | server/stateless-version | server/subscriptions | server/tasks-poll | server/version-negotiation |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| anthropic/claude-opus-4-8:medium | 0 | 100 | 50 | 100 | 0 | 0 | 25 | 100 | 50 | 100 | 50 | 50 | 0 | 50 | 100 | 0 | 0 | 0 | 100 | 50 | 100 | 50 | 25 | 100 | 100 | 50 | 50 | 100 | 100 | 0 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 100 | 25 | 75 | 50 | 75 | 50 | 100 | 100 | 100 | 100 | 50 | 50 | 50 | 50 | 75 | 50 | 100 |
| anthropic/claude-sonnet-5:medium | 0 | 50 | 50 | 50 | 50 | 0 | 25 | 100 | 50 | 100 | 50 | 0 | 50 | 0 | 100 | 0 | 0 | 50 | 50 | 50 | 50 | 50 | 0 | 50 | 100 | 100 | 0 | 100 | 100 | 50 | 50 | 50 | 50 | 50 | 50 | 0 | 50 | 100 | 25 | 75 | 50 | 75 | 50 | 100 | 100 | 100 | 100 | 50 | 0 | 50 | 50 | 75 | 50 | 75 |
| moonshotai/kimi-k2.7-code | 0 | 50 | 50 | 50 | 0 | 0 | 50 | 100 | 50 | 100 | 50 | 0 | 50 | 0 | 100 | 0 | 0 | 50 | 0 | 50 | 50 | 50 | 0 | 0 | 0 | 50 | 50 | 100 | 100 | 0 | 50 | 0 | 0 | 50 | 50 | 25 | 50 | 50 | 50 | 50 | 50 | 75 | 100 | 100 | 100 | 100 | 100 | 50 | 0 | 0 | 50 | 50 | 50 | 50 |
| openai/gpt-5.6-luna:medium | 0 | 100 | 25 | 100 | 0 | 0 | 50 | 100 | 50 | 100 | 50 | 25 | 50 | 50 | 100 | 0 | 0 | 50 | 50 | 63 | 50 | 50 | 0 | 100 | 100 | 50 | 0 | 100 | 100 | 0 | 50 | 25 | 50 | 50 | 50 | 50 | 50 | 100 | 50 | 75 | 25 | 75 | 100 | 100 | 100 | 100 | 100 | 50 | 50 | 50 | 50 | 75 | 50 | 75 |
| openai/gpt-5.6-sol:medium | 50 | 100 | 50 | 50 | 100 | 0 | 50 | 100 | 50 | 100 | 50 | 50 | 50 | 100 | 100 | 75 | 0 | 50 | 100 | 75 | 50 | 50 | 25 | 100 | 100 | 100 | 0 | 100 | 100 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 100 | 50 | 75 | 25 | 75 | 100 | 50 | 50 | 100 | 100 | 50 | 50 | 50 | 50 | 75 | 50 | 75 |
| openai/gpt-5.6-terra:medium | 0 | 100 | 50 | 50 | 50 | 0 | 50 | 100 | 50 | 100 | 50 | 50 | 50 | 50 | 100 | 0 | 0 | 50 | 100 | 50 | 0 | 50 | 0 | 100 | 50 | 50 | 0 | 100 | 100 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 100 | 25 | 75 | 25 | 75 | 100 | 100 | 100 | 100 | 50 | 0 | 50 | 50 | 0 | 75 | 50 | 50 |