Heatmap

Model:effort × scenario mean score (0–100). A trial scores 0 if any MUST checkpoint fails (task success is a MUST), otherwise the fraction of SHOULD checkpoints passed; MAY checkpoints never affect the score. Defaults to the headline none (no-docs) condition — priors only; select full to see the docs-assisted scores. With a single version + docs selected, cells show the mean over repeat trials and link to trial detail.

Spec version Docs

Model	client/bad-version-response	client/composite	client/discover-instructions	client/drop-stream-mid-call	client/elicit-form	client/ema-id-jag-flow	client/get-stream	client/header-mirroring	client/initialize-handshake	client/mrtr-basic	client/mrtr-elicit-url	client/mrtr-multi-round	client/mrtr-no-state	client/no-batching	client/oauth-cimd	client/oauth-dcr-application-type	client/oauth-iss-mismatch	client/oauth-pkce	client/oauth-refresh-token	client/param-headers	client/pkce-unsupported-as	client/prompts-get	client/protocol-version-header	client/resources-read	client/session-echo	client/session-recovery	client/sse-response	client/stalled-tool	client/stateless-composite	client/stateless-prompts	client/stateless-resources	client/stateless-sse-response	client/stateless-tool-call	client/task-cancel	client/task-polling	client/tool-call-basic
anthropic/claude-opus-4-8:medium	0	100	50	100	0	25	100	50	100	50	50	0	50	100	0	0	100	50	100	50	25	100	100	50	50	100	100	0	50	50	50	50	50	50	50	100
moonshotai/kimi-k2.7-code	0	50	50	50	0	50	100	50	100	50	0	50	0	100	0	50	0	50	50	50	0	0	0	50	50	100	100	0	50	0	0	50	50	25	50	50
openai/gpt-5.6-luna:medium	0	100	25	100	0	50	100	50	100	50	25	50	50	100	0	50	50	63	50	50	0	100	100	50	0	100	100	0	50	25	50	50	50	50	50	100
openai/gpt-5.6-sol:medium	50	100	50	50	100	50	100	50	100	50	50	50	100	100	75	50	100	75	50	50	25	100	100	100	0	100	100	50	50	50	50	50	50	50	50	100
openai/gpt-5.6-terra:medium	0	100	50	50	50	50	100	50	100	50	50	50	50	100	0	50	100	50	0	50	0	100	50	50	0	100	100	50	50	50	50	50	50	50	50	100