LLM Wire Format Benchmark: Which Format Can AI Actually Read and Write?
Every LLM wire format claims token savings. Nobody proves whether AI models can actually comprehend the format at scale, or produce valid output in it. We ran 23 comprehension evals across 10 models and 3 providers. We ran generation evals across 11 models. Deterministic ground truth. No LLM judge. Reproducible from one command. JSON breaks at 500 records. GPT-5.5 returns empty strings. It can't even attempt an answer. Opus miscounts 500 as 356 and then spends 143 lines manually enumerating symbols to verify its own wrong answer. The format designed for "human readability" is incomprehensible to the systems actually reading it. TOON can't produce valid output. Claude Opus, the most capable model on the planet, scores 0/5 on TOON generation. GPT-5.4: 0/5. GPT-5.4-mini: 0/5. Gemini 3.1 Flash Lite: 0/5. The error is always the same: toon: cannot assign string to int . The model writes "target" in the distance column. TOON expects 0 . Every model fails the same way because the format's design forces an unnatural encoding step that models cannot perform unprompted. GCF wins both dimensions on every model tested. 100% comprehension on Claude Sonnet, Gemini 2.5 Pro, Gemini 3.1 Pro, and Gemini 3.5 Flash. 5/5 valid generation on every frontier model. Zero prior training. The format didn't exist until we built it and every model speaks it natively. Comprehension: 500 Symbols, 13 Questions, Zero Instructions A 500-symbol, 200-edge code graph. Encoded in GCF, TOON, and JSON. 13 structured extraction questions. The model gets the payload and a question. No format instructions. No system prompt. No hints. 23 runs. 22 wins. 0 losses. Model Runs GCF avg TOON avg JSON avg GCF margin Claude Opus 4.6 2 96.2% 84.6% 73.1% +11.6 vs TOON Claude Sonnet 4.6 2 100% 73.1% 53.8% +26.9 vs TOON Claude Haiku 4.5 2 96.2% 69.2% 57.7% +27.0 vs TOON GPT-5.5 5 84.1% 67.7% 45.8% +16.4 vs TOON GPT-5.4 4 76.4% 56.0% 44.1% +20.4 vs TOON GPT-5.4-mini 2 71.8% 64.1% 54.2% +7.7 vs TOON Gemini 2.5 Flash 3 80.6