标签：#benchmarks

Five stories moved the AI-coding world today. None are about a single model winning forever — they are about the ground shifting under who runs the agents and who pays for them. Musk puts Grok 4.5 to work at Tesla and SpaceX Tesla and SpaceX have been told to trial Grok 4.5 . The signal is not the benchmark — it is that a frontier model is being pointed at real engineering and ops inside hardware companies. When a model moves from a chatbot to a mandate inside a manufacturing and launch pipeline, the feedback loop gets brutally honest fast. Perplexity's orchestrator beats Opus on a benchmark Perplexity added Grok 4.5 to its orchestrator and reports beating Opus on the WANDR benchmark. Orchestrators are the quiet winners of this cycle: instead of one model doing everything, a router picks per-subtask. A smaller-or-cheaper mix outperforming a single flagship on a targeted benchmark is the trend to watch — it is how teams cut cost without giving up quality on the hard parts. Meta launches Muse Spark 1.1 at 25% of competitor pricing Meta shipped Muse Spark 1.1 through an API priced at roughly a quarter of what competitors charge. Price is a feature. At 25% of the field, an API becomes the default fallback router for cost-sensitive agents even if it is not the best at everything. Expect orchestrators to slot it in for the boring 80%. ByteDance rolls out Seedream 5.0 Pro ByteDance pushed Seedream 5.0 Pro across multiple platforms. Image generation keeps consolidating into a few vendor-backed models with wide distribution — relevant to coding agents the moment they need to generate UI mockups or assets inline. Cursor builds an "Office Agent" to challenge Anthropic Cursor is building a Sand AI office agent aimed at Anthropic's turf. The coding-agent wars are expanding from "writes code" to "runs the surrounding workflow" — email, docs, tickets. That is the same expansion the open-source side is feeling: oh-my-pi's model hub and OpenClaw's session fleet are both bets that th

Android Bench is evolving, and developers can help guide that process.

Video AI systems consistently fail to track what happens when the camera looks away: when a scene pans away from an object in motion and returns, current models re-render the object in its original position rather than showing the logical result of off-screen change. Scaling to more parameters makes this failure worse, not better, according to WRBench , a new benchmark that tests what researchers call "world model reliability." The benchmark presents AI video systems with scenes where something happens off-screen — the camera pans away while an object is in motion, or while a light changes, or while an open door should stay open — then pans back to see what the system believes should have happened. A system that genuinely models the world would track what occurred during the off-screen interval. Current systems mostly don't. Key facts What: A new benchmark tests whether video AI systems can track what happens to parts of a scene the camera isn't currently showing. Across 23 models, the answer is mostly no — and making the models larger made the problem worse, not better. When: 2026-06-19 Primary source: read the source (arXiv 2606.20545) The benchmark covers twenty-three different video generation models and nearly ten thousand video clips across six categories of off-screen change, each designed to test a different aspect of world continuity: objects in motion, light sources changing, object states such as open or closed doors, and several others. This gives a comprehensive picture rather than a single narrow test. The most striking finding is the scaling result. The researchers tested one of the more capable video generation systems at two different sizes: a smaller version and one with more than ten times as many parameters. More parameters didn't help. Scaling made the off-screen tracking problem measurably worse. The larger model produced more realistic-looking frames, but it was less accurate about what should have happened to the parts of the scene it wasn't

A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but not valid, meaning they give the same answer repeatedly without that answer being correct. Published work and popular benchmarks like Chatbot Arena have treated consistency as proof of trustworthiness, and the audit shows that assumption is unfounded. Key facts What: Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency. When: 2026-06-19 Primary source: read the source (arXiv 2606.19544) The distinction matters: a judge is reliable if it's consistent (same question, same answer), and valid if those answers are actually correct. The audit's central finding is that AI judges are reliable without being valid, and the field has been treating the first as evidence of the second. Because consistency is easy to measure and looks reassuring, it has stood in for actual trustworthiness across a lot of published work. A new audit makes the problem stark: a judge that ignores both answers and always picks the one labeled "A" would be perfectly consistent — flawless reliability, identical verdict every time — and completely worthless, because it never read anything. Consistency is trivially easy to fake and says almost nothing about whether the judging is sound. Yet "the judge agrees with itself" has done significant reassurance work in papers and benchmarks, and the always-pick-A example shows exactly how empty that reassurance is. When the researchers corrected for the agreement you'd get by chance — as any fair test should — confident-looking scores deflated noticeably. Gaps between models that seemed meaningful shrank or blurred. Accepted folk wisdom also took a hit: the long-standing worry that AI judges are suckers for longer, wordier answers turned out to be far weaker than assumed

Agent-testing startup Patronus AI, founded by former Meta AI researchers, is experiencing nearly insatiable demand, its investor says.

The general shape of the problem is that every public LLM benchmark is on a saturation clock that runs from the moment of its publication to the moment a model's training corpus has eaten it. The clock has been running, on the visible benchmarks of the last five years, for somewhere between twelve and thirty months before each one is no longer useful for differentiating frontier models. The benchmarks are not failing. They are doing exactly what they were designed to do, in the order they were designed to do it, and the field has been running through them faster than the people designing them anticipated. I want to put numbers on the saturation pattern, walk through what the contamination evidence actually says, and then sit with the question of what an honest benchmark would have to look like in 2026 — because the "private held-out eval" answer that the labs are converging on has economics that are worth examining carefully before any of us salute it as the solution. The saturation timeline, with numbers HumanEval (Chen et al., OpenAI, July 2021). 164 hand-written Python problems. The benchmark was published with Codex at 28.8% pass@1; the underlying GPT-3 base model scored 0%. GPT-4 (March 2023) hit 67% in the original Technical Report. By late 2024, OpenAI's o1-preview and o1-mini both reached 96.3% pass@1 ; Claude 3.5 Sonnet sat at 93.7%. The benchmark is saturated in the operational sense — the relative spread across the top ten models is around 10 percentage points, which is too small a gap to differentiate them on, and most of the new models arrive within a percentage point or two of the ceiling. The successor variants (HumanEval+ from EvalPlus, with augmented test cases) are the field's response. Lifespan from publication to operational saturation: about 36 months. MMLU (Hendrycks et al., September 2020). 57 subjects, ~14,000 multiple-choice questions, taken from publicly-available test prep and academic sources. The problem with MMLU is not that it's satura

Overview PDF text extraction is a common pre-processing step in data pipelines — ingesting research papers, legal documents, or reports before embedding or indexing. Both pypdf and PdfPig are pure managed-code parsers: no native binaries, no OCR, no system PDF renderer. They implement the same PDF specification operations in their respective languages. This makes the benchmark unusually clean: the performance difference is entirely due to language execution speed, not library architecture differences. Benchmark Setup 200 recent arXiv PDFs (mixed technical papers, 5–40 pages each). Tested on subsets of 10, 50, 100, and 200 files. Both libraries extract all text from all pages; output is validated for page-count agreement and character-count agreement within 15% (pypdf and PdfPig decode whitespace and encoding tables slightly differently). Results PDFs Pages Python (pypdf) .NET (PdfPig) Speedup 10 ~120 ~0.9 s ~230 ms 3.9× 50 ~600 ~4.2 s ~810 ms 5.2× 100 ~1,200 ~8.5 s ~1.4 s 6.1× 200 ~2,400 ~17 s ~2.7 s 6.2× The speedup grows slightly with corpus size, suggesting pypdf has a per-document startup cost that compounds as PdfPig's JIT gets warmer. Why PdfPig Is Faster PDF parsing is byte-heavy: every page is a stream of PostScript-like operators (move, show text, set font, etc.). Each operator must be lexed, looked up in a dispatch table, and executed against a graphics state machine. In Python, each operator dispatch is a Python method call — the CPython bytecode interpreter has overhead per call regardless of what the method does. In .NET, the JIT compiles the dispatch loop to native code the first time it runs; subsequent pages pay only the cost of the actual work. Additionally, PdfPig's content-stream parser operates on ReadOnlySpan<byte> — zero-copy slicing through the raw page bytes with no intermediate string allocations. pypdf builds Python string objects for each token. Key Code // PdfPig — zero-copy span-based page extraction public Result Extract ( string path )

Overview PageRank is the canonical graph algorithm. NetworkX implements it in pure Python — its dict-of-dict adjacency representation means every power-iteration step dispatches millions of Python attribute lookups. When the graph has 1.8 million nodes and 28.5 million edges (Wikipedia category hyperlinks), those lookups dominate the runtime. The .NET replacement uses a CSR (Compressed Sparse Row) matrix — two flat int[] arrays for the graph structure — and TensorPrimitives for the SIMD-accelerated normalization step inside each iteration. Benchmark Setup Five SNAP datasets of increasing size: Dataset Nodes Edges wiki-Vote 7,115 103,689 soc-Epinions1 75,879 508,837 web-Stanford 281,903 2,312,497 web-Google 875,713 5,105,039 wiki-topcats 1,791,489 28,511,807 Algorithm: power-iteration PageRank, damping=0.85, tol=1e-6. Both implementations converge to identical top-10 node rankings. Results Dataset Python (NetworkX) .NET (CSR) Speedup wiki-Vote (103k edges) ~0.8 s ~100 ms ~8× soc-Epinions1 (508k edges) ~8 s ~600 ms ~13× web-Stanford (2.3M edges) ~120 s ~5 s ~24× web-Google (5.1M edges) ~5.5 min ~12 s ~28× wiki-topcats (28.5M edges) ~47 min ~60 s ~47× The speedup grows with graph size because NetworkX's Python dispatch cost scales with edge count, while the CSR inner loop is a tight JIT-compiled SIMD pass. Why CSR Beats NetworkX NetworkX represents each node's neighbors as a Python dict. Iterating the adjacency in one power-iteration step means: Calling G.neighbors(node) — a Python method call Iterating a dict — unboxing int keys, chasing heap pointers Accumulating a float into another dict value — another boxing step That happens for every edge, every iteration, roughly 50–80 times to convergence. CSR collapses the graph to two arrays: rowPtr[n+1] (where each node's neighbors start) and colIdx[edges] (the neighbor list). Iterating neighbors of node v is a tight C loop from rowPtr[v] to rowPtr[v+1] . No Python objects, no dict hashing, no pointer chasing. Key Code // P