标签：#benchmark

Five stories moved the AI-coding world today. None are about a single model winning forever — they are about the ground shifting under who runs the agents and who pays for them. Musk puts Grok 4.5 to work at Tesla and SpaceX Tesla and SpaceX have been told to trial Grok 4.5 . The signal is not the benchmark — it is that a frontier model is being pointed at real engineering and ops inside hardware companies. When a model moves from a chatbot to a mandate inside a manufacturing and launch pipeline, the feedback loop gets brutally honest fast. Perplexity's orchestrator beats Opus on a benchmark Perplexity added Grok 4.5 to its orchestrator and reports beating Opus on the WANDR benchmark. Orchestrators are the quiet winners of this cycle: instead of one model doing everything, a router picks per-subtask. A smaller-or-cheaper mix outperforming a single flagship on a targeted benchmark is the trend to watch — it is how teams cut cost without giving up quality on the hard parts. Meta launches Muse Spark 1.1 at 25% of competitor pricing Meta shipped Muse Spark 1.1 through an API priced at roughly a quarter of what competitors charge. Price is a feature. At 25% of the field, an API becomes the default fallback router for cost-sensitive agents even if it is not the best at everything. Expect orchestrators to slot it in for the boring 80%. ByteDance rolls out Seedream 5.0 Pro ByteDance pushed Seedream 5.0 Pro across multiple platforms. Image generation keeps consolidating into a few vendor-backed models with wide distribution — relevant to coding agents the moment they need to generate UI mockups or assets inline. Cursor builds an "Office Agent" to challenge Anthropic Cursor is building a Sand AI office agent aimed at Anthropic's turf. The coding-agent wars are expanding from "writes code" to "runs the surrounding workflow" — email, docs, tickets. That is the same expansion the open-source side is feeling: oh-my-pi's model hub and OpenClaw's session fleet are both bets that th

I run ezyZip , a browser-based archive tool, so "which format should I use?" is a question I field constantly. The honest answer is usually "it depends," which satisfies nobody. So I stopped hand-waving and measured it. We benchmarked 42 archive and compression formats, spanning four decades, from 1984's Unix compress through today's Zstandard, Brotli, and context-mixing paq8px. Everything ran against the same realistic 55 MB corpus, every archive was round-trip verified byte for byte, and the whole thing reproduces from a single command. Here's what came out of it, and what I'd actually reach for. The setup Most compression benchmarks measure raw codecs on standardized corpora like Silesia. That's the right call for algorithm research and the wrong call for answering "what should I zip my folder with?" I wanted end-user formats, real CLI tools, container overhead and all, on data that looks like an actual folder. So the corpus is deliberately mixed: about 11 MB of text, 15 MB of office documents, 16 MB of images, and 13 MB of video, all public domain so it can be committed and redistributed. That mix matters. Office documents ( .docx , .xlsx , .pptx ) are themselves ZIP containers, so they stress how a tool handles already-compressed data. The JPEG and H.264 media is near-incompressible and sets an honest lower bound. The plain text and uncompressed images are where formats actually separate. Two rules kept it fair and practical: Only two levels per tool: its default, and its one "maximum compression" dial. No method tuning, no dictionary sizes, no thread-count games. That's what a normal person can reach. Everything is round-trip verified. Each archive gets extracted, and every file is hashed with SHA-256 against the original manifest. Exit codes are not trusted. That last rule earned its keep immediately. The verification gotcha On the image category, a 1985-era ARC build produced an archive that its own extractor happily unpacked, while printing a CRC warning an

Benchmark-backed Ollama has amassed 176,000 stars, and nearly 17,000 forks on Github by helping developers easily run AI on their PCs.

Android Bench is evolving, and developers can help guide that process.

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

placeholder

Video AI systems consistently fail to track what happens when the camera looks away: when a scene pans away from an object in motion and returns, current models re-render the object in its original position rather than showing the logical result of off-screen change. Scaling to more parameters makes this failure worse, not better, according to WRBench , a new benchmark that tests what researchers call "world model reliability." The benchmark presents AI video systems with scenes where something happens off-screen — the camera pans away while an object is in motion, or while a light changes, or while an open door should stay open — then pans back to see what the system believes should have happened. A system that genuinely models the world would track what occurred during the off-screen interval. Current systems mostly don't. Key facts What: A new benchmark tests whether video AI systems can track what happens to parts of a scene the camera isn't currently showing. Across 23 models, the answer is mostly no — and making the models larger made the problem worse, not better. When: 2026-06-19 Primary source: read the source (arXiv 2606.20545) The benchmark covers twenty-three different video generation models and nearly ten thousand video clips across six categories of off-screen change, each designed to test a different aspect of world continuity: objects in motion, light sources changing, object states such as open or closed doors, and several others. This gives a comprehensive picture rather than a single narrow test. The most striking finding is the scaling result. The researchers tested one of the more capable video generation systems at two different sizes: a smaller version and one with more than ten times as many parameters. More parameters didn't help. Scaling made the off-screen tracking problem measurably worse. The larger model produced more realistic-looking frames, but it was less accurate about what should have happened to the parts of the scene it wasn't

A large-scale audit of AI-as-judge evaluation — covering over half a million individual judgments — finds that AI judges are consistently reliable but not valid, meaning they give the same answer repeatedly without that answer being correct. Published work and popular benchmarks like Chatbot Arena have treated consistency as proof of trustworthiness, and the audit shows that assumption is unfounded. Key facts What: Using one AI to grade another is now common — but the biggest audit yet shows these graders are consistent without being correct. A judge that always picks "answer A" scores perfectly on consistency. When: 2026-06-19 Primary source: read the source (arXiv 2606.19544) The distinction matters: a judge is reliable if it's consistent (same question, same answer), and valid if those answers are actually correct. The audit's central finding is that AI judges are reliable without being valid, and the field has been treating the first as evidence of the second. Because consistency is easy to measure and looks reassuring, it has stood in for actual trustworthiness across a lot of published work. A new audit makes the problem stark: a judge that ignores both answers and always picks the one labeled "A" would be perfectly consistent — flawless reliability, identical verdict every time — and completely worthless, because it never read anything. Consistency is trivially easy to fake and says almost nothing about whether the judging is sound. Yet "the judge agrees with itself" has done significant reassurance work in papers and benchmarks, and the always-pick-A example shows exactly how empty that reassurance is. When the researchers corrected for the agreement you'd get by chance — as any fair test should — confident-looking scores deflated noticeably. Gaps between models that seemed meaningful shrank or blurred. Accepted folk wisdom also took a hit: the long-standing worry that AI judges are suckers for longer, wordier answers turned out to be far weaker than assumed

Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency, while maintaining flexibility to choose among more than 20 models. The post Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks appeared first on The GitHub Blog .

Agent-testing startup Patronus AI, founded by former Meta AI researchers, is experiencing nearly insatiable demand, its investor says.

I replaced aaPanel/OpenLiteSpeed with Caddy and shell scripts and turned the process into a benchmark. Two phases (architecture then code), one external code review. The winning model? Not the one you'd expect.

The general shape of the problem is that every public LLM benchmark is on a saturation clock that runs from the moment of its publication to the moment a model's training corpus has eaten it. The clock has been running, on the visible benchmarks of the last five years, for somewhere between twelve and thirty months before each one is no longer useful for differentiating frontier models. The benchmarks are not failing. They are doing exactly what they were designed to do, in the order they were designed to do it, and the field has been running through them faster than the people designing them anticipated. I want to put numbers on the saturation pattern, walk through what the contamination evidence actually says, and then sit with the question of what an honest benchmark would have to look like in 2026 — because the "private held-out eval" answer that the labs are converging on has economics that are worth examining carefully before any of us salute it as the solution. The saturation timeline, with numbers HumanEval (Chen et al., OpenAI, July 2021). 164 hand-written Python problems. The benchmark was published with Codex at 28.8% pass@1; the underlying GPT-3 base model scored 0%. GPT-4 (March 2023) hit 67% in the original Technical Report. By late 2024, OpenAI's o1-preview and o1-mini both reached 96.3% pass@1 ; Claude 3.5 Sonnet sat at 93.7%. The benchmark is saturated in the operational sense — the relative spread across the top ten models is around 10 percentage points, which is too small a gap to differentiate them on, and most of the new models arrive within a percentage point or two of the ceiling. The successor variants (HumanEval+ from EvalPlus, with augmented test cases) are the field's response. Lifespan from publication to operational saturation: about 36 months. MMLU (Hendrycks et al., September 2020). 57 subjects, ~14,000 multiple-choice questions, taken from publicly-available test prep and academic sources. The problem with MMLU is not that it's satura

Every LLM wire format claims token savings. Nobody proves whether AI models can actually comprehend the format at scale, or produce valid output in it. We ran 23 comprehension evals across 10 models and 3 providers. We ran generation evals across 11 models. Deterministic ground truth. No LLM judge. Reproducible from one command. JSON breaks at 500 records. GPT-5.5 returns empty strings. It can't even attempt an answer. Opus miscounts 500 as 356 and then spends 143 lines manually enumerating symbols to verify its own wrong answer. The format designed for "human readability" is incomprehensible to the systems actually reading it. TOON can't produce valid output. Claude Opus, the most capable model on the planet, scores 0/5 on TOON generation. GPT-5.4: 0/5. GPT-5.4-mini: 0/5. Gemini 3.1 Flash Lite: 0/5. The error is always the same: toon: cannot assign string to int . The model writes "target" in the distance column. TOON expects 0 . Every model fails the same way because the format's design forces an unnatural encoding step that models cannot perform unprompted. GCF wins both dimensions on every model tested. 100% comprehension on Claude Sonnet, Gemini 2.5 Pro, Gemini 3.1 Pro, and Gemini 3.5 Flash. 5/5 valid generation on every frontier model. Zero prior training. The format didn't exist until we built it and every model speaks it natively. Comprehension: 500 Symbols, 13 Questions, Zero Instructions A 500-symbol, 200-edge code graph. Encoded in GCF, TOON, and JSON. 13 structured extraction questions. The model gets the payload and a question. No format instructions. No system prompt. No hints. 23 runs. 22 wins. 0 losses. Model Runs GCF avg TOON avg JSON avg GCF margin Claude Opus 4.6 2 96.2% 84.6% 73.1% +11.6 vs TOON Claude Sonnet 4.6 2 100% 73.1% 53.8% +26.9 vs TOON Claude Haiku 4.5 2 96.2% 69.2% 57.7% +27.0 vs TOON GPT-5.5 5 84.1% 67.7% 45.8% +16.4 vs TOON GPT-5.4 4 76.4% 56.0% 44.1% +20.4 vs TOON GPT-5.4-mini 2 71.8% 64.1% 54.2% +7.7 vs TOON Gemini 2.5 Flash 3 80.6

A blind test across 240 images and 10 professional designers just dropped. Ideogram 4.0 against Gemini 3.1, Grok Imagine, and FLUX.2 Max. The results are clean. Ideogram won typography in nearly half of every blind matchup. 47.9 percent. Next closest was Gemini at 30 percent. FLUX.2 and Grok sat around 15 percent each. On the question that actually matters to designers -- would I ship this -- Ideogram scored 3.55 out of 5. Gemini got 2.84. Nobody else cleared 3. That is a real lead in text rendering. The model was trained exclusively on structured JSON caption datasets, which means it understands composition and layout differently than models trained on alt-text scraped from the web. The JSON prompting is genuinely useful for automated pipelines. You can specify bounding boxes, color palettes, object positions. It is not just better at text. It is more controllable. I tested it. It works. The text in images is readable. That has been the white whale of AI image generation for two years and Ideogram 4.0 mostly solves it. But as an overall image model, it is just good. Competitive, not dominant. On busy, highly detailed scenes with specific counts and attributes, Ideogram scored 3.42. Gemini scored 3.37. That is a statistical tie. FLUX.2 scored 3.01 and Grok 2.82, which are worse, but the gap between the top two is noise. For general image quality, you are splitting hairs between Ideogram and Gemini. For photorealism, FLUX and Reve still lead. For artistic generation, Midjourney is Midjourney. The prompting behavior is interesting. Lean prompts won across the board. Long, over-specified prompts lost. The model was trained on structured data, so it wants structure, not paragraphs. "A poster for a coffee shop. The text says Morning Blend in serif. Warm tones, natural light." That works. Adding stylistic directives and adjectives and "make it pop" language degrades the output. Where to actually use this thing: fal.ai has it at three cents per megapixel in Turbo mode. Tha

The legendary abandons its more than 20 year tradition of keeping its funds to about $425 million.

Overview PDF text extraction is a common pre-processing step in data pipelines — ingesting research papers, legal documents, or reports before embedding or indexing. Both pypdf and PdfPig are pure managed-code parsers: no native binaries, no OCR, no system PDF renderer. They implement the same PDF specification operations in their respective languages. This makes the benchmark unusually clean: the performance difference is entirely due to language execution speed, not library architecture differences. Benchmark Setup 200 recent arXiv PDFs (mixed technical papers, 5–40 pages each). Tested on subsets of 10, 50, 100, and 200 files. Both libraries extract all text from all pages; output is validated for page-count agreement and character-count agreement within 15% (pypdf and PdfPig decode whitespace and encoding tables slightly differently). Results PDFs Pages Python (pypdf) .NET (PdfPig) Speedup 10 ~120 ~0.9 s ~230 ms 3.9× 50 ~600 ~4.2 s ~810 ms 5.2× 100 ~1,200 ~8.5 s ~1.4 s 6.1× 200 ~2,400 ~17 s ~2.7 s 6.2× The speedup grows slightly with corpus size, suggesting pypdf has a per-document startup cost that compounds as PdfPig's JIT gets warmer. Why PdfPig Is Faster PDF parsing is byte-heavy: every page is a stream of PostScript-like operators (move, show text, set font, etc.). Each operator must be lexed, looked up in a dispatch table, and executed against a graphics state machine. In Python, each operator dispatch is a Python method call — the CPython bytecode interpreter has overhead per call regardless of what the method does. In .NET, the JIT compiles the dispatch loop to native code the first time it runs; subsequent pages pay only the cost of the actual work. Additionally, PdfPig's content-stream parser operates on ReadOnlySpan<byte> — zero-copy slicing through the raw page bytes with no intermediate string allocations. pypdf builds Python string objects for each token. Key Code // PdfPig — zero-copy span-based page extraction public Result Extract ( string path )

Overview PageRank is the canonical graph algorithm. NetworkX implements it in pure Python — its dict-of-dict adjacency representation means every power-iteration step dispatches millions of Python attribute lookups. When the graph has 1.8 million nodes and 28.5 million edges (Wikipedia category hyperlinks), those lookups dominate the runtime. The .NET replacement uses a CSR (Compressed Sparse Row) matrix — two flat int[] arrays for the graph structure — and TensorPrimitives for the SIMD-accelerated normalization step inside each iteration. Benchmark Setup Five SNAP datasets of increasing size: Dataset Nodes Edges wiki-Vote 7,115 103,689 soc-Epinions1 75,879 508,837 web-Stanford 281,903 2,312,497 web-Google 875,713 5,105,039 wiki-topcats 1,791,489 28,511,807 Algorithm: power-iteration PageRank, damping=0.85, tol=1e-6. Both implementations converge to identical top-10 node rankings. Results Dataset Python (NetworkX) .NET (CSR) Speedup wiki-Vote (103k edges) ~0.8 s ~100 ms ~8× soc-Epinions1 (508k edges) ~8 s ~600 ms ~13× web-Stanford (2.3M edges) ~120 s ~5 s ~24× web-Google (5.1M edges) ~5.5 min ~12 s ~28× wiki-topcats (28.5M edges) ~47 min ~60 s ~47× The speedup grows with graph size because NetworkX's Python dispatch cost scales with edge count, while the CSR inner loop is a tight JIT-compiled SIMD pass. Why CSR Beats NetworkX NetworkX represents each node's neighbors as a Python dict. Iterating the adjacency in one power-iteration step means: Calling G.neighbors(node) — a Python method call Iterating a dict — unboxing int keys, chasing heap pointers Accumulating a float into another dict value — another boxing step That happens for every edge, every iteration, roughly 50–80 times to convergence. CSR collapses the graph to two arrays: rowPtr[n+1] (where each node's neighbors start) and colIdx[edges] (the neighbor list). Iterating neighbors of node v is a tight C loop from rowPtr[v] to rowPtr[v+1] . No Python objects, no dict hashing, no pointer chasing. Key Code // P