标签：#Benchmark

We held off on posting a benchmark for a long time. Not because we didn't have runs - because most memory benchmarks you read are a number with no way to check it. A blog says "X%", and you have no idea what reader answered the questions, what judge scored them, how much context the retriever was allowed to feed, or whether an LLM quietly did the hard part inside the "memory" layer. So the number tells you almost nothing about the memory system. Here is one we're comfortable standing behind, because you can run it yourself. The result On LongMemEval-S , the full 500-question set, Engrava 0.6.0 scored 81.6% micro in August 2026 - 81.76% averaged across the six question categories. The run uses the canonical LongMemEval scorer (pinned to a known upstream commit), the standard gpt-4o-2024-08-06 reader and judge over the OpenAI API, and a top_k of 20 retrieved turns. Nothing about the reader, the prompt, or the scorer is ours; the only thing we swapped in is the memory. It is compared against the previous release: 0.5.0, run in July 2026, scored 82.4% micro / 82.58% macro on the same 500 questions, same reader, same judge, same scorer, same top_k . Both rows are on the leaderboard, both verified , and both ship their reproduction artifacts. We are leading with 0.6.0 because that is the version this post is about; the older row stays because removing it when the number goes down is exactly the move that makes benchmark pages worthless. 0.5.0 (2026-07-10) 0.6.0 (2026-08-11) micro 82.4% 81.6% macro 82.58% 81.76% n 500 500 Both figures are dated on purpose. This post is a record of two specific runs, not a running scoreboard; the current table, whatever version is newest when you read this, lives on the Engrava benchmarks page . The run also has no LLM in the memory pipeline. Ingestion and retrieval are deterministic - hybrid search over a typed graph, no model doing extraction, summarization, or re-ranking behind the curtain. In the benchmark's own terms this is a Group A

AWS has released aws-bench, an open-source benchmark for evaluating AI agents on real AWS tasks such as misconfigurations and infrastructure provisioning. Unlike traditional benchmarks, it uses real resources in disposable AWS accounts, scoring agent performance through automated verifiers. By Gianmarco Nalin

Jordan Nanos discusses how semiconductor constraints, data center expansion, and networking bottlenecks impact AI software architecture. Drawing from SemiAnalysis research, he shares insights on benchmark performance, GPU scaling, and tokenomics from chip fab to model inference. By Jordan Nanos

Last week my feed filled with screenshots of MiniMax H3 benchmark results, and every post seemed to reach a different conclusion about whether the release mattered. I have been through enough launch-day hype cycles to know that a public leaderboard does not predict how a model will behave on my team's actual error logs. So I treated the H3 discussion as a trigger for a controlled experiment instead of as evidence that we should switch tools. This article walks through a lightweight, reproducible smoke test you can run on a free model tier before you commit to a new model. It focuses on code-generation and debugging tasks because those are the areas where a strong vendor benchmark often hides the biggest day-to-day failures. The goal is not to rank MiniMax H3 against every other option; the goal is to create a baseline you can rerun whenever a new model appears. We can run this workflow on MonkeyCode's free model access and free server option, which removes the cost of a quick initial evaluation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The idea is to use that free capacity for a time-boxed, reproducible test rather than for unstructured prompt tinkering. Why a public benchmark can mislead you A vendor benchmark is usually a point-in-time measurement with a specific harness, sampling strategy, and temperature setting. When a model scores high on a general coding benchmark, it tells you very little about the three failure modes that actually break your work: internal tool calls, long-context edits, and boundary handling in your language stack. I prefer to start with a fixed set of five tasks that I can run in about 30 minutes on any model endpoint. Each task returns a machine-readable result, so the output can be diffed across runs and across models without relying on my memory of how good a response felt. The smoke test harness The Python script below sends five prompts to a generic HTTP endpoint and records latency, output leng

Every model launch comes with a chart. Bars, usually, or a spider diagram, showing the new model edging past its rivals on a row of benchmarks with acronyms most people cannot expand. The bar is taller. The press writes it up as a leap. And within a week, users report that the new state-of-the-art model is, for their actual work, about the same as the last one or occasionally worse. The benchmark said one thing. Reality said another. This happens so reliably that it is worth understanding the mechanics of the gap. The test is public, which ruins the test The most fundamental problem is contamination. Many popular benchmarks are published, discussed, and sitting on the open web — which is exactly where models get their training data. When the questions and answers to your exam are in the study material, a high score measures memorisation as much as ability. Nobody needs to cheat deliberately; the leak is structural. A model can score brilliantly on a benchmark it has effectively already seen and then flounder on a genuinely novel version of the same task. A benchmark stops measuring intelligence the moment it becomes famous enough to end up in the training data. Fame is the thing that breaks it. The number becomes the marketing, and the marketing corrupts the number There is a commercial feedback loop that makes benchmark figures even less trustworthy than their technical limitations alone would suggest. A high score is not just an engineering result; it is a marketing asset worth an enormous amount in attention, funding and credibility. That raises the stakes on every fractional improvement, and where the stakes are high, the temptation to select, frame and present the numbers favourably is irresistible. Vendors choose which benchmarks to headline, which comparisons to draw, and which unflattering results to leave in an appendix or omit entirely. The chart on the launch slide is not a neutral readout; it is a curated argument. This is not necessarily fraud — it rare

Code: Megapixel99/capture-the-flag In April I ran five games of an AI capture-the-flag tournament between five small open-weight models (1.0B to 2.5B parameters). Each was given root on an identical Ubuntu container and told to steal /root/flag.txt from the others while defending its own. Qwen 3.5 at 2.3B captured 13 flags and lost none. The other four captured two flags between them. Nobody found any of the five bonus flags planted on each machine, even after hints were revealed at rounds 6, 9 and 12, so I wrote a preliminary report with two conclusions in it: Model size matters enormously for security reasoning. Multi-step exploitation is not feasible for models under 3B parameters. The tournament has since run 327 more games with much larger models, which makes the first conclusion checkable. It does not survive. (This is the second post about this project. The first is about a dedup pass that deleted the training weights , and the two findings are independent.) The later tournament does not order by size Five hosted models plus one local 3B fine-tune, over the 327 games with a scoreboard. Captures here are events the game engine credited, counted from each game's own event log: model flag captures bonus flags GPT-OSS 120B 401 38 custom bot (3B local fine-tune, 221 games) 404 2 GLM-5.1 315 111 Nemotron 3 Super 100 30 Gemini 3 Flash 74 8 RNJ-1 8B 2 0 Two things in that table contradict the report. Those 189 bonus flags break down by tier as 50, 25, 58, 23 and 33, and tier 4 is the encrypted vault: read /etc/app/database.yml , take the password out of it, decrypt /root/.vault/encrypted.flag with openssl . It was captured 23 times, so multi-step exploitation is not out of reach here. And a model with more parameters than every entrant in the April tournament, RNJ-1 8B, finished last by a factor of 37 against the second-worst model, while a 3B fine-tune running on my desk led on main flags. That fine-tune has never gone past tier 1; both of its bonus flags are the hi

The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? Tested on a 50K LOC Python codebase Mikhail Mikhail Mikhail Follow Aug 11 The Mechanical vs. The Semantic: What Happens When AI Memory is Wrong? # ai # agents # architecture # mcp 19 reactions 39 comments 6 min read The verify-on-read experiment (1-V) used a deterministic proxy agent to catch false claims in memory before surfacing them to the user. Proxy FA=0 by construction — that's a useful property, but it tells you nothing about what a real LLM would do with the same claims. A reviewer's note from Part 3 was blunt: "headline numbers were a property of the heuristic, not LLM behavior." So we ran it with live models. 50 facts, 2 arms, 14 models, ~3300 API calls, $0.14 total. Here's what we found. The setup Dataset: memory_contamination_facts_v4_rep.json , N=50 (R01–R50), sha256 fingerprint 820bbbf60a0fc930 . kind n what it tests real 25 TRUE claims — grep-validated against code absent-mutation 16 FALSE — component doesn't exist in the project present-trap 6 FALSE — file exists but claim is about wrong subject/value silent 3 FALSE — external systems the codebase doesn't mention Two arms per fact: memory_first — model sees only the claim text, no code context. Does it trust memory without evidence? code_first — model sees claim + support_patterns + section . Does it correctly evaluate the anchors? Model verdict: {"verdict": "true"|"false"|"unknown"} , JSON-only, max_tokens=100, temp=0, seed=42, --no-reasoning . Leak-guard: assert "truth" not in prompt on every fact, unit-tested. Metric we care about: false_accept rate (FA) — fraction of false claims the model returned "true" for. This is the contamination risk number. Results (V2 prompt, canonical) model FA mem FA code unknown mem unknown code $/100 calls qwen3.6-flash 0.00 0.00 0.58 0.38 $0.003 qwen3.7-flash 0.00 0.00 0.68 0.24 $0.0005 claude-sonnet-5 0.00 0.00 0.86 0.70 $0.049 deepseek-v4-pro 0.04 0.00 0.66 0.88 $0.018 glm-5.2 0.00 0.02 0.96 0

I Benchmarked Two Local LLMs on Real Dev Work — Qwopus 27B vs Muse Glimmer 30B Two open-weight models, one 20 GB GPU, two real development tasks, and a third model as the referee. Here is what actually happened when I made Qwopus 3.6 27B and Meta's Muse Glimmer 30B implement a bug fix and then a full feature in my own project. The setup Both models ran fully local on an AMD Radeon RX 7900 XT (20 GB VRAM) via a llama.cpp multi-model router (one OpenAI-compatible endpoint, GGUF models, load-mode=dio — more on why below). Each model was driven by the pi CLI in non-interactive mode with --thinking high . A third model — Codex, through a disciplined stdin wrapper — reviewed both outputs and gave the verdict. The fairness method was simple but strict: One task , described in a markdown spec, copied byte-identical into two isolated git clones of my project. Each model worked in its own clone, its own branch , never seeing the other's work. Objective verification by script: existing test suite + new tests + production build. Cross-review by Codex , examining both branches against the same criteria. The test project: Jeu de Cochons (a "Pass the Pigs" dice game, vanilla JS PWA on Vite + Vitest) — real code, real tests, no toy repo. Qwopus 3.6 27B Muse Glimmer 30B Source Community fine-tune of Qwen 3.6 Meta (distilled from Muse Spark) Size 27B 29.6B Quant IQ4_XS (~15 GB) UD-Q4_K_XL (~14.8 GB) Round 1 — fixing a regression (short task) The project had a broken PWA: a commit that added a /jeu-de-cochons/ base path for GitHub Pages had broken 3 service-worker tests (manifest, precache, offline navigation fallback). Task: fix the regression without touching the tests , keep the other 84 green. Qwopus Muse PWA tests (11) 11/11 ✅ 11/11 ✅ Full suite (87) 87/87 ✅ 87/87 ✅ Files touched 2 2 Diff size +4/−4 +4/−4 Wall time ~8.5 min ~21 min Leftover artifacts none one .bak file The remarkable result: both models produced a byte-identical diff. Same diagnosis (a lost capture group in the a

There's an argument running about MCP right now. You've probably seen it: a 400-point thread called "MCP is dead?" with real token numbers in it, four connected servers eating 21,077 tokens of context before anyone asks a question. The argument is about what MCP costs. Almost nobody in it has measured what agents actually do with the data a tool returns. I ended up measuring that, not because I planned to, but because a maintainer refused to let me guess. The question nobody wanted to answer with opinions I contribute to CNCF Jaeger's MCP server. Last April I proposed exposing service performance metrics (latencies, call rates, error rates) as an MCP tool, and hit an immediate design fork: what shape should the output be? Option one, summary rows: pre-aggregated stats per service, compact, cheap. Option two, per-bucket time series: the raw points, roughly 720 of them per service at default resolution, expensive but complete. I asked which the maintainer preferred. The answer, verbatim, from the issue thread: This type of decision should not be based on opinion, but on benchmarks with a real agent troubleshooting some issues and using this MCP tool to access metrics, where you could do A/B testing of different output formats. Fair. So I built the A/B. The setup Everything below is public in jaeger-mcp-bench , including the harness, the tasks, the scorer, and a research log of everything that went wrong. The fixture is Jaeger v2 with the spanmetrics connector, hotrod generating traffic, and Prometheus behind it, snapshotted so every run sees identical metric state. In front of the metrics API sits a thin bench server with exactly one switch: --format=summary|series . No new semantics, just the shape of what comes back. Six troubleshooting tasks, and this part matters: three were chosen because I predicted summary would win them (point questions: current latency, ranking, threshold checks) and three because I predicted series would win (temporal questions: spike detect

Recently, I've added a bunch of hype-monsters to my AI Werewolf : Kimi K3 Qwen 3.8 Max, Qwen 3.7 Plus, Qwen 3.7 Flash MiniMax M3 Plus the ones I've had for a while DeepSeek V4 Pro and Flash GLM-5.2 Sakana Fugu base and Ultra The last one is Japanese, not Chinese, but it was in the news a month or two ago and it belongs in this story. I sat down to write about how slow Chinese models are. Because they are all annoyingly slow even with a small context. I had the numbers, the thesis wrote itself, and then... I discovered something I didn't expect. Okay, let me go straight to the problem All Chinese official APIs are extremely slow. DeepSeek got better with their v4, the rest are just terrible. So slow that they are barely usable in a text game. No, seriously, take a look. Time to produce one four-sentence vote: Kimi K3 : 29 to 34 seconds MiniMax M3 : 25 to 30 seconds Qwen 3.8 Max : 25 to 27 seconds, and only because I capped its thinking. Uncapped it hit 100. DeepSeek V4 Pro : 14 to 22 seconds, the best of the group Same prompt, same afternoon: Claude 5 Opus answers in 5.9 seconds. I have a test which simulates the voting at the end of a game day. Some chat has happened, some players have already cast their votes, and now the model under test has to do the same. The prompt contains all of that: 36,000 characters, which works out to 8-13k tokens depending on the model's tokenizer. Not much, assuming those models have 1M contexts. More US/EU models: Model Time Input Output tokens Avg cost GPT-5.6 Luna 3.2-3.6s 12,124 190-231 $0.0020 Mistral Large 3 2.9-5.2s 8,175 73-134 $0.0042 Claude Fable 5 4.8-6.0s 13,027 183-209 $0.1399 Gemini 3.6 Flash 4.6-7.1s 7,979 297-532 $0.0152 GPT-5.6 Sol 6.7-7.3s 12,124 150-156 $0.0471 Claude 5 Opus 5.9-7.8s 13,027 220-387 $0.0726 Gemini 3.1 Pro 11.6-12.1s 7,979 821-940 $0.0263 Claude 5 Sonnet 6.8-14.8s 13,027 252-821 $0.0464 Grok 4.5 10.0-14.8s 8,190 428-607 $0.0149 Mistral Medium 3.5 2.9-35.0s 8,175 57-109 $0.0128 Worth staring at the input

Engineering posts often end with: The new design is correct, scalable, and fast. Fast compared with what? When we changed Podium so tied players rank by arrival time instead of player ID, we added: a Lua script; a per-leaderboard sequence; a public-ID mapping; a second sorted set for ascending order. That design is fairer. It is also impossible for it to be free. So we built two benchmark layers: direct Redis strategy benchmarks to isolate the data-model cost, and end-to-end HTTP benchmarks to show what users actually experience. We are publishing the results, including the regression, because performance claims are useful only when readers can inspect the workload and reproduce the measurement. TeneficGames / podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium High-performance, Redis-backed leaderboards for games and competitive applications. Podium provides ready-to-run HTTP and gRPC APIs for scores, ranks, seasons, and player-relative views. It is designed for backend teams operating large fleets of independent leaderboards without provisioning each leaderboard in advance. Fair, deterministic ordering when scores are equal. Single and bulk score updates, including multi-leaderboard fan-out. Standalone Redis and real Redis Cluster integration coverage. Deploy one multi-architecture OCI image with Docker, containerd, Kubernetes or another OCI-compatible runtime. Quickstart · Performance · API · Documentation · Helm chart · Docker Hub · GHCR Quickstart Start Redis 8.2 and the latest stable Podium image: docker network create podium docker run --detach --name podium-redis --network podium redis:8.2-alpine docker run --detach --rm --name podium \ --network podium \ --publish 8880:8880 \ --publish 8881:8881 \ --env PODIUM_REDIS_HOST=podium-redis \ --env PODIUM_REDIS_PORT=6379 \ trungdlp/podium:latest start Verify the service: curl http://localhost:8880/healthcheck WORKING Submit two equal scores: curl --request … View on Gi

You don't need to know anything about Go to read this. The game is just the fixed yardstick. The story is a hardware benchmark: the same program, the same problem, the same settings — only the machine changed, from a 2017 GPU cluster to a single 2026 graphics card. That makes it a rare clean measurement of one decade of progress. What "solving" means here There are two very different things a computer can do with a board game. It can play it well — that's what AlphaGo did. Or it can solve it: mathematically prove the outcome under perfect play from both sides, leaving no doubt. Solving is the hard one. You explore an enormous tree of "if I play here, they play there…" move sequences until you have an airtight proof. Each node in that tree is one position examined. The target here is a single 7x7 opening called JA . In 2023, a NeurIPS paper ( Game Solving with Online Fine-Tuning , Wu et al.) proved its verdict — the attacker cannot win — using a cluster of twelve GTX 1080Ti GPUs running 384 parallel workers. The solver is guided by a neural network that estimates how hard each branch is, and crucially that network is fine-tuned online — it keeps learning during the solve. I rebuilt that exact solver (same code, same problem, same initial model, same search settings) and ran it on one RTX 5090 . It reached the identical proof . Everything but the hardware was held fixed, so the two runs line up as a generation-vs-generation benchmark — and it doubled as a full shakedown of the new Blackwell workstation. The numbers 1x RTX 5090 (2026) 12x GTX 1080Ti (2017) ratio Worker slots 24 384 1/16 the parallelism Per-slot throughput 284 nodes/s 141 nodes/s 2.01x faster Search work to proof 1.01B nodes 1.73B nodes 0.59x (41% less work) Avg work per sub-job 4,189 nodes 6,136 nodes shallower proofs Live model updates 4,007 208 19.3x more Wall-clock time 41.4 h 8.9 h 4.64x slower Verdict loss (proven) loss identical The single card finished slower in wall-clock time (41 h vs 9 h) — b

OpenAI published an audit of SWE-Bench Pro on July 8, 2026 and estimated that roughly 30% of its tasks are broken. The reported issues make a familiar leaderboard assumption unsafe: every task in the denominator is a valid, equally interpretable trial. Primary source: OpenAI, “Separating signal from noise in coding evaluations” . The operational response should not be “ignore all benchmarks.” It should be: version task validity, preserve disputed cases, and publish how conclusions change across plausible denominators. Model task state separately from model result task validity: unreviewed | valid | broken | disputed model result: pass | fail | infrastructure_error | missing Never convert infrastructure_error to model failure without reporting that policy. Never delete broken tasks while retaining an old score label. A row needs provenance: { "task_id" : "repo-issue-17" , "dataset_revision" : "sha256:..." , "harness_revision" : "git:..." , "model_config" : "immutable-config-id" , "validity" : "disputed" , "result" : "pass" , "review_revision" : 3 , "evidence" : [ "fixture.log" , "review.json" ] } Publish three denominators Let: P_v , N_v : passes and total among reviewed-valid tasks; P_a , N_a : passes and total across all attempted tasks; D : disputed tasks. Report: valid-only score = P_v / N_v all-attempted score = P_a / N_a uncertainty interval = score if every disputed task hurts conclusion .. score if every disputed task helps conclusion This interval is not a statistical confidence interval. It is a sensitivity bound for unresolved task validity. A tiny sensitivity calculator #!/usr/bin/env python3 import json , sys rows = [ json . loads ( line ) for line in open ( sys . argv [ 1 ]) if line . strip ()] valid = [ r for r in rows if r [ " validity " ] == " valid " ] disputed = [ r for r in rows if r [ " validity " ] in ( " unreviewed " , " disputed " )] attempted = [ r for r in rows if r [ " result " ] in ( " pass " , " fail " )] rate = lambda passed , total : pa

Five stories moved the AI-coding world today. None are about a single model winning forever — they are about the ground shifting under who runs the agents and who pays for them. Musk puts Grok 4.5 to work at Tesla and SpaceX Tesla and SpaceX have been told to trial Grok 4.5 . The signal is not the benchmark — it is that a frontier model is being pointed at real engineering and ops inside hardware companies. When a model moves from a chatbot to a mandate inside a manufacturing and launch pipeline, the feedback loop gets brutally honest fast. Perplexity's orchestrator beats Opus on a benchmark Perplexity added Grok 4.5 to its orchestrator and reports beating Opus on the WANDR benchmark. Orchestrators are the quiet winners of this cycle: instead of one model doing everything, a router picks per-subtask. A smaller-or-cheaper mix outperforming a single flagship on a targeted benchmark is the trend to watch — it is how teams cut cost without giving up quality on the hard parts. Meta launches Muse Spark 1.1 at 25% of competitor pricing Meta shipped Muse Spark 1.1 through an API priced at roughly a quarter of what competitors charge. Price is a feature. At 25% of the field, an API becomes the default fallback router for cost-sensitive agents even if it is not the best at everything. Expect orchestrators to slot it in for the boring 80%. ByteDance rolls out Seedream 5.0 Pro ByteDance pushed Seedream 5.0 Pro across multiple platforms. Image generation keeps consolidating into a few vendor-backed models with wide distribution — relevant to coding agents the moment they need to generate UI mockups or assets inline. Cursor builds an "Office Agent" to challenge Anthropic Cursor is building a Sand AI office agent aimed at Anthropic's turf. The coding-agent wars are expanding from "writes code" to "runs the surrounding workflow" — email, docs, tickets. That is the same expansion the open-source side is feeling: oh-my-pi's model hub and OpenClaw's session fleet are both bets that th

I run ezyZip , a browser-based archive tool, so "which format should I use?" is a question I field constantly. The honest answer is usually "it depends," which satisfies nobody. So I stopped hand-waving and measured it. We benchmarked 42 archive and compression formats, spanning four decades, from 1984's Unix compress through today's Zstandard, Brotli, and context-mixing paq8px. Everything ran against the same realistic 55 MB corpus, every archive was round-trip verified byte for byte, and the whole thing reproduces from a single command. Here's what came out of it, and what I'd actually reach for. The setup Most compression benchmarks measure raw codecs on standardized corpora like Silesia. That's the right call for algorithm research and the wrong call for answering "what should I zip my folder with?" I wanted end-user formats, real CLI tools, container overhead and all, on data that looks like an actual folder. So the corpus is deliberately mixed: about 11 MB of text, 15 MB of office documents, 16 MB of images, and 13 MB of video, all public domain so it can be committed and redistributed. That mix matters. Office documents ( .docx , .xlsx , .pptx ) are themselves ZIP containers, so they stress how a tool handles already-compressed data. The JPEG and H.264 media is near-incompressible and sets an honest lower bound. The plain text and uncompressed images are where formats actually separate. Two rules kept it fair and practical: Only two levels per tool: its default, and its one "maximum compression" dial. No method tuning, no dictionary sizes, no thread-count games. That's what a normal person can reach. Everything is round-trip verified. Each archive gets extracted, and every file is hashed with SHA-256 against the original manifest. Exit codes are not trusted. That last rule earned its keep immediately. The verification gotcha On the image category, a 1985-era ARC build produced an archive that its own extractor happily unpacked, while printing a CRC warning an

Benchmark-backed Ollama has amassed 176,000 stars, and nearly 17,000 forks on Github by helping developers easily run AI on their PCs.

Android Bench is evolving, and developers can help guide that process.

"It works on my test input" is the most dangerous sentence in on-device AI development. I typed that sentence - or some version of it - a dozen times while building Redacto, our on-device PII redaction app running Gemma 4 E2B on a Samsung Galaxy S25 Ultra. The model would redact a patient name from a clinical note, I would nod, and I would move on. Then I would hand the phone to a teammate, they would type a police report, and the model would redact the suspect description instead of the victim name. The problem is not the model. The problem is that manual spot-checking is not validation. You are testing a single input against your own expectations, with all the confirmation bias that entails. When you have five domain modes (HIPAA, Financial, Tactical, Journalism, Field Service), three difficulty levels, and two candidate models, you need something systematic. You need a benchmark suite. This post covers how I built one - from dataset curation to scoring methodology to on-device infrastructure - for a hackathon app running entirely on a phone. No cloud. No API calls. No data leaving the device. Why Not Use an Existing Framework? The LLM evaluation space has mature tools. EleutherAI's lm-eval-harness is the community standard for evaluating language models against academic benchmarks like MMLU, HellaSwag, and ARC. Stanford's HELM (Holistic Evaluation of Language Models) provides a multi-metric evaluation framework with standardized scenarios. Google's BIG-bench offers hundreds of tasks for probing specific capabilities. These frameworks are excellent for what they do. They are also completely wrong for this problem, for three reasons. First, they assume server-side inference. lm-eval-harness expects to call a model through an API or load it in PyTorch on a GPU server. Redacto's model runs on a Qualcomm Hexagon NPU inside a phone. There is no Python runtime, no HuggingFace tokenizer at evaluation time, no way to hook into the framework's inference loop. Second, their

placeholder

Video AI systems consistently fail to track what happens when the camera looks away: when a scene pans away from an object in motion and returns, current models re-render the object in its original position rather than showing the logical result of off-screen change. Scaling to more parameters makes this failure worse, not better, according to WRBench , a new benchmark that tests what researchers call "world model reliability." The benchmark presents AI video systems with scenes where something happens off-screen — the camera pans away while an object is in motion, or while a light changes, or while an open door should stay open — then pans back to see what the system believes should have happened. A system that genuinely models the world would track what occurred during the off-screen interval. Current systems mostly don't. Key facts What: A new benchmark tests whether video AI systems can track what happens to parts of a scene the camera isn't currently showing. Across 23 models, the answer is mostly no — and making the models larger made the problem worse, not better. When: 2026-06-19 Primary source: read the source (arXiv 2606.20545) The benchmark covers twenty-three different video generation models and nearly ten thousand video clips across six categories of off-screen change, each designed to test a different aspect of world continuity: objects in motion, light sources changing, object states such as open or closed doors, and several others. This gives a comprehensive picture rather than a single narrow test. The most striking finding is the scaling result. The researchers tested one of the more capable video generation systems at two different sizes: a smaller version and one with more than ten times as many parameters. More parameters didn't help. Scaling made the off-screen tracking problem measurably worse. The larger model produced more realistic-looking frames, but it was less accurate about what should have happened to the parts of the scene it wasn't