标签：#data

This is a Plain English Papers summary of a research paper called OpenART Red-Teams Stateful Agents Across 10,000 Evolving Environment Scenarios . If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter . OpenART turns persistent state into the red-team target OpenART evaluates agent safety across more than 10,000 validated stateful scenarios spanning 50 domains and requiring a median of 97 tool calls. Its central claim is that safety failures can emerge from trajectories in which workspace data, permissions, memory, and plans are repeatedly modified, rather than from isolated prompts alone. The arena keeps each benign task objective and hidden safety contract fixed while changing only the target-visible environment state. This design targets delayed failures that static benchmarks can miss: an early authorized mutation may influence later decisions, expose protected resources, or produce unsafe output many steps after the original change. OpenART extends the broader idea of agent safety evaluation by making persistent environment state the object that evolves during testing. OpenART reports a pooled strict Attack Success Rate of 85.0% across 75 agent-model configurations. Strict success requires both the deterministic evaluator and a GLM-5.2 judge to identify the attack condition, so disagreements count as failures rather than being treated as partial evidence.... Continue reading the full paper summary on AIModels.fyi →

This is a Plain English Papers summary of a research paper called RA-Bench Reveals Why Crisis-Video Deepfake Detectors Fail Across Generators and Social Media . If you like these kinds of analyses, you can find more AI and machine-learning research on AIModels.fyi or follow us on Twitter . The crisis detection problem we've been getting wrong Video synthesis has reached an inflection point. Recent generators can fabricate realistic depictions of wars, natural disasters, infrastructure failures, and public emergencies so convincingly that they fool both people and current detection systems. The threat isn't hypothetical anymore. A fabricated video of a nuclear plant explosion, a hospital collapse during an earthquake, or a terrorist attack could trigger panic, military response, or severe economic disruption within hours. Yet here's the troubling part: we don't actually know if our best detection tools can handle these high-stakes scenarios in the wild. Researchers have built impressive deepfake detectors, trained them on standard benchmarks, and measured their performance. But those benchmarks test detectors against generic synthetic videos, not against the specific threat that actually matters: AI-generated crisis footage designed to fool people about real things that happened. It's like training a border guard to spot counterfeit passports in a lab with perfect lighting and a magnifying glass, then sending them to a busy airport where they have to make decisions in three seconds. The guard's failure has nothing to do with their skill. The problem is that the testing environment was completely divorced from the real scenario.... Continue reading the full paper summary on AIModels.fyi →

This is a Plain English Papers summary of a research paper called Macaron-V1: Continual Learning with Self-Improvement and Mixture-of-LoRA Adapters . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The problem with frozen models Most AI systems today follow a familiar pattern: train, evaluate, deploy, and then stop. The model is locked at that moment, treated as a finished product rather than a living system. But the real world immediately begins to diverge from training data. Users interact with the system in ways the training process never anticipated. New domains emerge. Preferences shift. The model that seemed smart on test day becomes gradually less relevant over time. This frozen-in-place approach isn't accidental. It reflects how machine learning has been practiced for decades. Retraining is expensive. Deploying new versions carries risk. The infrastructure to continuously improve systems in production barely exists. So instead, teams ship a model and move on, accepting that it will decay slowly but inevitably. Macaron-V1 asks a different question: what if AI systems could continuously improve themselves through real-world experience, learning from the billions of interactions that happen after deployment? Not in theory, but actually, in production, with users. The answer isn't magic. It requires two architectural shifts. First, treat deployment as the beginning of a learning process, not the end of one. Build versioning, evaluation contracts, and feedback loops directly into the system. Second, stop assuming you need to retrain your entire model. Instead, freeze a stable base and compose lightweight specialist adapters around it, allowing the system to grow in capability without losing its foundation. Rethinking deployment as a continuous learning opportunity The insight here is architectural. Instead of viewing the deployed model as the final form, Macaron-V1 treats it as the first link in an infinit

This is a Plain English Papers summary of a research paper called BDH-CQ Uses Recurrent Latent Reasoning to Cut ARC-AGI Inference Costs . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The cost-accuracy trap in visual reasoning Large language models are fundamentally mismatched for visual reasoning tasks. They're forced to describe every thought out loud, generating token after token to explain their logic. This verbosity taxes compute budgets, yet paradoxically doesn't improve performance. Ask a language model to solve an ARC-AGI puzzle (a visual reasoning benchmark designed to test abstract thinking), and it either struggles despite the verbosity or succeeds expensively. The root problem runs deeper than just inference cost: the model learns from demonstrations by parsing them as language tokens, which is an indirect and inefficient way to absorb a visual pattern. The efficiency frontier has been unforgiving. If you want cheap inference, you sacrifice accuracy. If you want accuracy, you sacrifice cost. Every model on the leaderboard until recently clustered into one of two camps, and no one had found a path that broke the tradeoff. BDH-CQ challenges this assumption by proposing something radical: reasoning doesn't need to be visible to work. The model absorbs demonstrations silently into its internal memory state, then solves problems through private iteration in hidden layers, without generating a single token of intermediate reasoning. A 150-parameter variant achieves 29.5% pass@2 on the ARC-AGI-1 benchmark at a computed cost of just $0.0007 per task, puncturing through the previous Pareto frontier and establishing a new state of the art in cost efficiency. Learning through hidden states The core insight is deceptively simple: a model's reasoning process doesn't need to match human communication. When you learn a new skill from examples, you don't narrate every observation. You absorb patterns directly i

This is a Plain English Papers summary of a research paper called How Cross-Model Compatibility Lets Attackers Extract Proprietary LLM Reasoning Traces . If you like these kinds of analyses, you can find more research on AIModels.fyi or follow us on Twitter . The illusion of safety Major AI companies now show users their models' step-by-step reasoning as a feature. OpenAI offers it through o1, Anthropic through extended thinking, Google through its reasoning-focused variants. But this reasoning is a double-edged sword. It's intellectually valuable to share, showing users why a model reached a conclusion. But it's also intellectually valuable to steal. Competitors want to understand how frontier models think. Researchers want to study their reasoning patterns. Attackers want to extract proprietary algorithms. So the companies made a choice: hide the reasoning from users by encrypting it. The idea sounds straightforward enough. Return the reasoning to the user's device in an encrypted, unreadable form. The user can't see it, competitors can't see it, but they can pass it back to the server in future requests if they need continuity with previous reasoning. The server alone holds the decryption keys. Problem solved. Except it wasn't. Researchers discovered that this encryption doesn't actually hide reasoning. It just makes it look hidden. The encrypted blocks are designed to work everywhere within a company's ecosystem, across different sessions and different models. That universal compatibility is a feature for convenience. But it's also an architectural vulnerability that anyone can exploit. The architectural gamble To understand where this went wrong, you need to see how the system actually works. When a user sends a request to a frontier model like GPT-4, the model internally generates a reasoning trace, the raw thought process behind its answer. Instead of returning this reasoning in plaintext, the company encrypts it on the server before sending it to the client.

We open-sourced 449 real equipment financing quotes so nobody has to trust our math Commercial equipment financing sites are almost always a black box: you land on a page, see a monthly payment, and have no way to check how that number was actually derived. The APR is picked out of thin air, the "starting at" price is aspirational, and the amortization math is never shown. We built Equipment Capital Index to do the opposite — every page shows the real per-machine price, the actual amortization schedule, and now we've published the whole underlying dataset so anyone can verify or build on it. What's actually in the dataset equipment-financing-rate-data is a CC BY 4.0 dataset of aggregate financing benchmarks computed from 449 individually priced, real machines — construction equipment, ag machinery, trucking fleet, power equipment, and material handling gear. No survey estimates, no fabricated averages. Current live snapshot: Category Machines tracked Avg APR Avg est. monthly payment Heavy Construction 222 8.25% $3,272 Agriculture 84 7.75% $4,411 Trucking Fleet 63 8.00% $2,501 Power Equipment 44 8.50% $957 Material Handling 36 8.50% $825 Site-wide average: 8.17% APR , $2,954/mo across all 449 machines. Why this exists A couple of principles drove the design: Every number traces back to a real machine. Each of the 449 rows has a sourced price (dealer listing, MSRP, or a documented class-typical estimate — and it's disclosed which one) and a real amortization calculation, not a rounded guess. The math is reproducible, not just displayed. The same aggregation logic that powers the /press page on the site also generates this dataset — one source of truth computed twice, so the numbers can't silently drift apart. It shouldn't require scraping a webpage. The data has three independent, permanent homes: A live JSON API: /api/rate-report.json ( OpenAPI spec ) A self-updating GitHub repo (regenerates from live data every 3 days via GitHub Actions) A permanent, versioned DOI o

La forma del problema Un directorio es una superficie: el miembro lo abre y adivina. Un recomendador es una superficie de empujar: el sistema propone y tiene que justificarse. La justificación es la parte difícil, y es donde vive la estadística. Tres restricciones hicieron esto distinto de un recomendador de contenido: El item es una persona con capacidad finita. Un hilo se le puede recomendar a diez mil personas. Un experto no. Una mala recomendación es cara de los dos lados. Quien pide desperdicia una petición, el experto desperdicia una hora, y los dos aprenden a ignorar la superficie. La afirmación tiene que ser checable. "Quizá te guste este hilo" no necesita evidencia. "Esta persona está un nivel adelante de ti en diseño de sistemas" sí. Recuperación: híbrida, fusionada con RRF Tres recuperadores independientes sobre el conjunto de expertos elegibles, fusionados con Reciprocal Rank Fusion: def rrf_fuse ( * ranked_lists , k = 60 ): """ Fusiona listas de ids rankeadas. El score depende solo del rank, nunca de la escala propia del recuperador, que es el punto: la similitud coseno y un conteo de hilos resueltos no son números comparables. """ fused = {} for lst in ranked_lists : for rank , key in enumerate ( lst ): fused [ key ] = fused . get ( key , 0.0 ) + 1.0 / ( k + rank ) return fused RRF es la primitiva correcta aquí por una razón que vale la pena decir: los recuperadores emiten cantidades incomparables. Uno regresa un coseno en [-1, 1] , uno regresa un conteo entero de hilos resueltos, uno regresa un delta de nivel de escalera. Normalizarlos a una escala común requiere supuestos sobre sus distribuciones que nadie tiene a este volumen de datos. RRF descarta las magnitudes y se queda solo con el orden, que es exactamente la información que sobrevive a una muestra chica. k = 60 es la constante estándar de la formulación original de Cormack et al. Aplana la cabeza: la diferencia entre el rank 1 y el rank 2 es 1/61 - 1/62 ≈ 0.00026 , así que un recuperador no pu

Electric-vehicle assistants rarely have just one job. A short Turkish question may ask for a nearby station, a charging-price comparison, help planning a route, or an explanation of battery health. Before an application can retrieve current data or generate an answer, it needs to identify that intent reliably. We created the Turkish EV Charging Intent Dataset as a small, transparent starting point for that routing problem. Version 1.0.0 contains 192 Turkish queries distributed evenly across eight intent classes. It is open under CC BY 4.0, includes fixed train, validation, and test splits, and is maintained by TekPedal , an EV charging map and vehicle decision platform for Türkiye. You can explore the dataset interactively , inspect the source and validation workflow on GitHub , or cite the permanent Zenodo release with DOI 10.5281/zenodo.22062688 . Why intent routing comes first An assistant should not answer every EV question in the same way. Different requests need different tools and freshness guarantees: a station request needs a map or location index; a price request needs current tariff data; route planning needs distance, range, and charging-stop logic; a battery question needs careful educational content; a vehicle comparison needs structured specifications. An intent router makes that separation explicit. It can send each query to the correct retrieval source, product page, or application workflow. This also makes evaluation easier: teams can test routing independently before measuring the quality of downstream answers. Dataset design The taxonomy contains eight balanced classes, with 24 records in each class: FIND_STATION COMPARE_PRICE ROUTE_PLANNING CHARGING_SPEED VEHICLE_COMPARISON HOME_CHARGING BATTERY_HEALTH OWNERSHIP_COST Every record includes a stable ID, the Turkish query, the intent identifier, a human-readable Turkish label, a suggested TekPedal content route, the assigned split, the language, and a provenance marker. Here is a simplified example

Managed Data Lake: A Guide for 2027 Apache Iceberg is the standard table format for production data lakes in 2027. Every major engine reads and writes it natively. The catalog ecosystem standardized on REST. You own your data on commodity storage with no lock-in. But Iceberg deliberately separates the table format from the system that keeps tables healthy. It gives you the primitives for maintenance — rewrite_data_files , expire_snapshots , remove_orphan_files , rewrite_manifests — but not the intelligence to decide when, how, and in what order to run them. Without that operational layer, every Iceberg table degrades over time: small files accumulate, snapshots bloat metadata, sort orders drift from query patterns, orphan files inflate storage costs, and query performance decays silently until something breaks visibly. This operational gap is the central challenge of running a data lake at production scale. Netflix built four internal services to address it — Autotune for compaction strategy selection, Polaris for catalog management, janitors for garbage collection, Metacat for cross-service observability — each staffed by dedicated teams over multiple years. Google engineered automatic compaction and garbage collection directly into BigLake , so their managed Iceberg tables stay healthy regardless of write volume or query pattern changes. In 2027, you do not need to replicate that investment. This guide covers what "managed" actually means for a data lake, the degradation mechanics that make it necessary, the control plane architecture that solves it, and the practical paths to getting there — whether you are running 50 tables or 5,000. Why Lakes Degrade — The Mechanics The degradation pattern is predictable and present in nearly every Iceberg lake running for more than three months without dedicated maintenance. Understanding these mechanics is necessary regardless of which management approach you choose. The Small-File Problem Every streaming writer — Flink, Spar

Opinion: Your Tests Can't See What a Migration Destroys — Dry-Run It on a Clone A green test suite is the wrong tool for judging an AI-generated migration, because tests run against the post-migration schema and never observe the intermediate states where data disappears. The up migration is the visible artifact that gets reviewed, while the down migration is treated as an afterthought even though it is the only safety net when the deployment goes wrong. Free model access makes the problem structural: generation cost drops to zero, so migration volume rises, and every additional migration multiplies the surface for unreviewed data loss. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Tests validate the destination, not the journey When a test suite runs against a migrated database, it confirms that the application can read the new schema, but it cannot confirm that the migration preserved the data it was supposed to preserve. The test runner connects after the migration has executed, so it never sees the moment when a column is dropped, a table is renamed, or a constraint is silently relaxed. A migration that passes every test can still destroy production data, because the tests were designed to validate application behavior, not migration safety. The standard mitigation is a staging database, but staging is a poor substitute for a dry run because it has different data, different volume, and different usage patterns. The dry run I recommend uses a clone of the production schema with a representative data sample, and it exercises both directions of the migration with data integrity checks at every step. The clone does not need to be large; a few thousand rows per table is enough to expose most destructive patterns. The dry-run workflow in five steps The workflow is deliberately mechanical, because the goal is to remove judgment from the verification process and reserve human attention for the migration's intent: Clone the schema and lo

I thought that when i join Lux Dev i would jump straight into building complex data pipelines and getting to understand kafka, kafka sounds like a really cool name, but if there's one thing I'm realizing quickly, it's that before you can orchestrate complex data pipelines or deploy web scrapers, you have to master the absolute basics of version control. This week, I was working on setting up a new local project, a health records analysis and pushing it to GitHub entirely through the command line. If you're just starting out with version control, here is exactly how I took a project from a completely blank folder on my desktop to a live repository on GitHub, including testing SSH keys. Setting Up the Local Project First, I needed a place for my project to live. I opened my bash terminal, navigated to my Desktop using he cd command, and created the main project folder along with a sub-folder for the data named Data. cd Desktop mkdir -p Kenya_Hospital_Health_Records_Project/Data cd Kenya_Hospital_Health_Records_Project With the directories created, I copied and pasted my Kenya_Hospital_Health_Records_Project.csv data set we were given in class into the Data folder. Writing the README via Terminal Instead of opening a text editor, I decided to build out my README.md right from the command line using echo command. The > operator adds new text the file, while >> adds text to the already creaed line. echo "# KENYA HEALTH RECORDS ANALYSIS" > README.md echo "## Project Overview" >> README.md echo "This project analyses health records of a hospital" >> README.md I also added a quick list of tools and challenges using the same method and used the cat README.md command to print the contents of the file directly in the terminal to confirm that everything looked right. Initializing and Staging Now it was time to turned this folder into a tracked Git repository. git init Running git status showed that my Data/ folder and README.md were untracked. To stage them for my first commit,

Short version of a post on gex.live/research ; the full write-up, definitions and reproduce block live there. Most published dealer-gamma numbers are built from open interest : yesterday's outstanding contracts, multiplied by a convention about who holds which side. Whether the convention is right is a separate question. The prior question is simpler: how much of what trades today was already in that book this morning — and how much of tomorrow's book is being created today? Open interest and volume are enough to answer it, with no assumption about who bought. Sample: SPX and SPXW, 2022-04-14 to 2026-08-14, 1,081 trading days, every expiry within about a month (0DTE plus the 21 nearest), 8.6 million contract-days, 4.3 million with volume. Definitions Per contract (expiry, strike, right) and session D: OI(D) is open interest at the start of D, OI(D+1) at the start of the next session, ΔOI = OI(D+1) − OI(D) , vol the day's volume in that contract. |ΔOI| / vol is a lower bound on how one-sided the day's trading in that contract was — 1.0 means every lot opened (or every lot closed), 0 means opens and closes cancelled. Contracts expiring on D have no next-day OI and drop out of the ΔOI statistics; 4.1% of rows (3.8% of volume) show |ΔOI| > vol, which is impossible (OI snapshot timing) and are excluded. The book grows by 40% of what trades, every day days to expiry on D net ΔOI / volume |ΔOI| / volume (lower bound on one-sidedness) share of volume in contracts whose OI rose contract-days 1–5 37.8% 41.8% 90.3% 813,013 6–21 42.7% 53.3% 81.1% 2,206,446 22+ 42.8% 57.6% 76.5% 831,896 Across the whole book, net ΔOI is 39.9% of the day's volume on the median session (IQR 36.2–44.0%), positive in every year and every expiry bucket: the SPX book is always being built faster than it is unwound, until expiry does the unwinding. Far expiries are open-and-hold (a day's trading in a 22+ DTE contract is at least 58% one-sided); the nearest expiries churn (42% at 1–5 DTE). Per contract-

An AI assistant will write you a query in ten seconds, the query will run, and the number that comes back will look completely reasonable. This page gives you the five checks that tell you whether that number is right. They take about two minutes, they need no tools beyond the database you already have, and they catch the four mistakes AI-written SQL actually makes. The order matters. The checks are arranged cheapest first, so the first one costs a single row count and the last one costs a short conversation. Most wrong queries fall to the first two. The short version. A query that runs has only passed a grammar check. The number is right when the rows, the filters and the denominator match the question you asked. The database only takes a query as far as the first gate. Why a query that runs can still be wrong Before the list: what do you think the database actually checks when it accepts a query? Grammar. That is the whole list. Spell a table name wrong and you get an error. Sum the wrong column, join in a way that doubles rows, or filter after grouping when the question needed it before, and you get a clean result set with a wrong number in it. Every mistake on this page is valid SQL. AI assistants add one specific difficulty: their queries are fluent. The aliases are tidy, the formatting is clean, and the shape looks like something a careful person wrote. Fluency reads as correctness, and it is not the same thing. Treat an AI query the way you would treat a first draft from a new colleague: with respect, and with the row counts open. The table the examples run on Everything below runs on one small shop dataset, so every number can be checked by hand. Thirteen orders in July, five customers, and a refunds table where two orders were refunded in two parts. Eleven of the thirteen orders are completed; one is refunded, one is pending. There is also a staff_accounts table listing internal accounts, and it contains one NULL row, because real lookup tables usually do.

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you will be running real SQL against a real database with nothing installed, and you will know which of the free browser tools suits which job. You will also know the thing none of them puts on the front page: some of them run entirely inside your browser, and some upload whatever you paste to a stranger's server. That difference decides what you are allowed to practise on. Here is what to actually do today. If you want a database already loaded and questions already written, open sql-practice.com . If you want to create your own tables and share the result with someone, open DB Fiddle . Both start working immediately with no account. The short version: browser-only tools keep your data on your machine, server-backed tools do not, and neither kind is the right place for anything from work. Where the data goes is the one idea that should drive your choice, so it gets the picture. The original carries a diagram here. In words: Two panels side by side, each drawn as a laptop outline containing a browser window. In the left panel a small data box sits inside the browser window, with a short circular arrow looping back into itself, showing the data never leaves the laptop. In the right panel the same data box has a long arrow leading out of the laptop, across a gap, and into a separate server rack drawn beyond the laptop's edge, with a copy of the data box now sitting in the rack as well. The original box remains, showing the data has been copied out rather than moved. Every tool below was opened and checked on 8 August 2026. These sites change often, so the descriptions describe what was actually on screen, and anything I could not confirm by looking is not claimed here. 1. Run your first query, right now Before the explanation: what do you think has to exist on your computer for a SELECT statement to return rows? The honest answer is nothing at all, and that surprises people who have sp

By Michael Nocito , data analyst · Published August 8, 2026 By the end of this page you will have a real database sitting on your own computer, with 11 tables, 3,503 tracks and 412 customer invoices in it, and you will have run a query that proves every table arrived intact. Then you will run a join across two of those tables, which is the thing a single spreadsheet can never teach you. It takes about five minutes and costs nothing. Here is what to actually do today. Download the Chinook database file, open it in DB Browser for SQLite, and run one query that counts the rows in every table. If the counts match the ones printed below, you have a working practice environment and you can stop shopping for one. The short version: get Chinook_Sqlite.sqlite , open it, count the rows, then join two tables. Northwind and Sakila are the other two names you will see, and there is a table further down saying when each is the right pick. The reason a sample database beats the CSV you already have is one idea, so it gets the picture. The original carries a diagram here. In words: Two panels side by side. The left panel holds a single grid of rows and columns, standing alone with nothing attached to it. The right panel holds four smaller grids arranged around each other. A highlighted column at the edge of each small grid is joined by a solid line to a matching highlighted column on a neighbouring grid, so all four grids are wired together into a connected shape. The left panel has no lines at all, because there is nothing for a line to reach. Every number on this page is real. I downloaded Chinook v1.4.5 and Northwind on 8 August 2026 and ran each query with SQLite 3.51.1. The counts, the outputs and the row multiplication are what came back, not what should have come back. If you have no database software at all yet, how to set up a SQL database is the fifteen-minute version of that step, and this page picks up right after it. 1. Why one CSV is not enough Before the explanation:

Cheap energy, abundant land, and proximity to Beijing have turned a city in Inner Mongolia into a crucial hub for data centers.

Nvidia continues to pour money into data center development — just as AI data centers bring lots of money into Nvidia.

There is a comforting story about how AI models handle the enormous quantities of text and images they are trained on: they do not store any of it, they merely learn general patterns, and once training is done the original data is gone in any meaningful sense. It is a reassuring account, and it is not quite true. Large models memorise fragments of their training data — verbatim, recoverable fragments — and a decade of research has produced reliable ways to detect and extract them. The answer-first version: if your data was in a model’s training set, the model may have memorised identifiable pieces of it, and those pieces can leak. Two families of attack make this concrete. Membership inference works out whether a specific record was in the training data at all. Data extraction pulls memorised content back out word-for-word. Neither is exotic; both are well documented against production systems. This is the mechanism underneath both the newspaper lawsuits alleging near-verbatim reproduction of their articles and the quieter privacy research showing that models leak the people in their training sets. Understanding it is the difference between trusting the comforting story and knowing its limits. Memorisation is a feature of the maths, not a bug Start with why models memorise at all. A large neural network has an enormous number of parameters — enough capacity to do more than compress general patterns. During training it is rewarded for predicting its training data accurately, and one very effective way to predict a specific example accurately is to memorise it. For data that appears once in an unusual form, or many times in an identical form, memorisation is often the path of least resistance for the optimiser. This is measurable. Researchers can show that a model assigns systematically higher confidence, and lower prediction error, to examples it was trained on than to otherwise-similar examples it has never seen. The size of that gap grows with the size of the model

Some Americans fear the FTC may be thinking about personalized pricing all wrong.

There's about to be a big fight to secure access to space.