标签：#ci

I watched a new tool try to get around the model safeguards of four major frontier companies. You might be surprised by how they performed.

Companies are making sure we have a surplus of options for building qubits.

Climate change is helping invasive species to not only establish themselves across the continent but to stay alive for longer. Scientists suggest we get used to it.

X and the World Federation of Advertisers say they're resetting their relationship.

X sued the WFA in 2024 for conducting what it called a "systematic illegal boycott" of the platform after it saw a decline in advertising revenue following Musk's $44 billion takeover of the social network

Elon Musk’s platform attacked the country’s “invasive” information-gathering powers as it demands the government drop efforts to strengthen its no-children policy.

AI is fantastic at spotting patterns, but human insight is the key.

DoorDash is launching a new drone delivery program called DoorDash Air. The largest food delivery app in the US said that it has approval from the Federal Aviation Administration that clears the way for drone delivery in the near future. DoorDash said it has received a Part 135 air carrier certification from the FAA that […]

Novelists, journalists, and power LinkedIn posters are embracing first-person narratives and idiosyncrasies to avoid being mistaken for chat bots.

Daniel Doubrovkine explains why traditional LeetCode whiteboard interviews fail to evaluate senior engineering talent. He discusses his own experience bombing basic algorithm tests despite decades of leadership, and shares actionable frameworks for redefining the interview loop. Discover how evaluating human judgment, system design, and hands-on AI collaboration yields far better hiring signals. By Daniel Doubrovkine

Startup Zanskar has created one of the most productive geothermal wells in the US at a power plant that had been in decline for years.

We like to think of our home as a nice, smooth sphere. But mapping the Earth’s gravitational field provides a different view of the planet.

It feels bad enough when an open letter signed by leading economists warns that AI might steal your job. The fact it may soon be better than you at making dinner? Insult to injury. But that’s exactly what the company 1X promised when it showed off a pair of new, impressively dexterous (and, to some,…

I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no. That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources. Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact. This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me. The setup I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like: "Best personal injury law firm in New York City?" "Top immigration lawyers in Mumbai?" For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose. First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap. Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer: Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself. If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal. The control The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

TL;DR Eight engineering disasters. Zero arithmetic errors. Every single one was caused by two numbers — both correct, both carefully computed — meaning different things on opposite sides of a software interface. One cost $65 billion. Another killed 28 soldiers because 0.1 can't be represented in binary. The fix is never the math. The fix is the label. There is a particular kind of silence in a control room when someone realizes the number on the screen is in the wrong unit. It lasts about two seconds. Then it's replaced by the kind of noise nobody wants to hear. On September 23, 1999, that silence happened at the Jet Propulsion Laboratory in Pasadena, California. The Mars Climate Orbiter had just disappeared behind the planet. Telemetry showed the spacecraft at 57 kilometers above the surface. It was supposed to be at 140. The silence was four seconds long. Then someone said "oh no" — the official NASA transcript uses a stronger word — and $327 million of aluminum, titanium, and human effort disintegrated into the Martian atmosphere. What follows are eight stories about the same bug, wearing different uniforms. Some are famous. Some you've never heard of. Two of them are pure software failures that every developer who's ever written for (let i = 0; i < 10; i += 0.1) has come within a rounding error of replicating. 1. The Patriot Missile — When 0.1 Is Not 0.1 (1991) Let's start with the one that belongs in every CS curriculum. Because this isn't a "unit conversion" error in the traditional sense — nobody confused meters and feet. The error was in the way a computer counted time. And it killed 28 American soldiers in a warehouse in Dhahran, Saudi Arabia. The MIM-104 Patriot missile system tracks incoming targets using a phased-array radar. The radar scans the sky, and the fire-control computer predicts where the target will be when the interceptor arrives. That prediction depends on knowing exactly when the radar echo returned. Time is measured by the system's interna

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

In a new disclosure, OpenAI says its agent used exposed logins to gain access to at least four “publicly available services” in its unhinged quest to solve a test.

"Preliminary investigation shows that two of Link's three reaction wheels currently are not operable."

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

“We're in the realm where we might be essentially killing off everything within that first hour or two."