标签：#Testing

If you've built an MCP App — the HTML widget an MCP server hands a host to render inline — you may have hit this: the tool call succeeds, structuredContent comes back fine, the model announces that a widget rendered, and the user sees nothing. No error. No console output. Just a gap in the conversation. There's a long issue full of people with this exact symptom, all of them (me included) posting variations of "my server is spec-correct and nothing renders." That's a hard thing to act on. So I built a probe server designed to answer one question at a time and ran it 36 times. This post is mostly about method — how you experiment on a host whose source you can't read and whose renderer you can't attach a debugger to. The MCP specifics are the worked example. The most useful part is at the end, where my measurements lied to me twice. Everything behavioural here was measured on 31 July 2026 against claude.ai web. Host behaviour changes; treat the numbers as a snapshot, not a spec. The finding, up front The most-upvoted lead in that thread says claude.ai silently refuses to place the iframe unless your resource declares _meta.ui.domain , computed as sha256(<your endpoint URL>)[:32] + ".claudemcpcontent.com" . Here's what varying that one field actually does: _meta.ui.domain iframe mounted sandbox origin computed value 10/10 one stable origin, every render absent 10/10 host default — differs per conversation present but wrong 0/8 never created Omitting it doesn't stop anything. What it actually controls is origin stability , which is exactly what the SDK docs say it's for: a fixed origin your API server can allowlist for CORS. But a wrong value is fatal. And the easy way to produce one is hashing an endpoint string that differs slightly from the URL the client connected with — a trailing slash, a missing path segment, http vs https . So the advice inverts the risk: follow it imprecisely and you convert a working app into a broken one. The original comment wasn't wrong ab

Delete a Telegram bot and https://t.me/your_deleted_bot keeps returning HTTP 200 with a page that looks completely normal. Every link checker I know of — CI actions, directory scripts, monitoring cron jobs — reports it as healthy forever. If you maintain anything that lists Telegram bots, some fraction of your list is already dead and your checks are telling you it is fine. Reproducing it Pick a username that has never existed: curl -s -o /dev/null -w "%{http_code} \n " https://t.me/nonexistent_test_bot_77712 # 200 Two hundred. No redirect, no 404, no soft-404 marker in the body that a status check would catch. Where the truth is The status code is useless here, but the Open Graph title is not. I measured four usernames — two live bots, two that do not exist: URL og:title t.me/BookClassBot (live) BookClass t.me/instanavy_bot (live) StoryViewer - anonymous instagram story viewer tool t.me/nonexistent_test_bot_77712 Telegram: Contact @nonexistent_test_bot_77712 t.me/zzz_definitely_not_a_real_bot_9182 Telegram – a new era of messaging A live bot puts its own display name in og:title . A dead one gets one of two Telegram placeholders: Telegram: Contact @<username> , or — if the username is not even syntactically valid — the generic Telegram – a new era of messaging . That is the whole signal. curl -s https://t.me/some_bot | grep -o '<meta property="og:title" content="[^"]*"' The check Standard library only, no dependencies: import re import urllib.request UA = " Mozilla/5.0 (compatible; linkcheck/1.0) " DEAD_EXACT = { " Telegram – a new era of messaging " , " Telegram " } DEAD_PREFIX = " Telegram: Contact @ " def telegram_bot_exists ( url : str ) -> bool : """ True if the bot behind a t.me URL still exists. HTTP status is not usable here: Telegram serves 200 with a placeholder page for usernames that were deleted or never existed. The Open Graph title is what actually differs. """ req = urllib . request . Request ( url , headers = { " User-Agent " : UA }) with urllib .

Quick check: does your agent stack have a memory.md in it somewhere? An AGENTS.md ? A notes file the agent appends to when something seems worth keeping? Thought so. Mine did too. It's the pattern everyone converges on, it takes twenty minutes to build, and it genuinely works — right up until the day it hands a customer a fact that stopped being true in March. This post does three things: shows you exactly why the pattern rots (with a real-shaped sample file we'll dissect), gives you a small script to audit your own file tonight, and walks through the architecture change that actually fixes it. No vendor required for any of it. The pattern we all built Strip away the framework and every self-managed memory loop looks like this: MEMORY = Path ( " memory.md " ) def run_task ( task : str ) -> str : context = MEMORY . read_text () # 1. dump everything in result = llm ( SYSTEM + context + task ) # 2. do the actual job note = llm ( # 3. agent grades its own homework " What from this interaction is worth remembering? " " Reply with one line, or NONE. \n\n " + result ) if note . strip () != " NONE " : with MEMORY . open ( " a " ) as f : # 4. append forever f . write ( f " - { note . strip () } \n " ) return result Be fair to it first: this is human-readable, versionable, greppable, zero-infrastructure. For one agent, one job, small working set — it's honestly hard to beat. Now look at what it doesn't do. Step 4 is the entire lifecycle. Nothing in this loop ever updates, merges, expires, or questions a line once written. The file has exactly one behavior: it grows. Dissecting a six-month-old memory file Here's a condensed, realistic slice of what that loop produces by month six. Read it the way retrieval reads it — every line equally true: - Customer Acme runs their workload in us-east1 - Acme prefers Slack over email for escalations - Acme's staging env uses the legacy auth flow - Acme contact is Priya (prefers email) - The feature flag `beta_router` must stay ON for Acme -

Every recorded mobile test I have ever inherited died the same way: someone moved a button. The recording said "tap at (340, 712)". The redesign moved that button up by one row, and the test kept tapping — now on empty space, or whatever happened to land there instead. It didn't fail right away. Three sprints later, it started failing in confusing ways, and by then nobody trusted the suite anymore. The fix isn't a better recorder. It's recording a different thing: not where you tapped, but what you tapped. That needs an element tree, and for a while we didn't have one. tapflow is an open-source, self-hosted tool that streams iOS simulators and Android emulators into a browser, so a whole team can test builds without installing anything. Until now, everything it moved was pixels in one direction and taps in the other. This post is about getting an element tree out of a simulator with no window, on both platforms. What we do with that tree — replaying flows that survive a redesign — is the next post in this series. The automation axis this feeds — the flow runner and the MCP server — is experimental . The manual browser QA path is the mature one. The constraint: no WebDriverAgent, and no simulator window tapflow already injects touches into the iOS simulator without WebDriverAgent — it loads CoreSimulator.framework and pushes HID events through SimDeviceLegacyHIDClient (that story is ep.1 ). Streaming reads the framebuffer IOSurface directly. Neither path needs Simulator.app on screen, and that's deliberate: an agent Mac in a closet running four simulators shouldn't be babysitting four windows. So whatever we used for the tree had to follow the same rule. No WDA to install and keep in sync with Xcode. No simulator window on screen. Our first attempt ran into exactly that limitation. macOS exposes an accessibility API ( AXUIElement ), and Simulator.app publishes its content through it. We wrote a helper around it, and it worked perfectly on a developer's laptop. On the

Every LLM evaluation framework today invents its own test case format, its own grader definitions, and its own results schema. DeepEval, Promptfoo, Inspect AI, and lm-evaluation-harness all solve the same core problem (checking whether a model's output is correct) but none of them can read each other's eval datasets. That means every time a team wants to compare frameworks, or move from a notebook prototype to a production eval pipeline, they end up hand-rewriting the same test cases over and over. OpenEval is an attempt to fix that by defining a small, portable JSON Schema for eval test cases, graders, and results, plus tooling to move data between frameworks instead of retyping it. What's in the repo: A versioned JSON Schema spec for test cases, grader configs, and result records. TypeScript and Python SDKs for reading and writing OpenEval-formatted datasets. A CLI with validate, convert, init, and summarize commands. Converters for popular frameworks so existing datasets can be brought in or exported out without a manual rewrite. The project just published v1.0.0 to npm and PyPI, and issues are open on 17+ framework integrations if anyone wants to help wire up a converter for a framework not yet covered. Repo: https://github.com/adhabnr-ux/openeval Would love feedback from anyone who has hit this same portability problem while switching between eval tools.

1. Introduction Most public demonstrations of AI coding agents begin with an empty directory and end when a visible feature works. That is useful, but it measures only the first and easiest part of software engineering: generation. Real projects are stateful. They contain existing behavior, data that must survive, security boundaries, partially completed work, and verification steps that cannot be replaced by a convincing demo. This benchmark was designed as a small, practical record of that harder problem. It documents two Codex runs on the same full-stack task-manager project. Phase 1 was a greenfield build. Phase 2 returned to the existing application for an authentication and database-migration refactor. The second run also included a long, user-initiated interruption, making recovery part of the observed workflow. The narrow result is encouraging: the initial application was recorded as complete in approximately 34 minutes, and the later refactor took approximately 42 minutes of active execution, excluding the user-initiated pause. The Phase 2 run reported 16 backend tests, 1 legacy-migration test, and 25 frontend tests passing, along with lint, type, build, dependency, and browser-acceptance checks. Those numbers need boundaries. This is a single-environment case study, not an official benchmark, not a controlled comparison between providers, and not a production guarantee. The current documentation set does not retain the complete source tree, exact prompts, raw terminal output, migration artifact, database fixture, or browser trace. Results are therefore presented as supplied run records rather than independently reproduced evidence. 2. Why Simple Code Generation Tests Are Insufficient A prompt such as "build a task app" primarily tests synthesis. The agent chooses a stack, creates files, connects components, and produces a working path. It may reveal speed and basic tool use, but it says little about how the agent behaves when constraints collide. Maintenan

Download my Automation Testing Interview Questions from ⬇️ Apple AppStore - https://apps.apple.com/us/app/qa-automation-interview-prep/id6786760948 👈 ⬇️ Playstore - https://play.google.com/store/apps/details?id=com.app.seleniuminterviewquestions 👈 The standard advice for passing a QA Engineering interview is broken. If you ask a forum how to prepare, you will be told to "learn Playwright," "memorize XPath," or "know how to write a basic API GET request in Postman." That advice worked in 2021. Today, engineering teams do not want manual testers who learned basic syntax. They are hiring Software Engineers in Test (SDETs) who understand system architecture, CI/CD pipelines, and data state. If you are failing technical rounds, it is rarely because you forgot a WebDriver command. It is because you are testing the syntax instead of the system. Here are the two architectural concepts you are actually being judged on in a modern QA interview, and how to approach them. 1. The API Race Condition & Idempotency Trap In a technical round, a senior engineer will rarely ask you to "test a login endpoint." Instead, they will give you a scenario like this: "We have a microservice that processes payments. The user clicks 'Submit', but the network drops, so they click it again. How do you automate a test to ensure they aren't charged twice?" The Junior Answer: "I will write an automated script that clicks the button twice quickly and checks the database." The Senior Answer (What they want to hear): "I will write a test that validates the API's idempotency . I will intercept the first request, capture the unique idempotency key from the header, and fire a duplicate POST request with the exact same payload and key. The test must assert that the backend returns a 409 Conflict or a 200 OK with the original transaction ID, verifying the database state didn't duplicate the charge." If you do not understand idempotency, payload validation, and race conditions, your API automation is just che

Missing data is not the same thing as a convenient default. That sounds obvious, yet temporal software regularly converts an empty time field into midnight, noon, the current time, or the start of a day. The interface may look complete after that conversion, but the program has silently changed an unknown fact into a known one. This matters anywhere an hour can change the result: medical timelines, transport schedules, legal deadlines, astronomical calculations, historical records, and calendrical systems. I encountered the problem while working with a BaZi calculation pipeline. A BaZi chart can use year, month, day, and hour components. If the birth time is absent, the honest result is a three-component analysis with hour-dependent conclusions withheld. Inserting noon would make the output look richer while making its provenance weaker. The useful engineering question is not “Which fallback time should we choose?” It is “How do we keep uncertainty visible through every layer of the system?” The public calculation evidence repository provides the concrete calendar-domain fixtures referenced below. The rest of this article focuses on the reusable software boundary behind them. Model knowledge, not just a string A common input model makes absence too easy to erase: const birthTime = form . time || " 12:00 " ; After this line runs, downstream code cannot tell whether noon came from the user or the fallback. Validation, analytics, caching, and the result renderer all see the same string. The information loss happens before the calculation begins. A small discriminated union keeps the two states separate: /** * @typedef {{ kind: "known", localTime: string, source: "user" }} * KnownTime * @typedef {{ kind: "unknown" }} UnknownTime * @typedef {KnownTime | UnknownTime} BirthTime */ function parseBirthTime ( value ) { const normalized = value ?. trim (); return normalized ? { kind : " known " , localTime : normalized , source : " user " } : { kind : " unknown " }; } This typ

I had a simple-sounding question: does ChatGPT recommend this business? You'd think you just ask it. Ask ChatGPT "best personal injury law firm in NYC", see if the business is named, record yes or no. That works exactly once. Ask again an hour later and you might get a different answer. Not slightly different — potentially a completely different set of firms and a completely different set of cited sources. Which means the naive version of this measurement is worthless. You're not measuring visibility, you're sampling a distribution once and calling it a fact. This is the same problem anyone gets when they try to test an LLM-backed feature. Your normal testing instinct — same input, assert on output — just doesn't apply. So here's how I ended up designing around it, and the numbers that came out, which surprised me. The setup I wanted to compare four assistants (GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, Perplexity Sonar, all with web search on) across 10 buyer-intent questions in one vertical. Something like: "Best personal injury law firm in New York City?" "Top immigration lawyers in Mumbai?" For each response I recorded two things: which businesses got named, and which URLs got cited. The cited sources come from each API's own citation metadata, so that part is structured — no scraping the prose. First pass, the results looked dramatic. The four assistants barely agreed on anything. Different firms, different sources, almost no overlap. Great finding. Except I couldn't publish it, because there was an obvious objection I couldn't answer: Maybe they weren't disagreeing with each other. Maybe each one was just disagreeing with itself. If a single assistant returns wildly different sources run to run, then "these four models cite different things" is a meaningless statement. You'd be measuring noise and calling it signal. The control The fix is the same idea as a control group. Measure the thing you're worried about, separately, and see if it explains your result.

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code. The entity making the claim ended up being the entity that decided whether the claim was right. Once you have the shape in your head you start seeing it everywhere. Three roles, not two Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three: Player — makes the claim. Your model, your service, your PR. Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard. Settler — determines what actually happened. Production outcomes. Reality. The scorer is a proxy. The settler is the thing the proxy is trying to approximate. The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure. What it looks like in code Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is , structurally — not a math failure, a role-collapse failure. LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer. Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player. Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your

Originally published at tengli.dev When I added an LLM-powered eval to mcpgrade , the first real run produced a result that looked like a scoop: context7 — a server with a perfect static score — failed tool selection 62% of the time. A model shown its two-tool catalog picked the "wrong" tool on 5 of 8 tasks. If I had shipped that number, it would have been wrong. Not slightly wrong — systematically, unfairly wrong. This post is about how I caught it, because the failure mode generalizes to most agent benchmarks people are building right now. The setup mcpgrade's --eval mode works like this: it reads a server's tool catalog, synthesizes realistic single-step tasks ("find the Slack channel where the incident was discussed"), shows a model the full catalog, and measures three things — does it pick the right tool, does it fill valid arguments, and does it correctly refuse tasks that no tool can handle. Round 1, on three real servers, cost about twelve cents and produced this: Server Static score Tool selection Args Refusal context7 (2 tools) 100 38% 100% 100% server-memory (9 tools) 81 93% 100% 100% server-slack (8 tools) 97 54% 100% 100% Two servers with excellent static scores, apparently failing live. Either static analysis was worthless, or the eval was broken. The eval was broken Every "miss" traced to one cause. Slack's post_message needs a thread_ts — a value you can only get from a previous call to get_channel_history . context7's get-library-docs needs a library ID that comes from resolve-library-id . These are pipelined tools : their required arguments are produced by other tools. My task synthesizer didn't know that. It generated tasks like "reply to the thread about the outage" — without a thread timestamp. The model, quite sensibly, picked get_channel_history first (to find the thread), or declined. My grader marked both choices wrong. The model wasn't confused. The model was right . The benchmark was grading correct multi-step reasoning as failure — and me

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets violate it. You have 40 questions generated from the same 8 documents, or 200 turns from the same 30 conversations, or 150 examples that are really 50 cases with 3 paraphrases each. Those are not 200 independent observations. And when you feed a correlated set into a formula that assumes independence, the interval comes out too narrow, which means you declare differences significant that aren't. I want to walk through why, put a number on how much it matters, and show the fix, because this one is invisible: the code runs, the interval prints, and it is quietly wrong. Why clustering shrinks your real sample size Independent examples each carry their own information. Correlated examples carry overlapping information. If five questions come from the same document, and the model either understands that document or doesn't, those five outcomes move together. You did not learn five independent things about the model. You learned something closer to one and a half. The survey-statistics name for this is the design effect (Kish, "Survey Sampling," 1965). For clustered data it is approximately: Deff = 1 + (m̄ - 1) · ICC where m̄ is the average cluster size and ICC is the intra-cluster correlation, the fraction of total variance that lives between clusters rather than within them. Your effective sample size is: n_eff = n / Deff That is the number of independent examples your clustered set is actually worth. The number Take a realistic eval set: n = 200 examples, drawn from 40 source documents, so average cluster size m̄ = 5. Suppose the ICC is 0.3, which is unremarkable for "questions from the same document" (I have measured higher). Deff = 1 + (5 - 1) · 0.3 = 2.2 n_eff = 200 / 2.2 ≈ 91 Your 200-example eval is worth about 91 independent examples. The correct confidence interval

Every unsupervised AI agent we've reviewed that wrote Flutter code made the same seven mistakes. These aren't typos or stylistic differences. They're structural failures that compound—bad state management plus missing tests plus hardcoded colors means the codebase becomes expensive to theme, hard to test, and impossible to maintain at scale. Here's a small one to set the tone: a developer asked an agent to implement a GET request to an external service in a Dart project. The agent's solution was to shell out to curl via Process.run and parse the stdout. Not package:http . Not dio . Not even dart:io 's own HttpClient . A subprocess call to a CLI tool, inside a language that's had first-class HTTP clients since Dart 1.0. That one is worth sitting with, because it's not really a Flutter problem — it's the whole pattern in miniature. The agent wasn't "wrong" that curl can make a GET request. It optimized for "this pattern appears constantly in training data" over "this is the idiomatic way to do it in the language I'm currently writing." Bash and curl show up in approximately every tutorial, README, and Stack Overflow answer ever written. package:http shows up in Dart-specific docs. Given no other constraint, the agent reached for the statistically dominant pattern, not the contextually correct one. The seven gaps below are the same failure mode, just less obvious than "shells out to curl." Here's what we found, with real code examples and the fixes that work. 1. Recomputing Derived State The Problem: Agents recalculate the same values across multiple locations instead of maintaining one source of truth. Imagine a checkout flow where the cart total is computed three separate ways: In the checkout page: (items.sum + tax) - discount In the footer: items.sum - discount + tax In the order summary: (items.sum - discount) * (1 + taxRate) Different calculations. Same semantic meaning. One will break first. The Fix: Derive values once in the state layer using streams. Let all w

How to Get the Maximum Value Out of This Framework Having built and iterated on this prompt through multiple production edge cases, here are the exact execution strategies I recommend depending on your workflow: 1. The Human-in-the-Loop Workflow (Recommended for Chat UI) Run it in two separate chat threads: Don’t let long conversation history degrade your test accuracy. Run Phase 1 in Thread A to get your gap analysis and critical questions. Review the gaps, clarify what you can, and then update your original requirement text. Start Thread B for Phase 2: Open a fresh conversation, paste the updated requirements + this framework, and jump straight into generation. This completely eliminates context drift and keeps the LLM laser-focused on state mutation rules. 2. The 2-Pass Programmatic Auditor (For Automated CI/CD Pipelines) If you’re calling an LLM via API or integrating this into a pre-commit GitHub Action, split the execution into two isolated passes: Pass 1: Run Phase 1 & 2 to generate the initial test table. Pass 2 (The Audit Pass): Feed the generated table into an isolated, secondary prompt whose only job is to enforce the Verification Check (verifying exact boundary literals, API status codes, and non-mutation assertions). Separation produces drastically higher assertion reliability than asking a model to self-audit in a single turn. 3. How to Live-Demo or Teach This For Live Streams & YouTube: This framework makes for a high-signal live demo. Paste an intentionally ambiguous user story (e.g., a webhook handler or payment endpoint), watch Phase 1 halt at the gate live, discuss the surfaced edge cases on camera, reply PROCEED, and review the generated DEFERRED risk rows. It shifts the content focus from “Look at this cool AI tool” to “This is how Senior SDETs think about systems.” For Technical Writing & Post-Mortems: The progression from a naive “write me test cases” prompt to a strict 2-phase state-machine framework is a technical narrative in itself. Break

A few years ago, the hardest part of building a browser test framework was getting started. You had to choose a runner, configure browsers, create page objects, wire up reporting, add retries, manage secrets, connect it to CI, and convince someone else on the team to learn how the whole thing worked. Today, you can open an AI assistant and ask it to generate most of that before lunch. That sounds like a dramatic improvement. In some ways, it is. But it also moves the bottleneck. The question is no longer, “Can we create a framework?” The question is, “Can we operate what was created?” That distinction matters more than it appears. Generation cost is not ownership cost A generated framework feels cheap because the first version arrives quickly. The code compiles, a few tests pass, and the pull request looks more complete than anything you could have written in an afternoon. Then reality starts applying pressure. The application changes. Authentication behaves differently in staging. A shared helper starts hiding failures. Parallel workers collide over test data. Someone upgrades a dependency and three reporters stop agreeing with one another. The initial generation was fast. The ownership cost was merely deferred. This is the central problem described in what actually breaks when Claude generates a large Playwright framework . Large generated systems often fail in the seams: fixtures, abstractions, environment assumptions, test data, and conventions that were never explicitly agreed upon. The code may be readable line by line while the system remains difficult to reason about as a whole. That is a dangerous form of complexity because it looks productive. More code can hide less understanding Teams sometimes evaluate AI-generated automation by counting output: number of test files; number of scenarios; number of passing checks; number of prompts completed; number of lines added. Those numbers are easy to produce and easy to report. They are also weak proxies for confi

Everything you need to know to find your first vulnerability, get paid, and build a real reputation in cybersecurity — without breaking any laws. If you've typed "how to start bug bounty hunting" into Google recently, you're not alone. It's one of the fastest-growing searches in cybersecurity right now, and for good reason: it's one of the only paths in tech where a total beginner with no degree can find a real flaw, report it, and get paid the same week. This guide answers the questions people are actually searching in 2026 — what bug bounty hunting is, which bugs pay the most right now, how AI has changed the game, and how to land your first bounty. What Is Bug Bounty Hunting, Exactly? Companies invite independent researchers to test their websites, apps, and APIs for security flaws — legally. When you find a real vulnerability, you write a report explaining what it is, how to reproduce it, and what damage it could cause. If the company confirms it, they pay you based on severity. It's not hacking in the movie sense. It's structured, permitted testing within a defined scope — the specific domains, apps, or features the company has authorized you to test. Step outside that scope, and you've crossed from "bug bounty hunter" into "unauthorized access," which is a crime in nearly every country. The Best Platforms to Start On in 2026 Three platforms dominate the space: HackerOne — the largest and most beginner-friendly, with the widest range of programs Bugcrowd — strong onboarding and clear scope documentation Immunefi — the go-to platform if you're interested in web3 and smart contract security, which currently pays some of the highest bounties in the industry Start with Vulnerability Disclosure Programs (VDPs) — these often don't pay, but they let you build a track record, earn private invites, and practice on real targets without competing against thousands of other hunters for a bounty. What Bugs Are Actually Paying Right Now The vulnerability landscape has shifte

Sequential Testing and the SPRT: How to Stop a Test Early Without Cheating Meta description: Peeking at a fixed-sample A/B test inflates false positives. Sequential testing lets you check results repeatedly and stop early without cheating. TL;DR Fixed-sample testing assumes you'll wait for a pre-calculated sample size before looking at results. Checking early and stopping the moment you see significance — "peeking" — quietly inflates your real false-positive rate, often far above the 5% you think you're getting. Abraham Wald's Sequential Probability Ratio Test (SPRT), developed for wartime quality control, is the mathematically rigorous alternative: a procedure built to be checked repeatedly, with pre-calculated boundaries that keep the false-positive rate honest by construction. The difference between the SPRT and peeking isn't willpower — it's that the SPRT's stopping rule is part of the math from the start, so stopping early doesn't cost you anything in error-rate control. Sequential design is the right call when traffic is limited, the cost of running a test too long is high, or the business genuinely can't commit to waiting for a fixed horizon — not a substitute for rigor, but a different kind of rigor suited to a different constraint. This is a methodology choice, not a shortcut — and it's one input into the broader question of how much certainty a given bet needs, covered in the Confidence Tier Model . Every experimentation program eventually hits the same moment: a test has been live for four days, the dashboard shows a lift, and someone — a stakeholder, a PM, sometimes you — asks "can we call it?" The honest answer depends entirely on what kind of test you designed, and most teams don't have a clean answer, because most teams designed a fixed-sample test and are now trying to read it like a sequential one. Those are not interchangeable. Knowing the difference, and choosing deliberately between them before the test starts, is the actual skill — not "wait lon

The Confidence Tier Model: How to Decide When Your Data Isn't Enough Meta description: Most testing programs are built for traffic they don't have. Three confidence tiers — proven, directional, speculative — each with its own bet-sizing rule. TL;DR Fixed-sample A/B testing assumes you can wait for statistical significance. Most teams can't — traffic is too thin, or the market is moving too fast to wait. The fix isn't lowering your standards. It's replacing the binary "significant / not significant" gate with three explicit confidence tiers — Proven, Directional, Speculative — each with its own evidence bar and its own bet-sizing rule. Underpowered tests systematically overestimate effect size (the "winner's curse" ). A confidence tier that accounts for this is more honest than a p-value that pretends otherwise. The way to move a learning up a tier isn't more of the same test — it's triangulation: stacking correlated, individually-weak signals until they converge. This is a methodology choice, not a compromise. Teams that name their confidence tier explicitly make faster, more defensible decisions than teams that either wait for certainty they'll never reach, or ship everything with false confidence. A product manager says: "Users want better deals." A brand marketer says: "TV is driving more direct demand." A performance marketer says: "This channel has a strong ROAS." Finance says: "But is this incremental?" Product says: "Will this hurt user trust?" Leadership says: "Should we scale this?" Six people, six kinds of evidence, and a decision that needs to get made this quarter — not whenever a test finally clears p<0.05. This is the actual job: not running tests, but converting six competing claims into one evidence base leadership can act on. Most experimentation methodology is written for a world where you have the traffic to wait for a clean answer. Most companies don't live in that world. The problem classic A/B testing doesn't solve Fixed-sample significance tes

I wrote about the Electricity Planning Engine a little while back, including a timezone bug that made a correct price look "not found" after a database round trip. A few days later, Alex Shev left this comment: Timezone bugs are brutal in planning engines because the result can look mathematically correct while being operationally wrong. Energy workflows especially need tests around boundaries, not just averages. That is a genuinely sharp way to put it, and it is not just a comment about the bug I already wrote about. It is a comment about how I test the project in general, and I did not like how well it applied once I went and checked. The part that stung a little "Looks mathematically correct while being operationally wrong" is exactly what the original timezone bug was. PriceSeries::priceAt() threw a clean "price not found" error, which is arguably the good version of that failure mode: loud, easy to catch, hard to ship. A quieter version of the same class of mistake, off by one hour instead of missing entirely, would not throw anything. It would just return a plan that looks completely reasonable and is wrong the entire time it runs. Alex's second point, boundaries over averages, is the one I actually had to go check rather than just agree with in the abstract. So I opened tests/Unit/Domain/Contract/PricingStrategyTest.php and looked at every hour used in every peak/off-peak assertion: new DateTimeImmutable ( '2026-07-18 14:00:00' ) // peak new DateTimeImmutable ( '2026-07-18 23:00:00' ) // off-peak new DateTimeImmutable ( '2026-07-18 05:00:00' ) // off-peak 14:00, 23:00, 05:00. Every single one comfortably inside its window. None of them anywhere near the actual transition. The off-peak slot in the config is 22:00 to 06:00 , and the comparison behind that lives in TimeSlot::contains() : // wraparound slot, e.g. 22:00 -> 06:00 return $minuteOfDay >= $this -> startMinuteOfDay || $minuteOfDay < $this -> endMinuteOfDay ; That >= versus < is exactly the kind of one-

The CI job just hit 28 minutes. Again. You pull up the duration report expecting to blame a bloated integration test or a slow environment spin‑up. Instead the longest stage stares back at you: a collection of Cucumber feature files that haven’t caught a real bug in months. Maybe years. They run on every commit, green circle after green circle, while your team mutters about slow pipelines and nobody dares touch them. Most teams treat those scenarios like documentation. “They describe the system,” someone once said, as if a Gherkin file were a legal contract. Others cling to the sunk cost: a year ago a whole squad spent two sprints writing them, polishing the grammar, aligning step definitions. Deleting them would feel like admitting waste. Experienced engineers see it differently. They treat a scenario that never fails as a liability you’re paying for on every push. Not neutral. Liable. Compute cycles, developer attention, flake‑debugging time, and the quiet toll it takes on trust in the pipeline. The principle is blunt: if a test hasn’t failed in the last few sprints, you’re already paying its full cost and receiving nothing in return. That doesn’t mean you delete everything green. But it does mean you audit with the same seriousness you’d use for a memory leak. What the green wall actually costs The damage is not abstract. A pipeline bloated with stale scenarios hurts you in five concrete ways. First, feedback slows. Every extra minute between push and result stretches the loop that tells a developer they’re safe to merge. Multiply across a team and you’re losing hours per week to waiting. Second, flakiness increases. When you have many scenarios, a single unstable environment variable can produce a handful of failures that are not regressions at all. Engineers learn to retry, then to ignore. Third, confidence erodes. If half the suite is ceremonial, a genuine failure might be dismissed as “just another flaky test” until it reaches production. Fourth, maintenance