标签：#Testing

I Reviewed 12 Free-Tier Integrations. The Same Six Myths Kept Appearing. Last month I reviewed twelve integrations that used free model servers. All twelve carried the same wrong assumptions. None of them tested those assumptions. That's the real problem. Not the free tier. The mental model. How many of these myths do you believe? I believed all of them. Here's what the code told me. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use their free server option in side projects. The probe below works with any OpenAI-compatible endpoint, including theirs. The Six Myths Myth 1: "Free tier is just a demo" Teams treat free servers like toy boxes. They build demos, then throw them away. Evidence: three of the twelve integrations were internal tools in daily use. The free tier was the production environment. Nobody planned for that. Corrected mental model: free tier is a constraint, not a demo. If the tool survives, the constraint becomes your architecture. Design for it from day one. Myth 2: "A 200 means it worked" The most dangerous assumption. A 200 only means the HTTP layer succeeded. It says nothing about the content. I found empty completions, truncated JSON, and repeated boilerplate. All returned 200. All broke the caller. Corrected mental model: validate the payload, not the status code. Check schema, length, and content markers. Myth 3: "Retries are free" When a request fails, developers retry immediately. Then again. Then again. That's a retry storm. It amplifies load exactly when the server struggles. I saw one integration fire eleven requests in four seconds. Corrected mental model: retries are a queue, not a hammer. Use exponential backoff with jitter. Add a circuit breaker. Myth 4: "The model is the same everywhere" Free and paid tiers often serve different models. Or the same name with different behavior. You cannot assume. Evidence: two integrations hard-coded model names that no longer existed. Responses came back, but from

The people who know your AI agent best may be the people least able to see all of its flaws. Not because they are bad engineers. Because they built it. Years ago, when I was taking art classes, my teacher told me something I've never forgotten: “Sara, you can't judge your own art.” I remember thinking, of course I can. 😂 Then she explained. After spending hours looking at the same piece, your eyes get filled with it. You stop seeing what is actually there. You see what you expect to see. I've used that lesson everywhere since. And I think AI agents have the same problem. You designed the requirements. You designed the system. You know why every decision was made. Then you design the evaluation and ask: “Does my agent actually work?” That's where the blind spot can appear. Your evaluation may end up testing the system according to the same assumptions that created it. The evaluator can inherit the system's assumptions Consider a simple requirement: “The agent should answer customer questions accurately.” Seems reasonable. So the team creates an evaluation set with questions that have clear intent and well-defined answers. The agent performs beautifully. 94%. Green dashboard. 🎉 But an external evaluator might ask a different question: What happens when the customer's request has two plausible interpretations? Now you have a different test: “Can I change my billing address?” Does the agent answer immediately? Does it ask which account or address the customer means? Does it make an assumption? The original evaluation may have been technically correct. It just never tested the ambiguity. That is the blind spot. Internal evaluation is still essential This isn't an argument that internal teams shouldn't evaluate their own systems. They absolutely should. The people who built the system understand its requirements, architecture, constraints, tools, and intended behavior better than anyone. That knowledge is extremely valuable when designing evaluations. But it can also crea

Measuring the impact of test automation goes beyond simple pass/fail ratios. To demonstrate real engineering excellence and business value, automation metrics must capture execution speed, suite stability, test coverage, maintenance cost, and CI/CD integration. Here is a comprehensive, unified KPI framework designed specifically for Playwright & JavaScript automation suites. 📊 Executive KPI Targets Category Metric Target Execution Speed Runtime Reduction 50% ↓ Efficiency Throughput +40% ↑ Stability Flaky Tests < 3% Reliability Retry Dependency < 5% Coverage Automation Coverage 80%+ Quality Defect Leakage 20–30% ↓ Productivity Script Dev Time 30% ↓ CI/CD Pipeline Time 40% ↓ ROI Automation ROI Positive (3–6 months) Cost Manual Effort Reduction 30–50% ↓ 1. Execution Efficiency & Speed Test Execution Time Reduction: Target 40–60% reduction vs legacy frameworks like Selenium. $$\text{Reduction \%} = \frac{\text{Old Time} - \text{New Time}}{\text{Old Time}} \times 100$$ Parallel Execution Efficiency: Measure tests executed per hour and parallel thread utilization. $$\text{Efficiency \%} = \frac{\text{Sequential Time} - \text{Parallel Time}}{\text{Sequential Time}} \times 100$$ Test Throughput: Maximize total test cases executed per CI window. CI/CD Pipeline Cycle Time: Aim for a 30–40% total reduction in build + test execution duration. 2. Stability & Reliability Flaky Test Rate: Keep flaky tests under 2–3% by leveraging Playwright's native auto-waiting and resilient locators. $$\text{Flakiness \%} = \frac{\text{Flaky Tests}}{\text{Total Tests}} \times 100$$ Retry Dependency Ratio: Track the percentage of tests passing only after retries to minimize false positives. Failure Root Cause Accuracy: Target >90% of test failures pointing directly to genuine application defects rather than script instability. 3. Coverage Metrics Automation Coverage: Maintain 80%+ regression coverage across all functional scenarios. Cross-Browser & Device Coverage: Measure test runs across Chromi

Tracking test automation metrics manually often leads to outdated figures and missed engineering gaps. To solve this, automated reporting directly from your test suites—such as Playwright and Cucumber—provides clear visibility into health, execution speed, and coverage. Below is a breakdown of how to structure an Automation KPI Dashboard to streamline test metrics, track trends, and establish actionable engineering goals. Executive Summary Dashboard KPI Metric Target Current Value Status Trend Total Test Cases 100% coverage 85% 🟡 Partial ↗️ Up Automated Test Coverage 90%+ 78% 🟡 Partial ↗️ Up Pass Rate (Last Run) 95%+ 92% 🟡 Partial ↔️ Stable Avg. Execution Time < 30 min 28 min 🟢 Good ↘️ Down Flaky Test Rate < 2% 1.5% 🟢 Good ↔️ Stable Defects Detected — 3 🟡 Review ↔️ Stable CI/CD Pipeline Success 100% 98% 🟡 Partial ↗️ Up Key Metric Breakdowns 1. Coverage & Execution Total Test Suite: 120 tests (94 Automated, 26 Manual). Latest Run (2026-05-29): 94 executed — 87 passed, 7 failed, 0 skipped. 2. Flakiness Tracking Flaky Tests (Last 10 Runs): 2 scenarios identified. Top Offenders: Scenario A: UI timeout issues. Scenario B: Data synchronization lag. 3. Defect Detection & CI/CD Performance Defect Lifecycle: 3 opened, 1 closed (Avg. resolution time: 2 days). Pipeline Health: 98% success rate, 12 min average build time. Primary Cause of Pipeline Failure: Dependency resolution errors. Execution & Pass Rate Trends (Last 6 Runs) Run Date Pass % Fail % Flaky % Duration (min) 2026-05-29 92% 8% 2% 28 2026-05-28 91% 9% 2% 29 2026-05-27 90% 10% 3% 30 2026-05-26 89% 11% 3% 31 2026-05-25 88% 12% 4% 32 2026-05-24 87% 13% 4% 33 Next Engineering Action Items Automation Expansion: Push total automated coverage past 90%. Flakiness Mitigation: Refactor explicit waits and isolation for UI timeout and data sync scenarios. Pipeline Stability: Resolve dependency caching errors to bring CI/CD success to 100%. Optimization: Lower execution suite duration below 25 minutes using parallel run setups.

More agents do not automatically produce better engineering. They usually add total tokens, duplicated context, handoff delay, and integration risk. Their defensible advantages are narrower: reduced elapsed time for independent work, isolated investigation, or specialist evidence that one agent might omit. The useful question is therefore not “Can this task use subagents?” It is: Does this task contain independent, bounded work whose value exceeds the coordination cost? Codex How To now includes a dependency-free benchmark for testing that question instead of answering it from intuition. Disclosure: I maintain Codex How To , the independent open-source project containing the benchmark, evaluator, and measurements used here. The minimum decision rule Use one agent when the change is small, the interface is unsettled, or several steps must edit the same central files. Consider bounded orchestration only when all of these are true: The task has at least two genuine ownership surfaces. Each writer can own exclusive paths. The interface between those paths is frozen before implementation. The controller retains integration, system checks, and final review. Every worker returns concise evidence rather than a narrative transcript. One external acceptance bar can evaluate every execution method. flowchart TD A["One task contract"] --> B{"Independent write surfaces?"} B -- "No" --> C["One agent or sequential work"] B -- "Yes" --> D{"Frozen interface and exclusive paths?"} D -- "No" --> C D -- "Yes" --> E["Bounded workers"] E --> F["Controller integrates and evaluates"] F --> G{"Coverage or elapsed-time value exceeds coordination cost?"} G -- "Unproven" --> H["Keep measuring"] G -- "Repeated evidence" --> I["Adopt for this task class"] Job titles are not ownership boundaries. “Backend agent,” “test agent,” and “review agent” may still collide on the same files or execute dependent stages. A useful boundary is concrete: one writer owns incident/** , another owns web/** , and n

Part 1 covered the mechanism: a fingerprint gate decides whether a change ships in minutes over-the-air or needs a full store release. But a gate that only checks "is this native-safe" says nothing about whether the change is good . If every merge to main can reach production within minutes, your safety net can't be a release train that gives everyone time to notice a problem before it ships — it has to be built into the pipeline itself, because there's no train to catch it on the way out. The PR gate Every pull request into main runs through the same automated gate before it's mergeable: a type check, a lint pass, an automated test suite, and end-to-end checks against a real device build. None of that is negotiable — it's the floor, not a nice-to-have. E2E is a big enough topic on its own — closing the loop between what a unit test can see and what actually happens on a phone in someone's hand — that it deserves its own dedicated post rather than a paragraph here. jobs : typecheck : run : npm run typecheck lint : run : npm run lint test : run : npm test e2e : run : npm run e2e Nothing exotic under the hood — ESLint for the lint pass, Husky for local pre-commit/pre-push hooks so the same checks catch you before CI even runs, Jest as the test runner, and React Native Testing Library for component-level tests. Popular, boring, well-documented tooling on purpose — the pipeline's value is in how these are wired together and gated, not in any one tool being clever. Feature flags are the real safety valve Here's the entry condition that makes OTA-from- main safe at all: shipping code and releasing a feature are two different actions. A merge can put new code on every user's device within minutes — that's deploy. Whether that code actually does anything visible is a separate switch, controlled by a remote feature flag, not by whether the code merged. That decoupling is what makes trunk-based development survivable. Nobody has to get the timing of a merge exactly right, bec

API documentation has a reliability problem. The code gets updated; the OpenAPI spec gets forgotten. The spec gets updated; the TypeScript client doesn't regenerate. By the time an enterprise client asks for your API contract, the document you hand them describes a system that no longer exists. Baklava, an open-source library by Iterators , solves this structurally: documentation is generated from the tests that verify your actual API behaviour, so it cannot drift. The problem Documentation drift is the default state of any API that lives long enough. The causes are well-understood: docs and code are maintained separately, documentation updates require extra discipline at every PR, and no automated check catches a route signature change that wasn't reflected in the OpenAPI file. The consequence is real. Clients building against a stale spec hit integration errors in production. Internal teams onboarding to a service spend hours reconciling the documented contract with actual behaviour. TypeScript front-ends break when an API response field changes without a corresponding client update. The problem compounds as the API grows. The solution Baklava integrates into your existing test suite. When routing tests run, baklava observes each request and response, infers the API surface, and generates documentation as a test output, not as a separate build step, not as a manually-maintained file. In baklava, the test is the documentation spec. Instead of a standard assertion block, each route is defined with path() , supports() , and onRequest() scenarios that both verify the API behaviour and describe it for documentation output: ​`// The test IS the documentation spec class UserApiSpec extends AnyFunSpec with BaklavaPekkoHttp[Unit, Unit, ScalatestAsExecution] with BaklavaScalatest[Route, ToEntityMarshaller, FromEntityUnmarshaller] { path("/users/{userId}")( supports( GET, pathParameters = p Long , summary = "Get user by ID" )( onRequest(pathParameters = 1L) .respondsWith Use

Free AI Tiers Bill You in Hours, Not Dollars Free model access looks like a bargain until you track the hours you spend feeding context back into a model with no memory. A zero-cost invoice hides the most expensive resource in your workflow: your own attention. My position is straightforward: treat a free tier like a metered service and measure the hidden costs before you adopt it. The token counter tells you almost nothing about the real price. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm using MonkeyCode's free model access and free server option as a concrete example; the measurement approach applies to any free tier. The dashboard shows tokens, not time Every free plan advertises a generous token allowance and a server that wakes up on demand. What the marketing page omits is the labor you spend reassembling context, waiting for cold starts, and double-checking output. Those costs do not appear on any invoice, but they consume your day in chunks. Four of them matter more than the token meter. Context reconstruction — Every new conversation starts from zero, so you re-explain your stack, your file layout, and your constraints. Those re-pasted tokens count against the same allowance you were trying to save. Cold-start waiting — A free server that sleeps after idle adds seconds to every call. Multiply that by a scheduled job that fires hourly and you have lost real time. Human verification — Confident output still needs a human to check it, and that check is the most expensive line item in the whole system. Attention fragmentation — A free allowance looks huge until you split it across codegen, debugging, and review. Small tasks nibble the budget faster than big ones. A ten-minute audit script The script below turns the argument into a reproducible measurement. It sends three representative prompts to any OpenAI-compatible endpoint, records wall-clock latency, and extracts token usage from the response. Run it several times du

Here's the conclusion up front: a free LLM endpoint is a moving target. You can't see the changes, but they're happening — model updates, quantization tweaks, server-side prompt rewrites. And your app will feel them, usually as a slow, invisible quality dip. I've spent weeks on this account probing free LLM servers, caching tokens, and building evaluation harnesses. The pattern I keep seeing: teams pick a free tier, wire it in, and then never look at it again. They treat it like a static API. It isn't. The fix is a nightly drift check. A small script that runs your most important prompts against the endpoint, compares the outputs to a baseline, and tells you when something changed. Not a benchmark. Not a one-time eval. A recurring alarm. This post walks through a 90-line harness you can run tonight. I'll use MonkeyCode's free server as the reference endpoint — it's an open-source project with free model access, a free server option, and, as advertised at the time of writing, a 10M token grant. The exact numbers may move, so check the repo's README before you depend on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Why drift is the silent killer of free-tier apps Let's be honest: free endpoints don't come with changelogs. The provider can swap the underlying model, adjust the temperature default, or add a safety filter without telling you. Your tests still pass. Your error rate stays flat. But the responses get a little shorter, a little more evasive, a little less useful. Users notice before you do. They don't file bugs for 'the bot got dumber.' They just stop using it. A drift check turns 'the bot got dumber' into a concrete signal: 'the pass rate on 12 core prompts dropped from 92% to 74% overnight.' That's something you can act on. Step 1: Define your core prompts Don't test everything. Pick 10-20 prompts that represent the actual workload your app handles. For each prompt, define what 'good' looks like. Prompt Expected beha

First post in a build-in-public series about verdict , an MCP server that gives coding agents structured, sandboxed test feedback. The problem Watch a coding agent work and you'll see it run pytest in your shell, unsandboxed, and then push 40,000 tokens of raw output through its context window to answer one question: did my change break anything? That's three problems in one command: Token waste. The agent needs ~10 lines of signal and pays for a wall of dots, warnings, and tracebacks. No sandbox. The tests run on your machine, in your environment, with your files writable. No memory. When a test fails, the agent can't tell whether it broke it or whether it was broken before it arrived - so it either "fixes" pre-existing failures nobody asked about, or ships regressions it assumes were already there. verdict is an MCP server that replaces the pytest shell-out with four tools: tool what it returns verify(scope?) impact-selected tests, run in an ephemeral container, as a ~400-token typed verdict explain_failure(check_id) the full traceback - only on demand history(fingerprint) first seen / last seen / times seen for a failure run_checks(["ruff","mypy"]) lint & type checks, same verdict shape ▶️ Watch the 30-second demo - Claude Code fixing a bug with verdict verifying in a container. The three ideas 1. Verdicts, not output. verify returns typed JSON: counts, per-failure message + location, and nothing else. Full tracebacks live behind explain_failure . The whole verdict for a real failing run is ~400 tokens - the raw pytest output it replaces was ~40k. The design rule in the repo is blunt: nothing bulky rides in the summary, ever. 2. Fingerprints give failures identity. Every failure is hashed from its normalized signature - volatile tokens (addresses, tmp paths, ids, durations) collapsed first. Same logical failure ⇒ same fingerprint, across runs and refactors. Fingerprints are what make the third idea possible: 3. History answers "was it me?" verdict keeps a small S

Originally published at ai.bedvibe.studio . I built a text-to-speech product and kept getting burned by the same thing. On normal sentences the model sounded great. Then it would hit a number, a date, an acronym or a name, and quietly mangle it. Worse, the metric everyone reaches for — Word Error Rate — was lying to me in both directions. It flagged perfectly good audio as broken because the script said 3:30 PM and the transcript said "three thirty pee em." And it missed real failures on short tokens, where the speech recogniser is as unreliable as the TTS. So I wrote the QA framework I wished I had, packaged it as ttsproof , and then ran it as a blind study against a production TTS service so the results would be more than an opinion. The two failures WER cannot see A TTS pipeline breaks in two different ways, and a single WER number blurs both. Structural defects. The clip is empty, truncated, three times too long, stuck in a repeated-chunk loop, clipping, or has a click at the tail. These have nothing to do with pronunciation — you can catch them with no model at all, straight from the waveform. Pronunciation and content errors on the hard cases: numbers, decimals, dates, clock times, acronyms, single letters, URLs, names. ttsproof splits them apart and handles each one honestly: Structural checks, no model needed — empty or truncated audio, duration explosions, long internal silences, clipping, loop detection, end-of-clip artifacts. numpy and soundfile, nothing else. Equivalence-aware WER/CER — the expected text and the ASR transcript are both canonicalised to spoken form before scoring, so 3:30 PM against "three thirty" stops counting as an error. ASR-uncertainty quarantine — when the audio is structurally clean but the recogniser disagrees on a very short utterance, the sample is set aside for a human instead of being auto-failed. At that length the ASR is as likely to be wrong as the TTS. The study: 390 samples, and a blind human check I evaluated the method

I once collected about 22,000 comments from roughly 140 Korean YouTube videos about AI coding tools and classified them. (Quotes below are translated from Korean.) I wanted to see what people were asking. What came out was something else. What the videos teach Put the titles and tags of those 140 videos in one pile and they say: How to install. How to get started. How to build an app. Which tool is best. All of it is "starting." Follow along, a result appears on the screen, the video ends. What the comments say The comments sweeping up the likes were telling a different story. "Verifying AI mistakes takes so much time. Checking every answer for nonsense got so tiring I just do the work myself now." (👍598) "Coding with AI makes me anxious. If one bug ships, I'm the one responsible. Checking and debugging everything one by one ends up being more work." (👍265) "I pay every month and it lies about work matters like it's nothing." (👍72) "Tokens burn too fast… added $50 and it was gone in half a day." (👍30) It compresses into three complaints: expensive, can't trust it, can't fix it. The videos teach the start. The people are dying right after the start. The scariest comment "Asked it for shampoo recommendations and it recommended one that doesn't exist. Slipped it in between real products — with the weight, the benefits, even a price." (👍49) That comment is the essence of the problem. When AI is wrong, it doesn't look wrong. The fake sits among the real ones, wearing plausible numbers. This is why "just write better prompts" is half an answer. Better prompts lower the odds of being wrong. They don't create a way to know when it's wrong. Drop the error rate from 10% to 3% and you still don't know where the 3% is hiding. If that 3% detonates inside payment logic, money leaves the building. One more finding — where the real questions live While collecting, I noticed the nature of comments changes with channel size. multi-million-sub videos real questions/needs = 12% of comm

A new arXiv paper watched coding agents get denied the facts they needed. They did not stop. They invented. On August 17th, a group of researchers posted a paper to arXiv with an unglamorous title and a genuinely unsettling core finding. The paper is "The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks" (arXiv:2608.16630), by Bardia Mohammadi, Lars Klein, Aman Chadha, Akhil Arora, and Laurent Bindschaedler. Before going further, one honesty note that will hold for this whole piece: I have read the paper's abstract, not its full text, and every quotation below comes from that abstract. It is enough for what this essay is about, because what this essay is about is one sentence. The setup first. The authors model repository-scale coding as reconstructing a web of coupled facts. Every edit an agent makes needs certain facts, and each fact arrives through one of two channels: it is either in the recent context, or it is in the model's memorized knowledge. Facts covered by neither channel are what the authors call coherence debt. Their experiment supplies and withholds each channel deliberately, injecting faults across "seven models and five harnesses" (abstract), and then watches what the agents do when a needed fact simply is not there. The comfortable prediction is that a competent agent, denied a fact, stops and says so. Here is what the authors report instead: "A missing fact produces wrong work rather than absent work" (abstract). The agent asked to act, acts. In the paper's words, "an agent asked to act acts, fabricating the file or guessing the value" (abstract). That much is alarming in a familiar way. Everyone who works with these systems has a story about an invented function or a guessed constant. The abstract has sharper findings than the fabrication itself, though. When the researchers renamed a real library to defeat memorized knowledge, the failure was collective: "all seven fail in the same place, passing and missing the same tests

I wanted to disagree with 'AI made me a worse reviewer' from Michael Amachree (@dev_michael) ....

Originally published on hexisteme notes . A user complaint sent me into a caption pipeline: "the subtitles cut to two words in places where the sentence doesn't make sense." The fix I shipped for that complaint introduced a second bug, one word narrower and easy to miss, because the code that measured whether a line of text would fit reproduced an assumption about the text that the code drawing the line didn't share. Every test passed the whole time. I only found it by watching the rendered video. The bug the complaint pointed at The captioning system splits a transcript into short chunks that pop onto screen a few words at a time. The chunking function was doing fixed-size slicing — take the next N words, regardless of what came before or after. That's blind to sentence boundaries, so two unrelated sentences could land in the same chunk: loss. Today reads as one visual unit even though it's the tail of one sentence and the head of the next. The fix was a rule set, not a single tweak: hard break after terminal punctuation ( . ! ? … ) soft break at commas, semicolons, and em-dashes extend or push a chunk rather than let it end on a function word ( of , the , than , is , and about thirty others) target three words per chunk, four as a ceiling a pixel-width cap on the rendered chunk, measured against the actual caption font (Montserrat ExtraBold), with a budget of 1080 × 0.92 = 993.6px The first four rules are about where a line is allowed to break. The fifth is a physical constraint: however good the break points are, a chunk still has to fit on screen at the font size actually in use. That's the one that went wrong. What the width guard actually measured To get the pixel width of a candidate chunk, the guard rendered the chunk's text through the font and measured the result — which is the correct approach in principle, not a shortcut. Text width isn't a fixed number of pixels per character; it depends on the specific glyphs, so measuring the real string through the r

Background I run a fleet of static sites that publish new content every day, mostly unattended. One of the house style rules is simple: no emoji anywhere in our own copy. That rule is impossible to hold by hand. A single site builds a few hundred HTML files, and emoji can slip into nav icons, button labels, <title> , the RSS feed, or JSON-LD (the JSON-formatted metadata embedded in a page to describe its structure to search engines). Nobody is going to review all of that before every deploy. So I wrote emoji-lint , a check that exits 1 the moment it finds a single emoji . It sits in the pre-deploy gate, which means a failure stops that day's publish. This post is not about the regex. It's about what happens when you put a failing check into real operation: you immediately discover the places where the rule must not apply. How it works The core is unremarkable. A regex holds the emoji code point ranges, the scanner walks each file line by line, and matching lines are reported as JSON. const EMOJI_RE = / [\u {1F000}- \u {1FAFF} \u {2600}- \u {27BF} \u {2B00}- \u {2BFF} \u {1F1E6}- \u {1F1FF} \u {FE0F} \u {200D} \u {2049} \u {203C} \u {2122} \u {2139} ] /u ; \u{FE0F} (variation selector) and \u{200D} (ZWJ) are in there because emoji are not always a single code point. Arrows and similar symbols used in ordinary technical writing are deliberately left out. Catch everything and the check drowns in false positives, at which point people stop reading it. The interesting part came later. Three categories of content look exactly like a violation but must not be treated as one: Verbatim quotes from other people Real proper nouns whose official spelling contains a symbol Passages where the emoji itself is the subject being explained Delete the emoji in any of those and you break something more important than the style rule. One term up front: "masking" here means replacing a range with spaces so the scanner cannot see it. Nothing is deleted from the file. Implementation Scope

This is a submission for DEV's Summer Bug Smash: Clear the Lineup , powered by Sentry . Project Overview I was working on a small Python component that performs a preflight check and then, if the check succeeds, invokes one synchronous operation callback. A counter records whether that callback invocation returned normally. The counter is used for diagnostics, so it must follow the control flow rather than the expected happy path. Bug Fix or Performance Improvement When a handled failure occurred, the old implementation still returned one: return 1 That value was hard-coded because the successful path was expected to invoke exactly one operation. If the preflight check failed, however, the operation was never entered and the function still returned one. An offline reproduction produced: operation_entries=0 old_count=1 The failure was handled, but the counter contradicted the actual control flow. Code Reduced to the relevant lines, the old behavior was: # Simplified pre-fix behavior def buggy_completed_calls ( * , preflight , operation ): try : preflight () operation () except Exception : pass return 1 Here is the complete fixed function from the standalone reproducer: from collections.abc import Callable Callback = Callable [[], None ] def completed_calls ( * , preflight : Callback , operation : Callback ) -> int : """ Return one only when the cooperative operation returned normally. """ try : preflight () operation () except Exception : return 0 return 1 The essential regression assertion is shown below. Both callbacks are local, so the test performs no network request: # Abbreviated test excerpt def test_preflight_failure_does_not_count_an_unentered_operation (): operation_entries = 0 def refuse_preflight (): raise RuntimeError ( " controlled preflight refusal " ) def operation (): nonlocal operation_entries operation_entries += 1 result = completed_calls ( preflight = refuse_preflight , operation = operation , ) assert operation_entries == 0 assert result == 0 My

In June a reviewer on DEV who goes by @anp2network told me to stop storing a conclusion. I had a...

Opinion: Your Tests Can't See What a Migration Destroys — Dry-Run It on a Clone A green test suite is the wrong tool for judging an AI-generated migration, because tests run against the post-migration schema and never observe the intermediate states where data disappears. The up migration is the visible artifact that gets reviewed, while the down migration is treated as an afterthought even though it is the only safety net when the deployment goes wrong. Free model access makes the problem structural: generation cost drops to zero, so migration volume rises, and every additional migration multiplies the surface for unreviewed data loss. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Tests validate the destination, not the journey When a test suite runs against a migrated database, it confirms that the application can read the new schema, but it cannot confirm that the migration preserved the data it was supposed to preserve. The test runner connects after the migration has executed, so it never sees the moment when a column is dropped, a table is renamed, or a constraint is silently relaxed. A migration that passes every test can still destroy production data, because the tests were designed to validate application behavior, not migration safety. The standard mitigation is a staging database, but staging is a poor substitute for a dry run because it has different data, different volume, and different usage patterns. The dry run I recommend uses a clone of the production schema with a representative data sample, and it exercises both directions of the migration with data integrity checks at every step. The clone does not need to be large; a few thousand rows per table is enough to expose most destructive patterns. The dry-run workflow in five steps The workflow is deliberately mechanical, because the goal is to remove judgment from the verification process and reserve human attention for the migration's intent: Clone the schema and lo

This is a submission for DEV's Summer Bug Smash : Smash Stories. TL;DR. I bounded a database read to make my analyzer faster. I derived the bound carefully, wrote the reasoning into the KDoc, and shipped it behind five passing tests. The bound was wrong in a way none of those tests could see. The result: if a lifter deloaded once in the middle of a stall, which is the correct thing for a lifter to do, my app stopped telling them they had plateaued. No crash. No error. No log line. The feature just quietly stopped being true for the people using the app correctly. The setup WhyRep analyzes your training rather than just recording it. The core promise is that it tells you when you have stalled and what to change about it, and that every verdict traces back to a methodology document rather than to something a language model made up. The architecture decision underneath that promise is that nothing is precomputed . Verdicts are derived from raw set logs on read, every time, so there is no cached judgement to go stale when the rules change. Which means every read walked the lifter's entire history for every exercise in the session. That is fine at ten sessions. It is not fine at three hundred. The obvious optimization is to bound the read. The obvious bound is "it only needs the last two weeks." That was my first wrong answer, and it is worth thirty seconds before I get to the interesting one. The plateau rules are not measured in calendar time. They are consecutive-miss counts, and the count varies by lifter tier and by whether the movement is a big or small joint action. The widest window in the signed methodology is an elite lifter on a small joint action: 14 consecutive sessions without progress. Train a lateral raise once a week and 14 sessions is over three months of data. A 14-day cutoff could never have fired a plateau for anyone above beginner tier. It would not have thrown. It would have quietly stopped detecting the exact thing the product exists to detect. Th