标签：#Testing

I have been building machin for a while — a Go-flavored, type-inferred language that compiles through C to a single native binary. It has grown a lot recently, and I wanted to answer the obvious question honestly: does it beat Rust and Zig at anything? It does, at two things, decisively. But the first thing I found was not a win. It was my own benchmark quietly lying to me, and the number it was lying about was the best one I had. The benchmark was measuring the order I ran things in machin's repo has had a bench/native-speed suite for months: four compute kernels — recursive fib, a mandelbrot, a sieve, a big integer loop — written in machin, Rust and Zig, producing byte-identical output, so the timing compares the same computation three ways. The published result claimed machin won the integer loop by 20-25% . That claim also shipped inside machin guide , which is what every coding agent reads to learn the language. When I re-ran it, the margin was gone. Not shrunk — gone. So I read the harness instead of the output: for kernel in kernels : for lang in [ machin , rust , zig ]: for _ in range ( 5 ): # all 5 machin, THEN all 5 rust, THEN all 5 zig time ( binary ) It ran every sample of one language before starting the next. On a laptop that heats up and down-clocks during a three-second kernel, that does not measure the languages. It measures who had the misfortune of running last . Zig always went last. Zig always looked slowest. The fix is four lines — interleave the rounds, rotate who starts each one. Here is what my headline number did: intsum 10^9 before (blocked) after (interleaved) machin 2832 ms 3079.7 ms rust 3764 ms 3223.8 ms zig 3556 ms 3189.7 ms "machin +20-25%" machin +3% = a TIE A 20-25% win became a tie. I deleted the claim from the README and from machin guide . The harness now also refuses to declare a winner inside a 3% band, because the worst run-to-run spread I measured was 41% of the min sample. Calling winners inside that is how benchmarks start

The most consequential bug in this project had been there since the beginning, survived several full end-to-end runs, and was reported as a PASS every time. ✓ PASS slow-and-low detected within 30m (7.3m), never exceeding legit rate That line is true. The scorer flagged it correctly, well inside the bound. What the line doesn't say is that the attacker was served every single request it ever made . Zero non-allow decisions, across the entire scenario. Detected and never once stopped. This is one instance of a pattern that accounts for more real bugs in this project than every other cause combined: a component contributes nothing, no error is raised, and every surrounding number stays plausible. The bug The scorer computes windows at 1m, 5m and 1h, and publishes each result to a per-client key in Redis and OPA. Each result. To the same key. So the last writer won. And 1-minute windows close most often, so they always won. slow-and-low issues about two requests a minute. Its 1m windows fall below the minimum request count and score zero. Its 5m and 1h windows accumulate the miss ratio that earns a deny . Every one of those zeroes immediately overwrote the deny. The entire premise of a multi-scale pipeline — that different attacks are visible at different scales — was silently violated by the publication step. Any detection that only appeared at a coarser scale was discarded. The fix is a roll-up: publish the most severe verdict across window sizes within the freshness horizon the policy already uses. Afterwards, the same attacker is denied on 24–31 of its 44 requests. Why it survived so long Because the report could not express it. Detection latency was computed as the earlier of two very different facts: the scorer's first non-allow window, and the gateway's first non-allow decision. Printed under one heading — detected — a client that was noticed but never touched looked identical to one that was noticed and blocked. A report that averages over the distinction you ar

A few weeks ago I wrote about building a reproducible test harness for comparing free AI coding models before you commit . That harness answers one question: which model should I use? It does not answer the harder follow-up: once a model generates a patch for my real codebase, when is it safe to merge? This week there was a great discussion on DEV about "understanding over origin" — the idea that it doesn't matter whether code came from a human or a model, only whether someone actually understands it. I agree with the principle, but principles don't survive contact with a busy afternoon. What survives is a checklist with teeth. So here is the pipeline I bolted onto my model harness: every AI-generated patch has to pass through a scripted review gate before I even read it, and the script produces a scorecard that tells me how carefully I need to read it. The problem with eyeballing diffs When a model produces a 40-line diff that looks idiomatic, my brain does a dangerous thing: it pattern-matches on style and skips semantics. The code reads like something I'd write, so I approve it like something I'd write. The failures I've actually shipped from AI-generated code were never syntax errors — the tests even passed. They were things like: A retry loop that retried on the wrong exception type, so real errors got swallowed. A query filter that was subtly wider than the one it replaced (tests passed because fixtures were too small to notice). A dependency added for a one-liner the standard library already covers. All three would have been caught by asking four boring questions before reading the code. So I scripted the questions. The review gate: a reproducible artifact The gate is a small shell script. It takes a patch file, applies it to a throwaway worktree, and runs four checks. It never touches my working branch, and it prints a one-line verdict at the end. #!/usr/bin/env bash # review-gate.sh <patch-file> <base-branch> set -euo pipefail PATCH = " $1 " BASE = " ${ 2 :

I had a podcast pipeline I was proud of. It took a transcript, turned it into a two-person conversation with text-to-speech, laid in the music, and produced an MP3 I could publish. I'd built it in one app, and it worked. I loved the output. So when I started a second app that needed the same flow, I didn't want to rebuild the pipeline. I already had one. I just wanted it over there. So I asked the AI to copy it. And it did. Here's the part that matters: I didn't just copy it and hope. I checked. I opened a fresh session (a clean one, no memory of the first) and told it to look at the new pipeline and make sure everything was right. It went and looked. It came back and told me everything was good. Everything looked good. Or so I was told. Then I loaded the first real transcript and ran it. It was wrong. Not a little wrong. The voices were wrong. The music didn't come in when it was supposed to. It didn't cut off when it was supposed to. It didn't fade. It just stopped. The words were all there, every one of them, in the right order. But everything that made the first pipeline good (the timing, the production, the feel) was gone. I walked away from my desk for a bit. It pissed me off, because I'd done what I was supposed to do. I'd asked. It had answered. The check was green. And the check was a lie. Here's what I think I actually got wrong, and it's not "I trusted the AI." It's subtler than that. When I asked a fresh session to "make sure everything's good," I got back a confident yes. But the session had no way of knowing what good sounded like. It never heard the first pipeline. It had no stake in whether the podcast was any good. It reported what it could see (the code looked reasonable) and what it could see was almost never the thing I actually cared about. That's the trap, and it isn't a beginner's trap. I have a whole process built to avoid exactly this: spec, adversarial review, a plan, a build, a code review. And I skipped it, on a task I decided was too sma

The last lesson was about validating what a model hands you. The story behind it: a set of prompts that had returned real, criteria-matched vendors for weeks came back in staging with placeholder junk, literally the words Vendor A, Vendor B, Vendor C. So I built the validation layer, and the last gate in it is a model checking a model. Then FromZeroToShip asked three questions in the comments, and all three were about the gate rather than the model. That's the harder thing to look at, and I hadn't written all of it down. Here's the long version. What was on the fail list that I hadn't already been burned by? More than the question assumes, and not because I got clever about imagining failures. The placeholder output changed what I do with a failure . I stopped fixing the instance and asked what class it belonged to, and that class is a lot wider than "the model emitted example data." It's a suggestion that looks fine and isn't usable. Two of those I had never hit went in on the back of it: A vendor that's wrong for the category. A vendor that's no longer in business. Neither has anything to do with placeholder text, and both would sail through a schema check looking like a perfectly real answer. They also changed the prompt that produces the suggestions, not just the gate. Fixing only the failure I actually met would have left both of them live. So the list isn't purely retrospective. It grows by generalizing from the one failure you hit to the class it sits in, and it keeps growing from what the running system actually throws at me rather than from what I remembered to imagine. Is it foolproof? No. What's left is the case worth worrying about: results that read as real, pass the schema, satisfy every criterion I gave, and are still wrong. You can't validate the truth of a guess from inside the system. You can only lower the cost of it being wrong. That means a human in the loop at the stage where being wrong is expensive, the confidence surfaced so the answer is ch

We had a checkout test at my last job that everyone called "the coin flip." Green for a week, red twice on a Tuesday, green again. Someone eventually wrapped it in a retry and it sat like that for eight months before anyone looked at it again. Turned out the real bug was a webhook that occasionally fired before the order record finished writing to the DB - a two-hundred-millisecond gap that only showed up under load. The test wasn't broken. It was the only thing in the entire pipeline that noticed. That's usually the story. Someone blames the test - bad selector, missing wait, a sleep(2) some intern left in there three years ago, and half the time they're right. But when a test flakes repeatedly and nobody can explain why, the test is rarely the actual problem. It's just the part of the system rude enough to say something. A few places I keep finding the real cause hiding. Tests that quietly depend on each other Test A writes a row, Test B reads it and never knew it needed to. Run B by itself, it passes. Run the suite in a different order, or in parallel, and B fails for no reason anyone can point to. I've lost a full afternoon to this exact thing more than once - a cache value from Test 12 leaking into Test 47. The actual fix is annoying and unglamorous: every test gets its own fixtures, its own scoped data, no assumptions about what ran before it. If your suite only goes green in one specific order, you don't have a flaky test. You have an undocumented dependency graph, and it's going to bite someone eventually. The app is racing, not the test Click a button, immediately assert on the result - that's a bet that the UI update lands the instant the click handler returns. It usually does, on your machine, on a good day. Add a debounce, a background job, or just enough network latency and that bet stops paying off. This one's frustrating because the test isn't being paranoid. The app genuinely has a race condition. The test just runs the interaction often enough, acro

Lets start with a bit of back story. I am a full stack developer. Developer being the keyword here, not a QA developer. But in my current role, I was recently asked to come up with a testing suite for the web application and the Flutter app I was managing and maintaining. At that time, I didn’t have anything better to do and thought this would be a fun little project to work on for a couple of weeks. Boy o boy, I was wrong. People in QA are so opinionated. Everyone has their preferred framework, structure, naming convention, abstraction, folder structure and a very strong opinion about why your approach is wrong. Starting with the industry best practices I started by trying to follow the trends and best practices used in the industry. Page Object Models, reusable helpers, proper assertions and all the usual bits and bobs. For the web application, which was built with React, I chose Playwright. For the Flutter app, I went with integration_test . Sounded simple enough. The login test that took three hours The first test I tried to write was a simple login flow. Open the application Enter the username and password Press the login button Wait for the dashboard Easy, right? It took me ages. And by ages, I mean roughly three hours just to get the web test to pass reliably. The actual Playwright test ended up being around 300 lines once I included the boilerplate, setup, selectors, assertions, waits, Page Object Model structure and everything else needed around the actual journey. Then came the Flutter app. That one was worse. The app has its own custom way of starting different flavors, and both the web application and Flutter app are white-labelled products. That means there are a lot of variations to cover. Different branding, configurations, screens and sometimes slightly different user journeys. Before I could even test the login flow, I needed a pile of setup code just to launch the correct version of the app. The Flutter test eventually went beyond 500 lines, includ

I Built an Agent Evaluation Harness for Local AI — Here's What Most People Get Wrong DOYR | Not financial/legal/tax advice. For educational purposes only. Three months ago, I started building AI agents for my trading business. First agent: Fetches Nifty option chain data. Second agent: Analyzes PCR, OI, max pain. Third agent: Predicts direction using XGBoost. Fourth agent: Sends Telegram alerts. I had 4 agents doing 5 jobs. And I had no idea if they were any good . Sure, my trading results were +₹96,000 over 6 months. But was that because my agents were smart, or because I was overriding their bad decisions? I couldn't answer that question. So I built something to find out. An Agent Evaluation Harness. What Is an Agent Evaluation Harness? An Agent Evaluation Harness is a systematic framework for testing AI agents. It answers one question: "How good is this agent, actually?" Most people skip evaluation. They build an agent, test it once or twice manually, and call it "done." Then they wonder why it fails in production. An evaluation harness forces you to: Define success metrics — what does "good" mean? Create test suites — what scenarios will you test? Run evaluations — how does the agent perform across all scenarios? Measure regressions — did a change make the agent worse? Track improvements — is version 2 better than version 1? This is not optional. This is engineering 101 . Why Most Agent Evaluations Are Wrong I reviewed 50+ "agent evaluation" frameworks online. Here's what I found: Mistake 1: Single-Task Testing What they do: Test the agent on one task. "Can it book a flight?" → Yes/No. What's wrong: Real agents face thousands of variations of the same task. "Book a flight from Delhi to Mumbai on Friday" vs "Book a flight from Delhi to Mumbai next Friday" vs "Book a flight from Delhi to Mumbai on August 15th." A good harness tests variations , not just one example. Mistake 2: No Edge Cases What they do: Test happy paths only. "Book a flight when everything works.

Most "which AI coding model is best?" debates I see devolve into vibes. Someone pastes a cherry-picked diff, someone else counters with a different cherry-picked diff, and nobody learns anything transferable. The problem isn't the models — it's that we almost never evaluate them on our code, with our constraints, using a method we could rerun tomorrow. This article is the harness I wish more teams built before arguing. It's a small, language-agnostic evaluation loop you can point at any model you have access to — including free tiers — and get a defensible answer to a narrow question: does this model help with the tasks I actually do? The evaluation trap Public benchmarks (HumanEval-style tasks, leaderboard scores) measure performance on curated problems with clean specifications. Your work is rarely that. Real tasks look like: "Add retry logic to this half-migrated HTTP client without breaking the old call sites." "Write tests for a function whose behavior depends on a config file three directories up." "Refactor this 200-line function, but the ORM calls must stay in the same transaction." These tasks share a trait: correctness is checkable, but only by you . Your test suite, your type checker, your lint rules. That's actually good news — it means evaluation can be automated against artifacts you already have. The artifact: a task-runner harness The core idea is dumb on purpose. Define a set of tasks as directories. Each task has a prompt, a snapshot of the relevant code, and a verification command. The harness applies a model's patch and runs the verifier. No scoring model, no LLM-as-judge — just your own build. eval/ ├── tasks/ │ ├── 001-retry-http-client/ │ │ ├── prompt.md │ │ ├── repo/ # snapshot of the relevant files │ │ └── verify.sh # exit 0 = pass │ ├── 002-test-config-loader/ │ └── 003-split-billing-fn/ └── run_eval.py Here's a minimal runner (Python 3.10+, stdlib only): #!/usr/bin/env python3 """ run_eval.py — apply a model-produced patch to each task and

If you've experimented with AI-driven testing, you've probably lived this cycle: you ask an AI agent to "test the checkout flow," and it does something — clicks around, declares success, and leaves you unsure what was actually verified. The next day you ask again and it does something different. The browser automation works; the testing discipline is missing. That gap is what Snagly is for. Rather than describe it, I pointed it at softwaretestingtrends.com — my own production site, nothing fixed beforehand — and recorded the whole thing. It found eleven issues, including a critical accessibility bug on my own signup page. One of its findings turned out to be wrong, and I'll come back to that, because it matters more than the ones it got right. 📺 Watch the full walkthrough — installed from an empty folder, run against production, ~20 minutes. What it is Snagly is a free, MIT-licensed set of 30 skills for AI coding agents — GitHub Copilot , Claude Code , Cursor, Codex and 70+ others — that turn "an AI that can drive a browser" into "an AI that tests like a QA professional." A skill, if you haven't met them yet, is a reusable instruction set that teaches the agent a specific working method — when to use it, what rigor it requires, what evidence to capture, and what it must never do. Each skill in Snagly has one job, and they hand off to each other the way a real testing practice does: start-testing is the front door — say "what can you test here?" and it routes you to the right skill, checking prerequisites before handing off. Discovery & strategy : scenario-mapper explores your site and produces a prioritized list of test scenarios; test-case-writer expands any of them into a reviewable spec; test-plan sets strategy, cadence, and release exit criteria; qa-onboarding writes the guide for your next hire. Execution : flow-runner drives real user journeys step by step, asserting outcomes (not just that clicks happened) and capturing evidence the moment anything fails. cru

I have been a huge Storybook and Chromatic fan for years. But at some point the bill got my attention, and when I looked into why, the fix turned out to be simple. This is the write-up of what I changed. It works on any per-snapshot tool, not just Chromatic. First, some backstory on how I got here, because it explains why the cost crept up in the first place. How I ended up paying for a lot of snapshots In the past I would build a gigantic end-to-end pipeline that was flaky as hell and made me spend time every week fixing it. It took 40 minutes to run, and when it went red someone would assume it was just flaky, merge the change anyway, and then find out it truly did break the system. So I stopped writing lots of E2Es and moved to Storybook for interaction and visual testing. Much better. But because I was rendering every state of every component as its own story to get the screenshots in place, I was generating a lot of screenshots. And every snapshot tool, Chromatic, Percy, Playwright screenshots, UI Verify, renders and bills per story. So the number of stories is the cost, and it is also the noise surface: more stories means more places for a diff to flake. I ended up paying a lot, which made me think about whether there were ways to optimise it. There were. Here they are. The core idea: combine states into one story The naive pattern is one story per variant times state times theme. A component with 5 sizes, 3 states, and 2 themes is 30 snapshots the naive way. The whole idea below is to collapse that matrix into a handful of stories while keeping full coverage. Move 1: one gallery story, not N stories For something like a Button, there is no need to have separate Primary, Secondary, and Tertiary stories. I prefer one AllVariants story that maps through the prop combinations and renders them in a grid. One snapshot then covers the entire matrix. As a bonus you get a nice grid that shows every permutation at a glance, with no extra clicks to see the variations. /

How EvalPort's Grader System Works When designing EvalPort, the grader system was the hardest part to get right. Every eval framework has its own way of scoring LLM outputs — DeepEval uses metric classes, Promptfoo uses assertion objects, Inspect AI uses solver functions. We needed a system expressive enough to cover 90%+ of real-world eval needs, but simple enough that any framework could implement it. The result: 11 grader types that carry their own semantics. A grader isn't just a name — it specifies its parameters, its model, its threshold. An eval suite is self-describing. The 11 Grader Types exact_match — Compare output to expected output, optionally ignoring case. contains — Check if the output contains a substring. regex — Match against a regular expression. semantic_similarity — Embed output and expected output, compare cosine similarity against a threshold. llm_judge — Use an LLM to evaluate the output against a prompt template. The most powerful grader. json_schema — Validate that the output is valid JSON matching a JSON Schema. json_path — Extract a value from JSON output using a JSONPath expression, then compare it. code — Run a function to evaluate the output. human — Defer to human review. model_graded — Compare the output to a reference answer using a model. custom — Escape hatch for graders not covered by built-in types. How Graders Connect to Test Cases A test case references graders by ID. Multiple graders can evaluate the same test case. The ResultSet records each grader's score separately. Why This Design Works Self-describing: An eval suite carries everything a framework needs to execute it. Framework-agnostic: Any framework can implement any subset of grader types. Extensible: The custom type lets frameworks bring their own graders. Comparable: Results from different frameworks use the same grader IDs. Try It pip install evalport-sdk npm install evalport-sdk Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md Repo: https://gith

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. Did anyone ask for...

I started this project because an LLM annoyed me. I gave a very strong model 322 , a small Dota 2 drafting game. The choices looked like the kind of work a computer should enjoy: repeated packs of players and heroes, visible ratings, familiarity scores, chemistry, rerolls and a simulated tournament at the end. I was disappointed by how well the LLM did. I am not a Dota expert, and I had only started watching it occasionally again during the previous six months or year. I still seemed to be doing better. The interesting engineering question was not how to write a longer prompt. It was how to replace the card-by-card language-model judgement with a deterministic policy, then test that policy without confusing improvement with luck. A stochastic benchmark needs shared randomness The browser history gave us a useful irritation and almost no reliable comparison. My earlier manual record contained 50 runs with a 14% title rate. The LLM won once in nine attempts. Putting 14% beside 11% looks temptingly quantitative, but the random offers, rejected packs and opponent fields were not preserved. The samples were small, unpaired and produced under different choices. That is not a model benchmark. It is a reason to build one. The offline solver generated every random choice from indexed tapes. Policy A and policy B received the same player offers, hero samples, field candidates and tournament randomness for a given episode. We could then compare the paired result: did the new policy win this exact episode where the old policy lost it? This is the common-random-numbers idea in a practical form. Sharing the luck removes a large amount of noise that has nothing to do with the policy change. Keep the simulator separate from the policy Before evaluating a strategy, we reproduced the game. The public client and seven data files were frozen with SHA-256 hashes. Draft legality, automatic hero allocation, chemistry, scoring and the tournament were ported into a deterministic Python engi

Prompts rot silently. I swap a model, tweak a system prompt, add a tool, and everything still runs. No exception is thrown, no test goes red, the JSON still parses. The output is just quietly worse, and I usually find out from a user rather than from CI. Unit tests are the wrong instrument here because there is nothing to catch: the failure mode is not a crash, it is a drop in quality. So I built evalgate , a small TypeScript tool that treats prompt and agent quality like a build artifact. You write a declarative eval suite, evalgate runs it, scores it, stores a baseline, and on every pull request it re-runs the suite, computes the quality delta against the base branch, and fails the build when the score regresses. Then it posts the delta table as a PR comment. The core idea The important design decision is what question CI is allowed to ask. "Is this prompt good?" is subjective and unwinnable in an automated gate. "Is this worse than it was on main?" is objective and answerable. evalgate is built around that second question. You capture a baseline once, and from then on every change is judged as a delta against it, not against some absolute notion of goodness. The second decision was that the whole thing has to run with zero API keys. evalgate ships a deterministic mock provider, so you can run a suite, save a baseline, compare runs, and execute the full test suite completely offline. The project itself has 67 tests and none of them touch the network. Every feature has to work in mock mode before it counts as done. How it works A suite is a YAML (or JSON) file that lives in version control next to the code it checks. Each case has an input, an expected reference value, and one or more scorers. Here is a minimal one: name : my-agent provider : mock # works with no API key threshold : 0.9 # mean score required to pass cases : - id : greeting input : prompt : | Reply with the standard greeting. exactly: Hi there! How can I help you today? expected : " Hi there! How ca

How do you actually test an AI agent? Not "does it respond," but: does it route to the right tool, chain calls correctly, recover from failure, resist prompt injection, and stay within cost/latency budget? I spent weeks working through this on a running agent, and open-sourced the entire methodology — framework-agnostic , so it applies regardless of your language, runtime, or toolset. What's inside •⁠ ⁠ 61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena, AgentDojo, LongMemEval and more, categorized by what they actually measure •⁠ ⁠ 58 universal test blocks across 7 tiers (L1–L4, Error Recovery, Multi-Turn, Security). Each block = a tool-agnostic capability definition + a concrete reference implementation •⁠ ⁠ Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to 6 universal security test blocks •⁠ ⁠ Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k, trajectory vs end-state, observability (OpenTelemetry GenAI), automated red-teaming (garak, PyRIT, DeepTeam) •⁠ ⁠ Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001 How to use it Take Part II, replace the reference-implementation fields with your own agent's tool names and expected outputs. The universal capability definitions need no changes. Blank templates are included. PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real reference case study — but the methodology is the product, not the agent. No marketing narrative: ⁠ STORY.md ⁠ documents the real bugs, real test runs, and real corrections that shaped each version. Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome. 👉 https://github.com/trgysvc/AgentTestMethodology

I'm building an app with Claude Code right now. The setup is a little unusual. One "commander" session writes the instructions, and separate "worker" sessions implement them in parallel. The commander never touches the keyboard itself. Its whole job is to turn "what to do next" into a handoff prompt that a worker can read and just run with. And that handoff prompt is quietly the scariest thing in the whole loop. The moment I hand it over, the worker trusts it as the spec and starts sprinting. If the spec is wrong, the wrong thing gets built. Fast, and with confidence. Before handing it over, I did my usual ritual I've picked up a habit lately. Before I throw the instructions at a worker, I run them past a subagent whose entire job is to demolish them. It's defined to never approve, to hunt for holes, and to never, ever close with "this looks broadly reasonable." What was different this time: this thing didn't just read the prose of my prompt. It went and read the actual code. And the reply it came back with made my stomach drop a little. Objection 1: I wrote "make the test pass (go green)", but that test's green meant nothing In the instructions, I'd written this as a definition of done: "Get the XX test passing (green)." The demolition agent's answer: "That test goes green even when the thing it's testing fails. " I read it. It was true. The test only checks "did the process run all the way to the end." It never checks the one thing that matters: did it succeed? Fail, and as long as it "returned a failure result and finished running," green. On top of that, the batch path was swallowing exceptions, so no matter what blew up, still green. So even if a worker reported back "DoD met, tests green!", nothing was actually proven. The completion criterion I wrote myself was an empty pass. Objection 2: I wrote "just flip a flag", but that switch didn't exist One more. I'd written " flip a config flag and it swaps in the real component ", as if it were a feature that alread

Every automated system receives a particular shape of the world. That shape is expressed through records, documents, events, exceptions, and missing values. If the designers have not identified those forms—and the ways they can become malformed—the machine inherits their ignorance and reproduces it at scale. The question is not simply whether the AI failed. The useful question is whether the human-built system knew what success meant, knew the shape of its data, and knew how to recognize when it was wrong. Start with the shape of the data Before selecting a model, draw the workflow as a sequence of data transformations. What enters each stage? In what form and from what source? Which values are valid, absent, duplicated, stale, delayed, or contradictory? How will each violation be detected? What must the workflow do next? Each data shape needs a corresponding failure model. An unknown here is not merely uncertainty for the machine; it is a measurement failure in the organization. The remedy is to collect the missing data or explicitly design for its absence. Otherwise, the system is being asked to operate in a world its designers have not described. Stabilize the deliverable A system cannot be stabilized around a target that continues to move. The deliverable must be more than an aspiration written in a prompt. It should be expressed as observable conditions and anchored to a representative corpus: examples that are acceptable; examples that are unacceptable; examples that are genuinely ambiguous. Human reviewers should first demonstrate that they can apply those distinctions consistently. If they cannot agree on what success looks like, the model is not being measured against a specification. It is being measured against human disagreement disguised as one. The model is not the system Only then does it become meaningful to place an AI model inside the workflow. The model is one transformation among many: Input → validation → retrieval → normalization → model infere

I compiled the rules engine of a shipped Android game to the browser. Same Java, two compilers. Then I checked whether the two agreed. They did not — and the test I already had for exactly this had been green the whole time. The same command twice: green against the current engine, then against the committed recording of the broken build. Play it as a terminal session if you want to select the text. The setup The rules live in one module with no Android on its classpath, which is what let me compile them a second time with TeaVM and run the same logic on a canvas in a browser tab. A seeded run should be reproducible. Give the engine seed 42 and a fixed sequence of inputs, and you should get the same game every time — that is what makes a run replayable and two builds comparable. Here is what I actually got, same seed, same inputs: JVM browser first obstacle x, frame 60 405.426 304.426 still alive at frame 360 yes no final score 9 6 Not a rounding difference. A different game. The cause is boring. The test failure is not. GameEngine used java.util.Random . Its algorithm is specified down to the constants — you can read the exact linear congruential generator in the Javadoc. So a seed ought to name exactly one sequence. But my code was not running that algorithm. It was running whichever implementation the runtime supplied , and TeaVM's is not the JVM's. The specification describes what java.util.Random does; it does not force a foreign runtime's reimplementation to match. The fix took ten minutes: write the LCG out longhand so both builds execute the same arithmetic instead of trusting that they will. The interesting part is the test. The test that could not have caught it I had a test called theSameSeedProducesTheSameRun . It ran the engine twice, with the same seed, and asserted the results matched. It passed on every commit, including every commit during which the browser build was playing a different game. It had to pass. It runs the engine twice in the same runt

CompTIA PenTest+ / Ethical Hacking Certification Series Professional Reference Guide — GitHub Edition Covers: Passive Reconnaissance · OSINT · DNS · Social Media · Cryptographic Analysis · Shodan Table of Contents 3.0 Introduction 3.1 Performing Passive Reconnaissance 3.1.1 Overview 3.1.2 Active Reconnaissance vs. Passive Reconnaissance 3.1.3 The OSINT Methodology — How Professionals Think 3.1.4 OSINT Tools — The Complete Professional Arsenal 3.1.5 DNS Lookups — Deep Dive 3.1.6 DNS Reconnaissance — Advanced Techniques 3.1.7 Identification of Technical and Administrative Contacts 3.1.8 WHOIS Intelligence — Extracting Maximum Value 3.1.9 DNS Lookups — Lab-Level Practical Reference 3.1.10 Cloud vs. Self-Hosted Applications and Related Subdomains 3.1.11 Social Media Scraping 3.1.12 Employee Intelligence Gathering 3.1.13 Cryptographic Flaws 3.1.14 Finding Information from SSL Certificates 3.1.15 Company Reputation and Security Posture 3.1.16 File Metadata 3.1.17 Web Archiving, Caching, and Public Code Repositories 3.1.18 Finding Out About the Organization — Aggregation Techniques 3.1.19 Advanced Searches — Google Dorking and Beyond 3.1.20 Open-Source Intelligence (OSINT) Gathering — Frameworks and Automation 3.1.21 Shodan — The Search Engine for Everything Connected 3.1.22 Breach Data Intelligence — Leaked Credentials and Exposure Monitoring 3.0 Introduction Module Overview: Information Gathering and Vulnerability Scanning Module Objective: Perform information gathering and vulnerability scanning activities at a professional, senior-level standard. Before a single exploit is launched, before a single payload is crafted, every professional penetration tester invests significant time in a discipline that separates competent practitioners from exceptional ones: information gathering . The reconnaissance phase is the intelligence foundation upon which the entire attack strategy is built. The quality of your reconnaissance directly determines the quality of your attack. Why T