标签：#Testing

A free AI visibility score is auditable only when you can inspect the prompt, engine, raw answer, date, and denominator. Treat the score as a test result, not a property of your brand. This tutorial builds a six-check control you can run by hand, store as plain data, and compare with any tool's output. The workflow takes three buyer questions, runs them in two AI surfaces, and records the six answers without trying to force agreement. It will not estimate your entire market. It will tell you whether a dashboard's headline number has enough evidence to be investigated. What does an AI visibility score measure? An AI visibility score usually summarizes brand presence across a defined set of generated answers. That definition contains the trap: the question set is part of the metric. So are the engine panel, run date, session state, retrieval mode, and rule used to count a “hit.” Remove those inputs and the number is not reproducible. Imagine a tool asks three questions in two engines. That creates six cells. If your brand appears in two cells, the simple presence result is: presence = brand_present_cells / total_cells presence = 2 / 6 presence = 0.333... = 33.3% The arithmetic is trivial. The evidence is not. A different tool can ask five different questions in three engines and produce a different score without contradicting the first run. The two tools measured different grids. Keep the unit explicit: “present in two of six generated answers on this date” is defensible. “Our AI visibility is 33” is incomplete. Which evidence fields should you require? Require five fields for every result: prompt, engine, raw answer, timestamp, and counting rule. Use a sixth field for cited sources when the surface exposes them. A source-only appearance and a prose mention can signal different problems, so do not merge them silently. Here is one real saved result from Webappski's public 14 June 2026 tracker report: { "run_date" : "2026-06-14" , "prompt" : "beste Answer Engine Optimiz

You can load-test search latency with a script and a graph. Relevance has no such gauge by default, so most teams ship a new ranking rule, eyeball a handful of queries, and hope nothing important regressed. The fix is a small, boring relevance test suite: a fixed set of queries, human-judged expected results, and a metric you compute the same way every time — so "did this ranking change help?" becomes a number you can diff, not an argument you have in Slack. This post is a build guide. By the end you'll have a judgments file, a scorer that outputs precision@k, MRR, and nDCG, and a before/after comparison you can wire into CI. The examples use Postgres full-text search, but the harness is engine-agnostic — Elasticsearch, Meilisearch, or a vector store all slot into the same shape. Why can't I just load-test relevance the way I load-test latency? Latency is a property of the system. Relevance is a property of the match between a query and what a human expected to see — and that judgment lives outside the database. A commenter on an earlier post about running Postgres search in production put it well: latency can be load-tested, but quality needs query sets, expected result buckets, bad-query examples, and a way to compare changes before shipping a new ranking rule. That's the whole job, and none of it comes for free with your index. The trap is thinking a passing query proves relevance. SELECT ... WHERE tsv @@ query returning rows tells you the index matched. It says nothing about whether the right rows landed in the top 5, which is all a user ever sees. The takeaway: relevance is measured against human judgments, not row counts — so the first artifact you build is the judgments, not the query. Building the golden query set Start with 20–50 real queries. Pull them from your search logs if you have them (the head terms plus a long tail of specific ones), or write them from real user intents if you don't. For each query, mark which documents should come back and how rel

Here is every promise a to-do list makes. VARIABLE tasks Init == tasks = [i \in Ids |-> "absent"] Add(i) == tasks[i] = "absent" /\ tasks' = [tasks EXCEPT ![i] = "open"] Complete(i) == tasks[i] = "open" /\ tasks' = [tasks EXCEPT ![i] = "done"] Reopen(i) == tasks[i] = "done" /\ tasks' = [tasks EXCEPT ![i] = "open"] Delete(i) == tasks[i] # "absent" /\ tasks' = [tasks EXCEPT ![i] = "absent"] ClearCompleted == /\ \E i \in Ids : tasks[i] = "done" /\ tasks' = [i \in Ids |-> IF tasks[i] = "done" THEN "absent" ELSE tasks[i]] Not a summary. Not the important ones. All of them. A task cannot go from absent straight to done. Clearing completed items leaves the open ones alone. You cannot delete something that was never there. Nine lines, and when you've read them you have read the entire contract. Now go find that list for the system you work on. You can't. It doesn't exist. It's distributed across a test suite that asserts outcomes rather than rules, some validation scattered through handlers, and the memory of whoever's been there longest. The guarantees are real — your users depend on every one of them — and there is no file you can open to see them. That's the gap I want to talk about, because you can close it in an afternoon, and because something has changed recently that makes closing it pay for itself. The prime mark and two operators That's most of the syntax, so let's get it out of the way. tasks' means "tasks, in the next state." /\ is and . \E is "there exists." A definition like Complete(i) is a formula relating the current state to the next one — read it out loud: the task is open, and afterwards it is done. That's it. That's the language, near enough, for this purpose. The real file adds about eight lines of scaffolding around what you saw: a module header, a TypeOK saying a task is always in exactly one of the three states, and the two lines that tie the actions together — Next == \/ \E i \in Ids : Add(i) \/ Complete(i) \/ Reopen(i) \/ Delete(i) \/ ClearComplete

Here's a bug your test suite probably wouldn't catch. A backend developer refactors the user model. The id field — an integer since forever — starts coming back as a string: "42" instead of 42 . Every value is still "correct". Your assertion pm.expect(user.id).to.eql(42) fails, sure — but only on the one endpoint you asserted id on, not the other nine that return users. Meanwhile three client apps that did user.id + 1 are now computing "421" . That's structural drift , and it's what actually breaks API consumers: renamed fields, changed types, properties that quietly vanish. Field-by-field value assertions catch it patchily and by accident. Schema validation catches it systematically — and in Postman it costs about ten lines, because the ajv JSON-schema validator is built into the script sandbox. The ten lines In Scripts → Post-response on any request that returns a user: const userSchema = { type : " object " , required : [ " id " , " name " , " email " ], properties : { id : { type : " integer " }, name : { type : " string " }, email : { type : " string " , pattern : " @ " } } }; pm . test ( " Response matches the user schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userSchema ); }); That single test now fails if id becomes a string, if email disappears, if name becomes an object — every structural mutation, whether or not you thought to assert on that field's value. For an endpoint returning an array of users: const userListSchema = { type : " array " , minItems : 1 , items : userSchema // reuse the object schema }; pm . test ( " List matches schema " , () => { pm . expect ( pm . response . json ()). to . be . jsonSchema ( userListSchema ); }); Share one schema across every endpoint The real power move: your API returns users from /users , /users/:id , /login , /teams/:id/members … and they should all be the same shape . Store the schema once as a collection variable (JSON, stringified), and every request validates against the sa

I have been building a thing that lets a language model propose an UPDATE , then executes it for real inside a transaction, measures the actual before and after values, and always rolls back. A human reads the measurement and decides. Only then does anything commit. The pitch is one sentence: what you approve is not the model's description of its SQL, it is what the database did when the SQL ran. Last week I found that the thing showing you that measurement was showing you a subset of it, and had been since the first release. The failure Real output, from @hyuga/llm-safe-sql@0.4.0 installed from npm. One row: name = 'Tanaka' , postcode = '00100' . UPDATE customers SET name='Sato', postcode='00100' WHERE id=1 What this touches customers — Customer records. The postcode is used for billing address and delivery. 1 row would change, across 1 column: name Measured by running the statement and rolling it back id = 1 name: 'Tanaka' -> 'Sato' One row, one column. postcode is not mentioned, and that is correct — it is being assigned the value it already holds, so nothing about it changes. The card is describing the diff accurately. Approve it. Then, before it is applied, somebody else notices the postcode is wrong and fixes it: UPDATE customers SET postcode = '90210' WHERE id = 1 ; Now apply the approved plan: Applied: UPDATE on customers, 1 row(s), at 2026-08-10T09:49:12.049Z. DB now: [{"name":"Sato","postcode":"00100"}] The fix is gone. Zero warnings. The word postcode never appeared on the approval card, never appeared in the audit record, and never appeared in the comparison the tool makes before it commits. One variable doing two jobs The diff was built like this: const changed : string [] = []; for ( const c of Object . keys ( before )) { if ( same ( before [ c ], after [ c ])) continue ; // drop what did not move if ( auto . has ( lower ( c ))) continue ; // drop what the DB maintains itself changed . push ( c ); } That is a correct answer to "what should the card sho

Long-running agents are usually tested at startup and during normal operation. The awkward middle is ignored: what happens when you need to deploy a new image, rotate a credential, migrate a database, or restart the host while the agent is halfway through a tool call? A process supervisor can restart a crashed agent. It cannot decide whether a browser checkout was committed, whether a webhook was acknowledged, or whether a tool call is safe to replay. That decision belongs in the agent runtime. This post presents a small maintenance-window protocol for agents that run for hours or days. It has four goals: stop accepting new work; let safe work finish or reach a checkpoint; make ambiguous work visible instead of guessing; resume with an explicit recovery decision. 1. Model maintenance as a state transition Do not treat maintenance as kill -TERM followed by hope. Give the runtime a durable state machine: RUNNING -> DRAINING -> QUIESCED -> STOPPED | +-> NEEDS_REVIEW DRAINING rejects new jobs but allows an active job to continue until its next checkpoint or deadline. QUIESCED means there are no unclassified side effects in flight. NEEDS_REVIEW is the safe outcome when the process died after sending a request but before recording the response. Persist the transition, not just an in-memory flag. A minimal record can look like this: { "runtime" : "agent-7" , "maintenance_id" : "mw-2026-08-10-001" , "state" : "DRAINING" , "started_at" : "2026-08-10T08:00:00Z" , "accepting_work" : false , "active_runs" : 2 } If the host disappears, the replacement process can see that the previous shutdown never reached QUIESCED . That is much more useful than inferring health from a missing PID. 2. Put checkpoints around side effects An LLM step is usually replayable. A payment, email, browser click, deployment, or Git push may not be. Record a checkpoint immediately before and after every non-idempotent boundary: PLANNED -> DISPATCHED -> ACKNOWLEDGED -> OBSERVED On restart: PLANNED can be

I shipped a page that reported zero axe violations . It had button text at a contrast ratio of 1.04:1 — which is, for practical purposes, invisible text. The scan wasn't broken. It was answering a narrower question than I thought I was asking. The bug I had a theme system built the ordinary way. Tokens on :root , overridden in a prefers-color-scheme media query, and overridden again by an explicit [data-theme] attribute so a manual toggle wins in both directions. Buttons came in two flavours: a solid primary and a bordered secondary. .btn { background : var ( --accent ); color : var ( --panel ); } .btn.sec { background : transparent ; color : var ( --ink ); } In dark mode the accent goes light green, so white-on-accent stops working. I patched it the way you patch things at 1am: :root [ data-theme = dark ] .btn { color : #10241b } @media ( prefers-color-scheme : dark ) { :root:not ([ data-theme = light ]) .btn { color : #10241b } } Now count the specificity. Selector Specificity .btn.sec 0,2,0 :root[data-theme=dark] .btn 0,3,0 :root:not([data-theme=light]) .btn 0,3,0 :not() doesn't add specificity of its own, but its argument does. So :root (0,1,0) + [data-theme=light] (0,1,0) + .btn (0,1,0) lands at 0,3,0. My theme patch outranks the component modifier. In dark mode, every secondary button — transparent background, sitting on a #1a1c1f panel — got painted #10241b . Dark green on near-black. 1.04:1. The nasty part is that this class of bug is invisible in review. The rule looks correct. It is correct, for the buttons it was written for. It just also matched buttons it was never meant to touch, in one theme only. Why the scan didn't catch it axe-core evaluates the DOM as currently rendered . It reads computed styles, and computed styles resolve exactly one colour scheme: whichever one the browser is in right now. So npx axe https://example.com is not "does this page pass contrast." It's "does this page pass contrast in the scheme this headless browser happened to boo

Last week I changed a system prompt based on a feeling. It was the first prompt change after the evaluation harness from Part 8 went live, and I was completely sure about it. The target was the markdown table. Part 8's first nightly run caught the agent answering price comparisons with a markdown table that renders broken in the chat frontend. The fix looked obvious: add one line to the system prompt demanding plain text. I checked six conversations by hand. All six looked better. I was ready to ship it to production. Then I ran the comparison the way Part 8 promised: the same 40 cases, the same judge, two prompts. The old prompt won. Not by a little. It won 18 pairs, lost 10, and tied 12, and the judge's rationales made the reason visible. The plain-text line had also made the agent terse, and terse answers dropped the order summary that customers actually need. My confidence was a sample size of one. The dataset was the jury. This part is about the pattern that settled that argument: pairwise comparison, the LLM-as-a-judge pattern for A/B testing prompts and tool descriptions before they reach production. It is the harness from Part 8, upgraded to answer "which version is better?" instead of "is this version good?" The Problem With Ship-by-Feeling Every prompt edit is an experiment with one sample. You notice one conversation where the agent is verbose, you add "be concise", and the change ships because that one conversation got better. The dataset from Part 8 makes the agent measurable, but a nightly score cannot tell you whether a change helped. One night is noise, three nights is a signal, and by the time you have three nights of data you have already shipped the change to every user. The variable itself is the problem. A system prompt and a tool description are the two things in an agent you cannot unit test. Part 6 proved the code is bug-free. Part 8 proved the answers are good on a fixed dataset. Neither says anything about whether your new wording is better

When building Model Context Protocol (MCP) servers or AI agents that consume them, traditional API testing tools fall short. An MCP server isn't just a basic REST endpoint—it's a dynamic interface exposed to non-deterministic LLMs through stdio, HTTP, or SSE transports. Testing tool schemas, transient network failures, and agent behaviors usually requires writing a mountain of boilerplate. I built bubblemcp-test-kit to eliminate that friction: no backend accounts, no complex test setup, and zero instrumentation required. What is bubblemcp-test-kit? bubblemcp-test-kit is a lightweight, standalone testing toolkit designed specifically for MCP server developers and AI agent engineers. Key features include: Transport Agnostic: Work with stdio, HTTP, or SSE behind a unified API. Fluent Assertions: Native matchers tailored for MCP response structures and JSON Schemas. Mocking & Replay: Fabricate tool outputs locally or record real server runs to replay in offline CI environments. Agent Trace & Fault Injection: Test if your AI agent calls tools in the right order and handles errors properly. Quickstart Example You can run a complete mock test suite without spinning up a live server: import { createMockMcpClient , expectMcp , validateAgainstSchema , withRecording , createReplayClient , } from ' bubblemcp-test-kit ' // 1. Define your tool contract const healthCheckTool = { name : ' health_check ' , description : ' Reports service health ' , inputSchema : { type : ' object ' , properties : { service : { type : ' string ' } }, required : [ ' service ' ], }, outputSchema : { type : ' object ' , properties : { service : { type : ' string ' }, status : { type : ' string ' , enum : [ ' ok ' , ' degraded ' , ' down ' ] }, latencyMs : { type : ' number ' }, }, required : [ ' service ' , ' status ' , ' latencyMs ' ], }, } // 2. Set up a mock MCP client const mock = createMockMcpClient ({ tools : [ healthCheckTool ] }) mock . mockTool ( ' health_check ' ). resolves ({ service : ' weat

Originally published on hexisteme notes . A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap. That note answered three cases I already knew about, because I'd already tripped over them. It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet? This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken. The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: SELECT COUNT ( * ) AS total , SUM ( some_column IS NOT NULL ) AS filled FROM some_table ; If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it. Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing. Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword. In one afternoon it produced both a false positive and a false negative. The sharper miss: a literal grep for a write path failed to find an INSERT OR REPLACE statement that was, in fact, live and doing exactly the writing I was looking for. Grep matched the shape of the bug I expected walking in, not the shape the code actually had. Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled. The question that works is: of all the rows that exist, how many have this column fi

A Practical Guide to Faster, Deterministic Flutter & Dart Unit Testing If you’ve ever written unit tests for classic package:bloc applications using bloc_test , you know the drill: build your BLoC, dispatch an event in act , and assert state emissions in expect . Under the hood, classic BLoC processes state updates asynchronously via Dart microtask-queue Streams . While robust, testing asynchronous streams can introduce microtask timing headaches, race conditions, or the need to drain queues or use fakeAsync when testing complex side-effects. In BlocSignal , state updates propagate synchronously . Calling emit(newState) updates the underlying signal graph in the exact same call stack frame. This handbook is a practical, recipe-based guide to testing BlocSignal and CubitSignal applications using package:bloc_signals_test . Whether you’re coming from classic BLoC or brand new to Signals, this guide shows you how to test every scenario cleanly—and why it’s significantly easier than classic stream-based testing. 🤖 AI Assistant Tip : Working with an AI coding assistant (like Antigravity, Gemini CLI, or Cursor)? The official bloc-signals plugin includes a pre-built testing skill ( plugins/bloc-signals/skills/bloc-signals/ ) that automatically teaches your AI assistant these exact testing conventions, observer scoping rules, and declarative blocSignalTest patterns! 🛠️ Quick Reference: BLoC Streams vs. BlocSignal Testing Testing Task Classic BLoC ( package:bloc_test ) BlocSignal ( package:bloc_signals_test ) Why it’s easier in BlocSignal Execution Environment Often requires flutter test engine Pure dart test execution Blazing Speed : Business logic tests run in pure Dart CLI without booting Flutter UI engine. Simple State Assertions Requires async stream listener or blocTest Direct expect(cubit.state, 1) or blocSignalTest Synchronous : State updates on the next line of code without microtask delay. Failure Diagnostics Legacy Instance of 'CounterCubit' Built-in toString() :

You're Not Comparing Models. You're Comparing Contracts. Two teams publish scores on the same agent benchmark. One lands in the low sixties. The other clears seventy. A procurement team reads the spread and makes a call. What they do not see: both teams may be running the same model. They did not need to change the weights for the gap to appear. The spread can come from scaffold alone. One team wrapped the model in a harness with better retries. Different tool defaults. A planner step the other team had skipped. None of that appears on the leaderboard. The comparison that drove the decision was not between two agents. It was between two contracts. There Is No Benchmark The mistake hiding behind this story is a category error. People talk about agent benchmarks as if they measure a thing called “the model.” They do not. They measure a coupled system. The model is one component. The rest is a stack of protocol decisions that are almost never disclosed and almost always matter. The score is the output of that stack. Change any layer and you change what the number means. Recent research on agent evaluation has named those layers explicitly. There are at least seven. Deployment regime. Observation channel. Harness and scaffold. Metric and action. Configured evaluator. Grader protocol. Audit bundle. Each is a contract. Each is negotiable. And each can silently change the verdict while the headline looks the same. That is what a benchmark actually is. Not a measurement of a model. A measurement of an entire testing contract, of which the model is one slot. There is structural reason the seven layers are the seven layers. They cluster into three corners that show up in almost every published agent-evaluation failure. What the model is rewarded for. How that reward is optimised. And how the test contract differs from production. Once you hold those three corners in view, the seven-layer stack stops feeling like a checklist and starts behaving like the actual shape of what is

This article is about a poker-analysis framework, but the engineering problem is common to LLM tool use. The framework uses an LLM as an input and control layer. It reads a natural-language poker question, chooses a local calculator, and proposes typed fields. A Python program, not the LLM, performs the numerical calculation and returns a structured result with verification data. In this evaluation, a coordinator manually passed each calculator-eligible saved proposal to the command-line calculator; no automatic runtime bridge connected them. The design intent was to reduce manual arithmetic checking by sending numerical claims to deterministic, internally verified local software. The evaluation below tests whether those claims were checked and whether the handoff remained auditable. It did not measure time saved or the overall quality of the resulting poker analysis. The calculator catalog is poker-specific. This is not a poker strategy guide, and you do not need to know poker strategy to follow the failure. The question is whether a correct calculator can produce a verified result after the LLM chooses an interpretation without asking the user to confirm it. The workflow was tested with 25 hand-authored cases that were fixed before execution. They are labeled C01 through C25: C01–C23 tested the LLM's routing, proposed input, and boundary decisions; C24 and C25 repeated two accepted inputs to check non-volatile result semantics. The labels are test numbers, not poker terminology. The main example uses one small pot-odds model. In this model, the pot is the shared pool of chips the players are competing for. The calculator's inputs are: pot_before_bet : the amount already in the pot before the opponent's new bet; opponent_bet : the amount the opponent adds; call_cost : the amount the player must add to continue; expected_rake : an optional amount removed from the final pot. The calculation is: net final pot = pot_before_bet + opponent_bet + call_cost - expected_rake

You just installed Cursor, opened a TypeScript file, and asked the agent to fix a bug. Ten seconds later it handed you a type SomeType = any and a @ts-ignore above the line that wouldn't compile. This is the moment most developers discover that AI coding agents are powerful but undisciplined. The fix isn't a better model. It's rules. Most AI coding agent best practices boil down to a single idea: tell the agent what good looks like before it starts typing. Cursor lets you define rules files in .cursor/rules/ that load alongside your project context and tell the agent how to behave. Claude Code has its own rules system, Windsurf has a rules directory, Copilot reads .github/copilot-instructions.md . Learn to configure cursor rules properly and your agent starts behaving like a careful senior engineer instead of an eager intern. What cursor rules files are Cursor rules are markdown files with a .mdc extension stored in .cursor/rules/ at your project root. Each file is a set of instructions the agent reads before it starts working. When a rule's conditions match the file being edited, the instruction is injected into the model's context window. A cursor rules file has two parts: a YAML frontmatter block between --- markers, and a markdown body with the actual instructions. How to configure cursor rules: the frontmatter fields Three fields matter. description (required). A short summary of what the rule enforces. Cursor surfaces this when you toggle rules, so make it specific. globs (optional). File patterns the rule applies to. Without globs, the rule applies to everything, which wastes context and creates conflicts. alwaysApply (optional). Set to true for rules that should load in every session, regardless of the files involved. Leave it false for rules that only trigger when matching files are touched. Real example: --- description : Enforce strict TypeScript, no any, no ts-ignore globs : ** /*.{ts,tsx} alwaysApply : false --- # Strict TypeScript ## Context This codeb

Every AI coding agent I've used shares one habit: it writes the plausible thing. The code compiles. The tests pass. And a senior engineer reviewing it would reach for a red pen. any where a union belongs. Tests that assert on implementation so they survive any refactor. catch (e) {} blocks that quietly swallow production errors. The fix isn't a better model or a cleverer prompt. It's a set of rules at the repo level, written in a format the agent is guaranteed to read. What rules files are Cursor reads .cursor/rules/*.mdc . Claude Code reads CLAUDE.md and AGENTS.md . The .mdc format is plain markdown with YAML frontmatter. Here's the opening of the TypeScript rule file from the AgentForge sample pack: --- description: "Strict TypeScript discipline for production code" globs: "**/*.{ts,tsx}" alwaysApply: true --- Three fields carry the weight. description tells the agent in one sentence what the file is for. globs scopes it, so a TypeScript rule never fires on a Python file. alwaysApply: true loads it into every session. Agents skip a 2,000-line rules file. They read a 40-line one. Write a discipline contract, not a wish list Claude Code follows instructions frighteningly well, and that cuts both ways. Tell it "write good code" and it will be confidently, grammatically wrong. An AGENTS.md contract fixes that. Start with a baseline of rules: think before you act, take small verifiable steps, never claim what you haven't verified, no drive-by refactoring. Then add a verification ladder with six rungs. Does it compile? Does the changed behavior work? Does it break anything adjacent? Does it follow the codebase's conventions? Does it hold at the boundaries? Is it observable in production? The first three are mandatory for every change. Then the failure protocol, which is the line that pays for itself: First failure: fix and re-verify. Second failure: re-derive, your mental model is wrong, form at least two new hypotheses. Third failure: stop, revert to last known-good, d

Last week I ran a demo of the agent for a colleague who was deciding whether to bet a feature on it. The approval gate from Part 7 worked exactly as designed. The agent searched, added to cart, asked for the address, and stopped at the confirmation link. My colleague nodded and asked one question: "OK, but is it actually good?" I did not have an answer. I had 31 passing tests from Part 6, which prove the agent is bug-free. I had a state machine, which proves it cannot place an order without a human. Neither of those proves the agent answers customers well. A bug-free agent can still tell a customer the shop ships within two days when the shipping partner takes five. No unit test catches that, because no unit test reads the answer. That is the gap this part closes. Part 7 ended with a promise: the next part would build an evaluation harness, so "is it good" stops being a feeling and becomes a score. This is that part. I built an LLM-as-a-judge harness for the same e-commerce agent as Parts 1 through 7: same nine tools, same supervisor, same memory. It runs 40 real conversations from production logs against five metrics every night and prints a score for each one. The first run was uncomfortable, and that is exactly why it exists. I am a Senior Software Engineer II at BS23 in Dhaka, and I have been building production AI agents with Spring Boot and Spring AI for over a year. Everything below is the harness as I actually run it. The Difference Between Tested and Good Part 6 tested the agent without an LLM: 31 tests, zero model calls, asserting on tool calls, services, and the order state machine. That suite answers "did the agent call the right tool, in the right order, with the right arguments?" It cannot answer "was the answer right?" because you cannot write an assertion for an LLM's wording. Evaluation is a different layer. You cannot assert on the answer, but you can judge it, and you can use an LLM to do the judging. Spring AI documents this pattern in its LLM-as

AI coding tools have solved one problem remarkably well: They can produce code extremely quickly. That sounds obviously good. And most of the time, it is. But software development has never really been constrained by how fast we can type. The expensive part comes later. Understanding the code. Reviewing it. Debugging it. Changing it six months later when the person—or model—that wrote it has forgotten why it exists. Test automation is where this becomes especially interesting. Generating the Test Is the Cheap Part You can ask an AI coding assistant: Write Playwright tests for our signup, login, checkout, password reset, dashboard, invoices, settings, and admin pages. And a few minutes later you might have hundreds or thousands of lines of test code. It feels like incredible leverage. Until the suite starts failing. That’s the argument behind looking at the hidden cost of AI-generated test code . Generation cost has collapsed. Maintenance cost hasn’t. In some cases, AI actually increases it because you now have more code than your team would have written manually. AI Pull Requests Need Different Review There’s another subtle problem. Humans tend to judge large AI-generated pull requests differently. When someone on your team writes 80 lines, you probably read them. When an AI assistant generates 1,800 lines? You skim. You look at the filenames. You check whether CI is green. Merge. That’s dangerous for normal application code and potentially worse for test code because a bad test can happily pass for months. There are good ideas in this guide to testing AI coding assistant pull requests , but the bigger principle is simple: AI-generated tests need validation just like AI-generated product code. “Generated successfully” does not mean “tests the right thing.” Agents Add Another Failure Mode Now we’re moving from AI that writes test code to AI that actually decides what actions to take. That introduces a new question: What if the model chooses the wrong tool? An agent m

Two pages on my site went live on July 22. On August 8 they had zero impressions in Google. Not low. Zero, across three weekly exports. URL Inspection didn't say "crawled, not indexed." It said Google could not recognise the URL. Referring sitemap: none detected. Referring pages: none detected. Last crawl: not applicable. Never discovered. Seventeen days. The pipeline was green the entire time My deploy is a small chain: rsync the file, import it into MySQL, restart the service, ping IndexNow. Every step returned success. The last step returned 200 on every URL, every deploy, for three weeks. Here's what I'd never examined: IndexNow doesn't feed Google. It's Bing, Yandex, Seznam, Naver. My green light was real — it was just about a different search engine than the one whose console I was reading. That's the whole bug, and it isn't an SEO bug. It's the generic one: system A returns 200 → I conclude something about system B → nothing in the response object ever objected If you've ever read a webhook 202 as "the downstream processed it," or a CDN purge 200 as "the edge is cold," it's the same shape. What actually broke Search Console's Sitemaps report: Submitted: 2026-07-22 Last read: 2026-07-22 ← seventeen days ago Discovered: 101 URLs ← the file has had 117 for weeks The two pages went live on July 22 — the same day as the only read. Google fetched the sitemap and moved on, within hours of the file changing. Then nothing brought it back, because a sitemap changing on your server notifies nobody. There is no push. It's a pull-only resource with no cache invalidation, and if the consumer doesn't happen to return, your new URLs live in a document no one is reading. Resubmitting took two minutes. Read immediately, 117 URLs. So I wrote the check. It doesn't catch the bug. This is the part worth more than the fix. I wrote a post-deploy verifier. It does two things: // 1. every published, non-redirected page appears in the live sitemap const missing = published.filter((p) =

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor

Last updated: August 2026 The first few orders of an online store rarely make anyone think seriously about infrastructure. When there are only a handful of orders, almost any problem can be handled manually: check inventory, correct a status, contact a supplier, or figure out why incorrect information is appearing on a product page. At 10 orders, this is still a perfectly workable model. At 100, manual tasks start consuming a noticeable amount of time. Once operations reach 1,000 and beyond, however, the nature of the problem changes: small inconveniences begin turning into systemic limitations. As we develop Droplox, we increasingly look at scaling from this perspective. Growth from 1 to 10, then 100, and eventually 1,000 orders may look like nothing more than an increase in a single number. For architecture, data, and operational processes, these are completely different operating conditions. 1–10 Orders: Almost Everything Can Be Fixed Manually At the earliest stage, manual work is not necessarily a problem. In fact, it can be useful. The team gets to observe real user scenarios and understand which processes are genuinely worth automating and which happen so rarely that building a dedicated system for them would be premature. An incorrect order status can be checked manually. Outdated product information can be corrected quickly. A supplier issue can be handled as an isolated case. The difficulty comes later. Temporary solutions have an unfortunate tendency to become permanent. A spreadsheet created “for a couple of weeks” is still being used months later. A manual check becomes a mandatory step. A field added for one specific scenario suddenly becomes involved in five more. As long as the number of operations remains small, the cost of these compromises is almost invisible. Around 100 Orders: Random Problems Start Repeating As order volume grows, it isn’t only the workload that increases. Situations that once seemed like isolated incidents begin appearing regula