标签：#testing

The dangerous part of web extraction is not the error. The dangerous part is a clean JSON response that looks correct and is not. If an AI agent uses that output, the mistake does not stay inside a scraper. It moves into a report, a lead list, a price alert, a CRM update, or an automated decision. So before you trust any web extraction API in an agent workflow, test whether it fails honestly. Here is the checklist I use. 1. Test a real page, not a demo page Demo pages are usually polite little museum exhibits. Use pages that behave like the actual internet: a JavaScript-heavy product page a search results page a page with missing fields a login-gated page a blocked or rate-limited page a thin page that has layout but little useful content If the tool only looks good on clean static pages, you have not learned much. 2. Check whether the API separates fetch success from extraction success A page can return HTTP 200 and still be useless. Common examples: the final page is a login screen the visible content is a bot challenge the useful data loads after hydration and never appeared in the fetch the page contains a generic country selector instead of the product the source has navigation text but no actual item data The extractor should not treat these as success just because the request completed. A better response says something like: { "status" : "failed" , "failure_type" : "login_required" , "extracted" : null , "next_step" : "Use a public URL, authorised access, or sample content." } That is boring. Boring is good here. Boring means the agent can stop safely. 3. Ask for fields that are not on the page This is the easiest lie detector. Ask the extractor for something impossible: Extract the product name, price, stock status, warranty length, CEO name, office address, and refund policy. If the page only contains product name and price, the API should say the other fields are missing or low confidence. It should not politely invent them because the schema asked nicely.

The test hung for 30 seconds. The response had already fired. One moved line fixed it. The test hung for 30 seconds, then timed out. The browser had received the response. The page had loaded. The data was there. The test was still waiting. The wizard I was writing a helper to walk through a 4-step booking wizard. After clicking "Next" on step 1, the page does a full navigation — window.location.href to step 2. Step 2 immediately loads doctor data from the API. The helper looked like this: await Promise . all ([ page . waitForURL ( /step=2/ ), step1Next . click ()]); await page . waitForResponse ( r => r . url (). includes ( ' /doctors ' )); Standard pattern: wait for navigation, then wait for the data request. Timeout. Every time. What I checked first The URL pattern. Maybe /doctors wasn't matching. Opened the network tab. The request was there: GET /api/v1/doctors , 200, 47ms. Correct URL, correct response. The page looked fine. The data was rendered. The test said it was waiting for a response that had already happened. Added waitForLoadState . Still hung. Added an explicit waitForSelector for an element that was clearly on the page. That passed. Then waitForResponse hung again. The response existed. The test couldn't see it. What was actually happening page.waitForResponse() is not a query. It doesn't look at what happened. It registers a listener — from that exact moment forward — and waits for the next matching response. The sequence in my code: Promise.all resolves when the URL changes to step=2 By the time the URL changed, step 2 had already loaded Step 2 had already sent and received /api/v1/doctors Then waitForResponse registered its listener Now it's waiting for the next /doctors response Which never comes Playwright doesn't buffer missed events. If the response fired before the listener was registered — it's gone. The fix await Promise . all ([ page . waitForURL ( /step=2/ ), page . waitForResponse ( r => r . url (). includes ( ' /doctors ' )), step1Next

You just merged an AI-assisted feature branch, the code review looks clean, and the app works in your local smoke test. Now comes the real question: do you add another traditional browser test, let an AI tool generate the coverage, or spend the time improving the observability around the existing suite? That decision is where a lot of teams get stuck. AI-assisted development changes more than coding speed. It changes the shape of bugs, the pace of UI churn, the expectations for review, and the amount of test maintenance you can tolerate. If you treat AI testing as a magic replacement for your current process, you will probably add noise. If you ignore it entirely, you miss a chance to reduce repetitive work and catch gaps earlier. The real choice is not AI vs non-AI The useful decision is usually this, should AI help create and maintain tests, should it assist human review, or should it stay out of the critical path and only support investigation? That splits into three practical modes: 1. AI assists development, but humans own test strategy This is the safest default. AI can help draft test cases, suggest assertions, summarize failing traces, or propose missing edge cases, but the team still decides what belongs in the suite. If your product has regulated flows, complex permissions, or revenue-critical paths, that ownership matters more than any automation shortcut. 2. AI generates or heals tests inside a human-defined framework This is useful when the team already knows what it wants to cover, but not every selector, fixture, or assertion has to be hand-written. AI can reduce repetitive maintenance, especially for UI-heavy apps that change often. The hidden cost is that you still need a way to judge whether the generated test reflects product intent or just mirrors the current page state. 3. AI becomes part of the evaluation and triage loop Here the value is not test creation, it is speed of diagnosis. AI can summarize logs, cluster failures, or explain a flaky pa

Introduction: The Controversial Rise of AI in Software Development The software development industry is at a crossroads. On one side, the rapid advancement of AI tools promises to revolutionize coding, automate repetitive tasks, and accelerate project timelines. On the other, a growing chorus of experts, led by figures like George Hotz , warns that the integration of AI agents into software development could become "one of the most costly mistakes in the field’s history." This bold prediction isn’t just hyperbole—it’s a call to scrutinize the mechanisms by which AI adoption could deform the very foundation of software engineering. At the heart of this debate are three critical failure points: over-reliance on AI without human oversight , insufficient real-world testing , and misalignment between AI capabilities and software development demands . Each of these factors acts as a stressor on the system, threatening to heat up development costs, expand systemic vulnerabilities, and ultimately break the delicate balance between innovation and reliability. Consider the causal chain: over-reliance on AI leads to a degradation of human expertise , as developers become less engaged in problem-solving. This, in turn, creates a feedback loop where AI-generated code, lacking nuanced understanding, introduces errors that go unnoticed. Without proper oversight , these errors propagate through systems, causing observable effects like reduced software quality and increased maintenance costs. Similarly, insufficient testing of AI agents in real-world scenarios means their failure modes remain unknown until they’re deployed at scale, risking systemic collapse in critical applications. The stakes are high. If unchecked, AI integration could lead to a loss of institutional knowledge , escalating development costs , and vulnerabilities in critical systems . The question isn’t whether AI has a role in software development—it’s how to implement it without deforming the field’s core princi

Internal note to the team, we need to improve test coverage and keep shipping, which means we should treat AI as a helper in the workflow, not as a replacement for testing discipline. AI-assisted development changes the shape of our risk. It can produce more code faster, but it also increases the chance that small logic mistakes, brittle selectors, and shallow test cases slip through review. The answer is not to add more manual checking everywhere. The answer is to be more deliberate about what we review, what we automate, and where we let AI help. What changes when AI writes part of the code The first thing that changes is review. When a developer uses AI to draft a feature, a test, or a refactor, the reviewer is no longer only checking intent and style. The reviewer also needs to check whether the generated code matches the product rule, whether it introduced a hidden dependency, and whether it quietly weakened coverage. That does not mean every AI-assisted change deserves extra ceremony. It means our review checklist should shift from "does this look correct" to "what did the model assume, and did we verify those assumptions?" That is especially important for test code, because generated tests often look plausible even when they do not prove much. Coverage should move from volume to signal AI tends to produce more test cases, but more cases are not the same as better coverage. If a generated test suite repeats the same happy path under slightly different names, the team gets a false sense of safety. Coverage should answer a more practical question, where are we most likely to break the user experience, and where will a test actually catch it? For chat and other AI features, prompt-by-prompt manual checks are a trap. They do not scale, and they encourage a habit of eyeballing output instead of verifying behavior. A better pattern is to build assertions around expected properties, create eval sets for representative prompts, and add regression coverage for failure

Critics say Trump plan to test AI models is short-sighted, performative.

Here's a trace that reset how I think about evaluating tool-calling agents. An agent tries to book a flight. It calls search_flights with departure_date="next Friday" . The endpoint expected an ISO date, so it returns a 400 . The agent retries the same string four times, then apologizes to the user and gives up. Now the part that actually bothered me. Tool selection was correct. The model picked the right function out of a registry of 28. My tool-selection accuracy logged a clean 1.0 . The aggregate task-completion logged a 0 . And neither number told me which of three things broke: the argument was wrong, the model never read the 400 body, or the retry policy looped on the same input. My eval wasn't wrong. It was asking the wrong question. What "tool-call accuracy" actually grades If the only thing you measure is did the agent call the right tool , you're testing intent, not execution. Tool selection is necessary, not sufficient. It passes the moment the right function name shows up in the trace, completely blind to whether the arguments were garbage, whether the model read what came back, or whether it recovered from the 400 . That's the gap. The metric checks that the agent started the right way. Production needs to know whether it finished the right way. The reframe: it's four eval problems, not one The thing I had to internalize is that tool-calling eval is four problems stacked, each with its own root cause: Tool selection , right tool, or correctly no tool Argument extraction , schema-valid and semantically correct Result utilization , did it actually use what the tool returned Error recovery , did it retry, fall back, or escalate Score them separately and "the agent failed" collapses into "the argument extractor regressed on date strings on the flight-booking path." One bisect instead of three days. What I rebuilt Layer 1: Tool selection (with the bucket everyone drops) F1 on the tool name, so a 28-tool registry doesn't hide a regression on one rare endpoint

Software testing is changing quickly. Teams are dealing with faster release cycles, more AI-assisted development, more complex browser behavior, and higher expectations around product quality. I collected a few practical articles that cover different parts of modern QA, test automation, developer workflows, and testing strategy. Recommended reads How to Test AI Agents for Tool Use, Memory, and Recovery Paths A practical framework for testing AI agents for tool use, memory retention, retries, and recovery paths, with concrete strategies for QA and engineering teams. How to Evaluate a Test Automation Tool for Shadow DOM, iframes, and Other Hard-to-Test UI Surfaces A practical buyer guide for evaluating test automation tools for shadow DOM testing, iframe testing, resilient selectors, and dynamic UI edge cases. How to Reproduce a Flaky Browser Test with Video, Logs, and Network Traces A practical workflow to reproduce a flaky browser test using video, logs, and network traces, then turn intermittent failures into repeatable bug reports. Endtest Review for Small QA Teams: Where Editable Test Flows Save the Most Time A practical Endtest review for small QA teams focused on editable test flows, maintainable test steps, and where no-code QA automation actually saves time. Editable Test Steps vs Generated Test Code: Which Holds Up Better After UI Changes? A practical comparison of editable test steps vs generated test code for UI change resilience, maintenance overhead, debugging, and team handoff, with guidance for QA and engineering leaders. Managed QA Services vs Staff Augmentation: What Changes in Ownership, Speed, and Cost A practical comparison of managed QA services vs staff augmentation, focusing on ownership, ramp time, communication overhead, cost, and maintenance risk. Automation Payback Period: How Long Does QA Test Automation Take to Break Even? Learn how to estimate the test automation payback period, model QA ROI, account for maintenance cost, and identify wh

Microsoft on Tuesday took the wraps off Adaptive Spec-driven Scoring for Evaluation and Regression Testing, an open source framework for spinning up AI evaluations.

Image by upklyak on Magnific Run an application. Click a few buttons. If the terminal doesn't have errors, then everything is working. Right? What's the point of writing tests if all seems to be fine. Let's explore testing discipline and why it's a habit every developer should build early. What is Testing Discipline? Testing discipline is the habit of verifying that your code works. It's not something you do at the end of a project. It's something you build into your development process. The goal is simple. Catch bugs as early as possible. A bug found while writing code usually takes minutes to fix. The same bug found in production can take hours to investigate, reproduce, and resolve. The earlier you find problems, the less expensive they become. Different Types of Tests When people talk about testing, they're usually referring to three categories. The first is unit testing . A unit test checks a single piece of functionality, usually a function or method. These tests are fast and easy to write, making them the best place for beginners to start. Next are integration tests . These verify that different parts of your application work together correctly. For example, does your service communicate properly with the database? Finally, there are end-to-end tests . These simulate a real user interacting with the application from start to finish. They provide the most realistic results but are usually slower and more complex. As a beginner, I recommend that you should focus on unit tests first. Different Testing Approaches As you continue learning, you'll come across different testing methodologies. One of the most popular is Test-Driven Development , often called TDD. The idea is simple. Write the test first. Watch it fail. Write enough code to make it pass. Many developers like this approach because it forces them to think about requirements before writing implementation details. You may also hear about Behaviour-Driven Development , or BDD. This approach focuses on desc

I build a security platform. Last night I stopped adding features and did something less fun and more honest: I sat down to make every capability prove it actually works — end to end, with real data, demanding a real pass or fail. "It ran" is not a pass. A page that renders is not a feature. A green checkmark is a claim, not evidence. So I went capability by capability and tried to break each one. I found four real bugs and one of them was a gut-punch: a whole detection engine that was wired into the UI, unit-tested, and never actually ran in production. Here's how the night went. The rule: drive it, don't admire it My method was boring on purpose. For each capability: Feed it real input through the real entry point (CLI or API), not a test fixture. Check the data actually landed (query the DB, don't trust the success message). Feed it a malicious input and a benign input — it has to fire on one and stay quiet on the other. The detection engine passed cleanly. I threw a PsExec process event at it and it lit up: $ zds-core detection eval --event '{"event_type":"process_create","process_name":"psexec.exe"}' 1 alert ( s ) : [ high] PsExec Execution — ( matched: map[process_name:psexec.exe] ) A wevtutil cl Security event tripped a critical "Log Clearing" rule. A plain notepad.exe matched nothing. Good — it detects, and it doesn't cry wolf. (Small UX papercut I fixed while I was there: if you forgot the event_type field, the engine silently matched nothing and printed "no rules matched" — which reads exactly like "you're safe." Now it warns you that the event can't match any rule. Silence that looks like safety is the most dangerous output a security tool can produce.) The one that hurt: ITDR Identity Threat Detection and Response. The engine has detectors for impossible travel, credential spraying, brute force, privilege escalation. All unit-tested. All green. I ran the real flow: POST two login events for one user — New York, then London thirty minutes later. That's ~5

There is a specific failure mode in AI-assisted QA work that most tooling discussions skip entirely, and it shows up earliest when you are working solo on a real engagement. Every new chat session is stateless. You paste the ticket, describe the feature, explain your severity logic, set up the context, and by the time the AI is actually useful, you have rebuilt your methodology from scratch for the third time that week. That is not a workflow problem you fix with better prompts. It is an architecture problem, and the fix is a skill file. QAJourney has a full breakdown of this system at qajourney.net/ai-qa-workflow-for-real-projects, including the actual skill files as free downloads. The short version: a skill file is a context document you load as a system prompt. It carries your test surface tiers, your three-path testing framework, your bug report format, your severity and priority logic, your Playwright conventions, and an explicit definition of what the AI does and does not get to call. Load it once per session. The AI operates inside your methodology from the first message instead of a blank slate. The local LLM layer solves a different problem. On a freelance or retainer engagement, tickets contain real product logic and real client data. Sending that to a cloud API on every session is a data exposure question whether or not it rises to a compliance issue. Running Ollama locally with the same skill file as system context keeps the engagement data on the machine. For the output quality required on QA tasks, current 7B to 14B models are sufficient. The cost at zero marginal per token makes it infrastructure rather than a service you pay by the session. The three-role setup in the workflow: engineer as judgment layer, cloud AI loaded with the skill file for complex reasoning and active session output, local LLM for lightweight tasks and client data work. The skill file is the constant across all three. The part that took time to internalize: AI dev teams already

By Suryansh Sinha (sinxcos07) Introduction Recently, I built AstroFit , a fitness-focused web application as a personal project to learn more about modern web development, deployment, databases, and building complete applications from idea to production. This project helped me understand how different parts of a web application work together, from the user interface to backend functionality and deployment. Why I Built AstroFit I wanted to work on a project that felt practical and useful while also helping me improve my development skills. Instead of creating a simple clone project, I decided to build a fitness application where I could experiment with real-world features and deployment workflows. Development Journey Building AstroFit involved much more than just creating pages and connecting them together. Some of the areas I explored while working on this project included: Frontend development Backend integration Database management Authentication systems Deployment and hosting Debugging production issues One of the biggest learning experiences was understanding how different technologies communicate with each other in a complete application. Future Plans I plan to continue improving AstroFit by adding more features, refining the user experience, and expanding its capabilities over time. This project is still evolving, and I'm excited to keep working on it. Project Links Live Demo: astrofit-fitness.vercel.app GitHub: sinxcos07 / astrofit-frontend Fitness platform combining workout tracking and astrology-inspired personalization. AstroFit AstroFit is a modern fitness web application that combines workout tracking with astrology-inspired personalization to create a unique and engaging fitness experience. Features Modern responsive UI Astrology-inspired fitness experience Workout tracking interface User authentication system Backend integration Smooth and interactive design Mobile-friendly layout Tech Stack Frontend HTML5 CSS3 JavaScript Backend Node.js Express.js SQL

Based on real QA scenarios. About what happens when AI-generated metrics replace real testing, and the quiet engineer in the back row has been running his own numbers the whole time. Act 1: The Review Meeting I was sitting at the back of the long table, a ThinkPad in front of me, screen dimmed. On the big screen, Zhang Lei was presenting the acceptance data for his "AI Automated Testing Platform." His delivery was smooth. Every slide was a beautiful chart — coverage trends, automation rate improvements, regression testing time curves. All three lines pointed up and to the right, exactly like the textbook ideal curves. "In the past three months, the AI testing platform has executed 47,000 test cases, achieving 97.3% functional coverage. Regression testing time has dropped from 12 hours to 2.1 hours." Sparse applause. Zhang Lei added the final slide: "Monthly savings: approximately 200 person-days in labor cost." General Manager Zhou nodded and started the applause. That number was what he cared about most. I glanced at the other end of the table — the client's representative from RuiJie Technology. Chief Engineer Shen. Early fifties, thinning on top, silver-rimmed glasses. He hadn't said a word through the entire presentation. Hands folded on the table, occasionally jotting notes in a small book. Zhang Lei opened the Q&A slide and looked around the room: "Any questions?" Chief Engineer Shen flipped through the printed materials in front of him, stopped at the appendix, and looked up. "Page 47, Table 3.2 — what's the confidence interval on that 97.3% coverage?" The room went silent for about 15 seconds. Not the kind of silence where people are thinking. The kind where nobody had ever thought about it. Zhang Lei stood by the projector, clicker still in his hand, paused for two seconds: "Uh... the model confidence is quite high. The specific number is in the technical report." "Which page?" "I'll need to look it up." Chief Engineer Shen didn't push further. He looked do

After months of using Jest for unit testing, we decided to take the plunge and migrate to the built-in node:test runner. The results were surprising, with some features working seamlessly and others requiring significant rework. In this post, we'll share our journey and the lessons we learned along the way. Introduction to node:test The node:test runner is a built-in testing framework that comes with Node.js. It's designed to be fast, efficient, and easy to use. Here's an example of a simple test using node:test: import { test } from ' node:test ' ; import { Command } from ' @aws-sdk/client-lambda ' ; test ( ' Lambda client test ' , async ( t ) => { const lambdaClient = new Command (); const response = await lambdaClient (); t . equal . responseStatusCode , 200 ; }); Warning: When using node:test, make sure to handle errors properly, as unhandled errors can cause the test runner to crash. Migrating from Jest to node:test Migrating from Jest to node:test requires some changes to your test code. One of the main differences is the way you handle ES modules. In Jest, you can use the jest.config.js file to configure how ES modules are handled. In node:test, you need to use the --test-type option to specify the type of test you're running. Here's an example of how to migrate a Jest test to node:test: // jest.test.js import { lambdaClient } from ' ../lambdaClient ' ; describe ( ' Lambda client test ' , () => { it ( ' should return 200 ' , async () => { const response = await lambdaClient (); expect ( response . statusCode ). toBe ( 200 ); }); }); // node-test.test.js import { test } from ' node:test ' ; import { lambdaClient } from ' ../lambdaClient ' ; test ( ' Lambda client test ' , async ( t ) => { const response = await lambdaClient (); t . equal ( response . statusCode , 200 ); }); Tip: Use the --coverage option to generate code coverage reports for your tests. Overcoming Incompatibilities and Limitations One of the main limitations of node:test is that it does not su

Here’s why Anthropic and OpenAI are on board with Illinois safety testing.

Why We Built JaisCloud — A Free, Single-Binary AWS Emulator in Go If you've ever tried to test AWS-dependent code locally, you've probably reached for LocalStack. It works — but it comes with baggage: Python runtime, Docker dependency, and the features most teams actually need locked behind a Pro subscription. JaisCloud is our answer to that problem. What is JaisCloud? JaisCloud is a free, open-source local AWS cloud emulator written entirely in Go. It implements the exact AWS wire protocols — Query/XML, JSON/Target, REST — so your existing SDK code points at JaisCloud and works unmodified. No shims, no proxy rewrites, no SDK patches. # Start it jaiscloud-aws start # Point your SDK at it export AWS_ENDPOINT_URL = http://localhost:4566 export AWS_ACCESS_KEY_ID = test export AWS_SECRET_ACCESS_KEY = test export AWS_REGION = us-east-1 # Your existing code works — no changes needed aws s3 mb s3://my-bucket aws sqs create-queue --queue-name my-queue The Problem With Existing Solutions JaisCloud LocalStack Community Moto Single static binary Yes No - Python + Docker No - Python library Zero runtime deps Yes No No Postgres persistence Yes Pro only No Real Spark/EMR execution Yes No No Apache Iceberg Yes No No Prometheus metrics Yes Pro only No State export / import Yes No No Deterministic time control Yes No No Written in Go Yes No No License Apache-2.0 Apache-2.0 Apache-2.0 Key Features Single Static Binary JaisCloud ships as a single Go binary per cloud. No Python, no Docker, no Node — just download and run. Works on laptop, CI runner, or Kubernetes pod. # Download and run — that is it ./jaiscloud-aws start # Or with Docker docker run --rm -p 4566:4566 rjaisval/jaiscloud-aws:latest Portable State Snapshots Export the complete emulator state — every resource, every account, every region to a single gzip tarball and restore it anywhere in milliseconds. # Capture a baseline jaiscloud-aws export -o baseline.tar.gz # Restore on a teammate's machine or CI runner jaiscloud-aws i