标签：#Testing

Overview Open WebUI exposed a real Ota lifecycle boundary. This was not mainly a parsing or contract-shape repo. The contract was already strong enough to model: source-checkout verification packaged native runtime through uv run open-webui serve frontend development runtime default Docker Compose runtime What the repo exposed was operational truth after proof: a successful native proof still left a host workload alive the first cleanup fix then widened too far and treated a Compose-owned runtime as the same class of host workload That made Open WebUI a valuable pressure repo. It forced Ota to get more precise about cleanup ownership instead of treating all successful runtime proof as one generic teardown problem. The current pressure contract pins released Ota v1.6.24 . Its latest green matrix run proves the release surface at the exact contract and workflow revision linked below. What Open WebUI exposed in Ota This repo exposed four meaningful weaknesses. 1. proof success was weaker than it looked The first issue was not that runtime proof failed. It was that runtime proof succeeded and still left the native workload alive afterward. In this repo, the packaged native workflow launches: serve:native : launch : kind : command exe : uv args : - run - open-webui - serve - --host - 0.0.0.0 - --port - " 8080" Ota proved that workflow, but the launched process tree was still alive after proof completed. In GitHub Actions, that surfaced through setup-uv post-job cleanup, which blocked while the uv cache was still in use. That was an Ota gap. If proof succeeds but leaves behind repo-owned runtime state that later breaks CI cleanup, the proof surface is still incomplete. 2. native service cleanup widened past its real ownership boundary The first core fix made Ota clean selected native service workloads after successful proof. That was directionally correct, but Open WebUI immediately exposed the next boundary. The Docker workflow uses a native task shape to launch Compose:

You have been here. The test suite is green. The deployment pipeline reports all checks passed. Then a customer opens a ticket with a screenshot that shows something your test never caught. The test passed in staging. It fails in production. And you cannot reproduce it locally. This is not a flaky test problem. It is a fidelity problem. Your test environment and your production environment are not the same thing. The gap between them is where real bugs live. Let me walk through one concrete example, the fix, and what it teaches about writing tests that survive the handoff to a real user. The Problem: Environment Drift A fintech team I worked with had a checkout flow. The test clicked "Pay Now", waited for a success message, and asserted the text "Payment successful" appeared on screen. It passed every time in staging. Customers reported that after paying, they saw a blank white page for several seconds before the success message appeared. Some of them closed the tab during that blank period, thinking the payment failed. The transaction went through. The customer never saw the confirmation. Support tickets piled up. The test never caught this because the staging environment served the success page in under 200 milliseconds. The blank period did not exist there. Production had a slower downstream service that introduced a three-second delay between the payment confirmation and the page render. The test was correct in what it checked. It was wrong in what it assumed about timing and state. The Fix: Test the Experience, Not Just the Outcome The fix was not to add a longer wait. The fix was to test what the user actually experiences during that gap. Here is a minimal Playwright test in TypeScript that catches this class of problem: import { test , expect } from ' @playwright/test ' ; test ( ' checkout shows loading state before success ' , async ({ page }) => { await page . goto ( ' /checkout ' ); await page . fill ( ' #card-number ' , ' 4111111111111111 ' ); await page

Gherkin is not a test automation tool. It never was. Yet here we are, five years into your SDET career, and you're staring at a feature file that reads like a step-by-step manual for a human tester. Given I log in with username "admin" and password "password123" . When I click the "Submit" button . Then I see the text "Welcome" on the screen . You've written two years of these. Your team calls it BDD. Your manager calls it "living documentation." And somewhere in the back of your mind, a quiet voice whispers: This is just a manual test case with extra steps. That voice is right. Let me say it plainly: if your Gherkin scenarios describe how the system works instead of what it should do, you are not doing BDD. You are writing manual test cases in a structured English format and calling it automation. The only thing you've automated is the illusion of progress. The problem isn't Gherkin. The problem is how we use it. Most teams adopt BDD because someone read a blog post about "collaboration" and "shared understanding." They install Cucumber or SpecFlow. They write feature files. They map steps to Selenium or Playwright code. And they call it a day. But look closely at what happens next. The product owner never reads the feature files. The developer skims them once and goes back to writing code. The QA engineer — that's you — becomes the sole maintainer of a growing pile of Gherkin that nobody else touches. You're not facilitating collaboration. You're translating manual test cases into a format that requires a compiler. Here's the real test. Take any feature file from your project. Hand it to a developer who has never seen it. Ask them to implement the feature using only the Gherkin as a spec. If they can write production code from it, you have real BDD. If they ask you for clarification, you have documentation theater. I've seen teams with hundreds of feature files. Beautifully formatted. Perfect indentation. Tags for every regression cycle. And not a single one of th

Most engineers expect AI-assisted automation to be the easy part. You describe a test, the model writes it, you move on. The first week will prove you wrong. Not because the code is bad. Because the code is almost right. And almost-right code is harder to debug than wrong code. Wrong code fails loudly. Almost-right code passes on Monday, fails on Tuesday, passes again on Wednesday, and by Thursday you are questioning whether you understand your own application. I have watched teams adopt AI copilots into their Playwright suites and spend the first five days doing nothing but untangling false passes. If you are about to start this journey, here is what that week actually looks like. The Problem: The Model Does Not Know What "Stable" Means A language model has never waited for a network response. It has never watched a flaky selector survive three CI runs and then collapse on the fourth. It writes tests from a static understanding of your page, not from the dynamic reality of your application. You will ask it to write a test that clicks a button and waits for a confirmation toast. The model will produce something like this: await page . click ( ' button:has-text("Submit") ' ); await page . waitForSelector ( ' .toast-success ' ); Looks fine. Runs fine. Then your team deploys a new build where the toast takes 400ms longer to appear because of an analytics call. The test fails. Not because the feature broke. Because the model assumed a timing that was never guaranteed. This is the core problem. The model writes tests that match the page as it was when the model saw it . It does not write tests that match the page as it will be . The Solution: Treat AI-Generated Tests as Drafts, Not Deliverables The shift is mental before it is technical. You cannot review AI-generated tests the way you review human-written tests. Human tests come with intent. AI tests come with patterns. You need a different review lens. First, look for every hardcoded wait. Replace it with a state-based

Cursor's "Sand" project targets non-developers. Anthropic's Claude Cowork runs cross-device. OpenAI's ChatGPT Work delivers documents, spreadsheets, and presentations. When an agent produces a deliverable for someone who cannot read the code, the verification interface matters more than the output format. A download button is not verification. The problem When a developer reviews agent output, they can read the diff, run the tests, and check the types. When a non-developer receives an agent-produced spreadsheet or document, they have no equivalent verification path. They either trust it completely or reject it completely. Neither is useful. What a verification UI needs Element Purpose Implementation Source list Show what inputs the agent used Links to source documents with timestamps Confidence indicator Flag low-confidence outputs Per-section confidence derived from source coverage Change highlights Show what the agent generated vs. copied Diff view between source and output Audit trail Record who requested what and when Append-only log with request ID, timestamp, and result hash Rejection path Let the user say "this is wrong" without starting over Feedback record linked to specific output section A minimal verification component <!-- agent-output-verification.html --> <!DOCTYPE html> <html lang= "en" > <head> <meta charset= "UTF-8" > <title> Agent Output Verification </title> <style> .verification-card { border : 1px solid #ddd ; border-radius : 8px ; padding : 16px ; margin : 8px 0 ; font-family : system-ui , sans-serif ; } .source-list { list-style : none ; padding : 0 ; } .source-list li { padding : 4px 0 ; border-bottom : 1px solid #eee ; } .confidence-low { color : #c0392b ; font-weight : bold ; } .confidence-medium { color : #e67e22 ; } .confidence-high { color : #27ae60 ; } .reject-btn { background : #fff ; border : 1px solid #c0392b ; color : #c0392b ; padding : 4px 12px ; border-radius : 4px ; cursor : pointer ; } .reject-btn :hover { background : #c0392b

Tu prueba pasó el lunes: misma entrada, mismo código y temperature=0 . El martes falló sin cambios en tu aplicación. La aserción esperaba una cadena exacta, pero el modelo devolvió la misma respuesta con una redacción ligeramente distinta. El agente funciona; tu suite de pruebas no. Prueba Apidog hoy Este es el coste de probar sistemas que llaman a modelos de lenguaje. Incluso con temperatura cero, no obtendrás salidas idénticas byte a byte entre ejecuciones. En lugar de probar la redacción exacta, prueba el contrato: estructura, campos, rangos y efectos esperados. Este artículo profundiza en el tercer modo de fallo de nuestra guía sobre por qué los agentes de IA fallan en producción . Por qué temperature=0 no significa determinismo La temperatura controla cómo el modelo selecciona el siguiente token. Con temperature=0 , el modelo elige el token más probable, lo que parece reproducible. Sin embargo, esa configuración no garantiza resultados idénticos. La causa está debajo del modelo: Las operaciones de punto flotante en GPU no son asociativas: el orden de las sumas puede modificar mínimamente el resultado. Esa diferencia puede cambiar cuál token queda en primer lugar. El orden de ejecución puede depender de cómo el proveedor agrupa solicitudes, del hardware, de la región o de la versión del kernel. Los proveedores pueden cambiar GPUs, bibliotecas de inferencia o cuantización de pesos. Una discusión extensa en vLLM explica por qué una semilla fija y temperature=0 no bastan para lograr reproducibilidad bit a bit. La conclusión práctica es simple: el determinismo es una propiedad de toda la pila de servicio, no una opción de tu solicitud. Tu prueba debe aceptar variaciones de redacción cuando el significado y el contrato siguen siendo correctos. Por qué las aserciones de cadena exacta vuelven inestable tu suite Esta prueba es frágil: assert response == " Your order total is $42.00. " Puede pasar hoy y fallar mañana si el modelo responde: Your total comes to $42.00. La

Your test passed on Monday. Same input, same code, temperature=0 . On Tuesday it failed, and you changed nothing. The assertion expected an exact string, but the model returned the same answer with slightly different wording. The agent is fine; the test is not. Try Apidog today This is the cost of testing language-model output. Even at temperature=0 , you cannot rely on byte-identical responses across runs. Instead of testing exact wording, test the API contract: structure, required fields, valid ranges, tool-call payloads, and safety constraints. This is a deeper look at failure mode three in our guide to why AI agents break in production . Why temperature=0 does not mean deterministic Temperature controls token sampling. At zero, the model selects the most probable next token, which sounds reproducible. In practice, the serving stack introduces variation. GPU floating-point math is not associative: adding the same values in a different order can produce small numerical differences. Those differences can change which token ranks highest. Once one token differs, every token after it can diverge. The order of operations can vary based on: Request batching GPU hardware Inference kernels Provider routing and regions Model-serving library versions Weight quantization changes Providers can also update infrastructure without changing your API request. As this vLLM discussion explains, a fixed seed and temperature=0 are not enough for bitwise reproducibility. Determinism is a property of the entire serving stack, not a request parameter. Design tests accordingly. Why exact-string assertions make tests flaky This assertion is fragile: assert . equal ( response , " Your order total is $42.00. " ); It fails if the model returns a correct variation: Your total comes to $42.00. The failure does not indicate a product regression. It indicates that the test is coupled to wording that is expected to vary. Flaky tests create a predictable failure pattern: Developers rerun the suite

A helpful comment on my EXIF test suggested using a metadata scrubber. That is useful for ordinary still images, but a mobile upload contract must define every asset it sends. An iOS Live Photo can include both a photo resource and a paired video resource. The failure case is simple: the JPEG derivative has no GPS EXIF, while metadata or an original file associated with the paired video survives in the upload or retry path. Build the resource inventory first: let resources = PHAssetResource . assetResources ( for : asset ) for resource in resources { print ( resource . type , resource . originalFilename ) } Then make privacy assertions per output, not per UI selection: Artifact Required check still derivative no EXIF GPS/location fields paired video derivative no location metadata upload manifest only derived filenames retry queue no path to original resources temporary directory deleted after success or cancellation My lifecycle test would start an upload, force the app into the background after the still image is prepared, kill it while the paired video is processing, and relaunch. Recovery must either regenerate both safe derivatives or delete the incomplete pair. It must never mix a scrubbed still with an original video. Apple’s PHAssetResource API exposes the resources associated with a Photos asset. That enumeration should become evidence in the test: fail when an unexpected resource type is present rather than silently uploading it. Also verify what reaches the server. Device-side inspection alone misses multipart manifests, queued originals, and server-generated previews. Download the stored pair in a test environment and run the same metadata assertions again. A “remove location” button needs a precise scope. For Live Photos, the user reasonably expects it to cover the complete paired asset and every retry copy—not just the JPEG they can see.

A saga test that stops after “payment succeeded, inventory failed, refund called” assumes compensation is reliable. It is another distributed operation, so test it under the same faults. Use this sequence: 1. payment capture succeeds 2. inventory reservation times out 3. refund succeeds at provider 4. refund response is lost 5. compensation message is delivered again 6. late inventory-failed event arrives again The invariant is not “refund endpoint called once.” It is: captured amount - confirmed refunded amount = final charged amount and final charged amount is never negative Persist an inbox record for consumed message IDs and an outbox record for each intended side effect. Give the provider request a stable idempotency key derived from saga and compensation step: { "sagaId" : "order-42" , "step" : "refund-payment-v1" , "idempotencyKey" : "order-42:refund-payment:v1" , "amount" : 4900 } A deterministic simulator should permute duplicate delivery, delayed acknowledgement, worker crash, and out-of-order events. After every run, assert one terminal order state, at most one economic refund, and a complete evidence trail. “Already refunded” must reconcile to success only after amount and payment identity match. AWS describes coordination choices and rollback behavior in its Saga pattern guidance . The implementation detail that deserves its own test is durable compensation progress: a process restart cannot erase whether the external side effect occurred. Alert on sagas stuck in compensating , but do not let an operator click “retry” with a new key. The runbook should first query provider state, compare amount and currency, and resume the same operation identity. Exactly-once delivery is not required to preserve money. Stable operation identity plus reconciliation is.

My upload CLI looked restart-safe until I tested two failures together: the process crashed at 63%, and the signed resume URL expired before restart. The checkpoint needs identity, not credentials: { "file" : "release.tar.gz" , "fingerprint" : "sha256:..." , "uploadId" : "up_123" , "localOffset" : 66060288 , "updatedAt" : "2026-07-19T12:00:00Z" } Do not persist the signed URL. On restart, exchange the durable upload ID for fresh authorization, query the server offset, then reconcile: const remote = await headUpload ( freshUrl ); if ( remote > file . size ) throw new Error ( " invalid remote offset " ); if ( remote !== checkpoint . localOffset ) { await saveCheckpoint ({ ... checkpoint , localOffset : remote }); } await sendFrom ( file , remote , freshUrl ); The server offset wins because a crash can occur after bytes are accepted but before the local checkpoint is renamed. Save checkpoints through write-to-temp plus atomic rename so a partial JSON file cannot destroy recovery. My fixture matrix includes: Failure Expected behavior crash after remote commit rewind/advance to remote offset expired URL refresh without creating a second upload changed local file stop on fingerprint mismatch missing remote upload ask before starting over checkpoint write interrupted retain previous valid checkpoint The tus resumable upload protocol specifies offset discovery and conflict handling that are useful even if the service uses a smaller custom protocol. The key idea is explicit reconciliation, not assuming client memory is authoritative. I also shipped upload status , upload forget , and an exportable checkpoint directory. Recovery is a user-facing feature; if users cannot inspect or remove state, “resumable” becomes hidden lock-in.

Multipart finalization is not the only retry boundary. Abort can succeed in object storage while the HTTP response is lost, leaving your database convinced the upload is active. Model abort as a state transition, not a button wired directly to an SDK call: active → aborting → aborted └──→ cleanup_pending The endpoint should own an idempotency key and durable intent: await db . transaction ( async tx => { const upload = await tx . lockUpload ( uploadId ); if ( upload . state === " aborted " ) return ; await tx . markAborting ( uploadId , idempotencyKey ); await outbox . enqueue ( " abort-multipart " , { uploadId , storageKey }); }); A worker calls storage. If a retry receives NoSuchUpload , do not blindly fail: reconcile whether the upload was already completed, already aborted, or expired. Only the “already absent because abort succeeded” path may converge to aborted ; completion requires a separate terminal state. Test this sequence: Step Fault Required invariant persist abort intent process dies outbox resumes work storage abort succeeds response is lost retry does not reactivate upload duplicate message delivered twice one terminal state client retries same key same operation result cleanup scan stale active row reconcile before deleting Amazon S3’s AbortMultipartUpload API notes that in-progress part uploads may still succeed around an abort and recommends verifying that parts are gone. That makes a post-abort verification pass part of correctness, not optional housekeeping. Expose abort_requested_at , attempt count, last storage result, and remaining-part count. Alert on age in aborting , then run a lifecycle cleanup policy as a backstop—not as a substitute for application reconciliation.

Removing EXIF GPS protects one privacy boundary. It does not prove that an uploaded JPEG contains only a JPEG. A useful hostile fixture is a valid image followed by bytes for another format. Many decoders stop at JPEG’s end marker and display the picture successfully, while downstream scanners, content sniffers, or accidental downloads may interpret the trailing payload differently. Build a fixture set rather than trusting extensions: clean.jpg valid JPEG exif-gps.jpg valid JPEG with location jpeg-plus-zip.jpg JPEG followed by ZIP bytes jpeg-plus-html.jpg JPEG followed by HTML truncated.jpg missing end marker oversized-dimensions.jpg small file, dangerous decode cost My upload gate has independent layers: Store the original in a non-public quarantine location. Enforce byte-size and decoded-dimension limits. Decode with a maintained image library. Re-encode pixels into a new file, discarding metadata and trailing bytes. Verify the output’s signature, dimensions, and complete parse. Publish only the derived object under a server-generated name. A boundary check can detect bytes after the end marker, but re-encoding is the stronger transformation because it creates a new representation from decoded pixels. Keep the original inaccessible and delete it according to a tested lifecycle. assert ( outputSize <= limit ); assert ( decoded . width * decoded . height <= pixelBudget ); assert ( parseConsumesEntireFile ( output )); assert ( noLocationMetadata ( output )); The OWASP File Upload Cheat Sheet recommends defense in depth: allowlisted types, generated filenames, size limits, storage separation, and content validation. No single MIME header or metadata scrubber replaces that chain. The important privacy lesson is broader than EXIF: inspect every representation that survives the upload pipeline, and publish only a constrained derivative.

An accessible combobox can follow the correct ARIA pattern and still become unusable when two search responses arrive in the wrong order. Reproduce this sequence: Type ca , then quickly type cat . The cat response arrives first and highlights cat facts . The slower ca response arrives and replaces the list. aria-activedescendant now points to an option that no longer exists. IME input adds another boundary: searching during composition can send partial text the user has not committed. I would model the request generation explicitly: let generation = 0 ; let composing = false ; async function search ( query : string ) { const mine = ++ generation ; const options = await fetchOptions ( query ); if ( mine !== generation || composing ) return ; render ( options ); restoreActiveOptionByKey (); } The stable key matters. An array index cannot preserve the active option when ranking changes. Regression matrix Input Injected failure Expected evidence ca → cat first request delayed only cat results render Arrow Down result refresh active key survives or resets visibly Escape response arrives afterward popup stays closed IME composition network is fast no request until compositionend A Playwright test should assert focus remains on the input, every aria-activedescendant resolves to a live element, and Escape invalidates outstanding generations. A manual screen-reader pass should confirm result-count announcements are not emitted for discarded responses. The WAI-ARIA Authoring Practices combobox pattern defines the keyboard contract. The missing production step is testing that contract under asynchronous replacement, not only against static example data. Which stale-response failure has been hardest to reproduce in your search UI?

GenPage is a generative AI system developed by Netflix to replace its traditional multi-stage recommendation pipeline by directly generating personalized user homepages. GenPage leverages user history and request context as a prompt to produce the entire page, resulting in improved user engagement and reduced serving latency. By Sergio De Simone

This is a submission for DEV's Summer Bug Smash: Clear the Lineup , powered by Sentry . Project Overview I built LeakLens Control Room to keep AI away from the financial math. Python calculates every dollar. The optional model explains the evidence. A human decides whether to act. LeakLens is a live, functional restaurant margin-review application built with Python, SQLite, HTML, CSS, and JavaScript. It uses synthetic data in the public demo. The optional application narrative provider defaults to GPT-5.6, but every rate, threshold, ranking, and dollar scenario comes from deterministic code. Then a date string proved that deterministic code can still lie. I had let a raw CSV field decide which operating day counted as "current." Bug Fix or Performance Improvement The symptom LeakLens groups channel metrics by date, sorts those dates, and analyzes the last one against prior days. That works when every input is canonical ISO data such as 2026-07-09 . The loader only checked that the date field was not blank. This input slipped through: metrics = [ DailyMetric ( " 2026-7-9 " , " direct " , 500 , 250 , 0 , 0 , 125 ), DailyMetric ( " 2026-07-10 " , " direct " , 1000 , 50 , 0 , 0 , 250 ), ] July 10 is later than July 9. Python's string ordering sees something else: sorted ([ " 2026-07-10 " , " 2026-7-9 " ]) # ['2026-07-10', '2026-7-9'] LeakLens selected the final string, 2026-7-9 , as the current day. That mistake spread through the report: expected current date: 2026-07-10 actual current date: 2026-7-9 actual gross sales: 500.00 sales-loss scenario: 500.00 discount scenario: 225.00 The input had not shown a current sales collapse or discount leak. The analyzer compared the days backward and produced $725 of false scenario impact from synthetic data. For a restaurant operator, this is worse than a visible crash. A crash asks for attention. A plausible financial report can send someone to audit the wrong promotion, question the wrong manager, or change a schedule that was

AI coding agents have dramatically shortened the distance between an idea and working software. They can inspect a project, create files, run commands, write tests, and help diagnose failures. What they have not eliminated is judgment. A polished screen is not proof that data survives a reload. A passing unit test is not proof that keyboard users can complete the core task. A successful deployment is not proof that the intended commit reached production. This is why I use proof gates : observable conditions that must be satisfied before a product claim becomes stronger. A proof gate is not a meeting, a long document, or an excuse to slow down. It is a compact question: What evidence would let another person verify that this claim is true? Here are five gates that separate a persuasive AI demo from a small MVP you can responsibly ship. Gate 1: Prove One Valuable User Loop AI makes feature generation cheap, which makes uncontrolled scope especially dangerous. Before requesting code, define one primary user in one specific situation. Then describe: Their observable before-state The smallest useful action they can take The immediate result The reason they might return This becomes the product’s core loop. For example, “build a productivity platform” is too broad. A more testable loop might be: A freelancer remembers a useful client outcome. They record the outcome and supporting evidence. The record appears in a searchable library. They can retrieve it later for a proposal or review. The gate is not passed because a form exists. It is passed when a new user can complete the entire loop and explain what changed without coaching. Write a Not Today list alongside the required capabilities. Authentication, dashboards, collaboration, billing, and AI-generated summaries may all be reasonable later. They should not compete with proof of the first useful loop. The goal is not the fewest possible features. It is the smallest complete behavior that tests whether the product creat

Originally published on hexisteme notes . I have a small pipeline that crossposts my notes to dev.to. It parses a Markdown file's front matter, builds a payload, and calls the dev.to API to publish. It has 19 gate tests, and every one of them was green the whole time it was shipping. It published nine articles. All nine went live with their titles broken — the front-matter quotes were sitting right there in the title, visible to anyone who looked, for nine days, and nothing in the pipeline noticed. I didn't notice either. A human had to open the dev.to profile page by accident before anyone found out. This is the postmortem, and the reason I'm writing it up as a general essay rather than just a fixed-bug log is that the root cause isn't specific to dev.to, or to Markdown front matter, or to Python. It's a category of mistake that any pipeline with an external endpoint on the other end can make: testing the payload you build, and never testing what the other system does with it. The pipeline that had "passed everything" The shape of it is ordinary. A draft file has YAML-style front matter — title: "Some Title" — because that's the convention. A parser reads the front matter and pulls out the title. A payload builder takes that title and a few other fields and assembles the JSON body for the dev.to API. The API gets called, dev.to accepts it, the article is live. Nineteen gate tests cover this path — the front-matter parsing and the payload/API contract of the pipeline's own code. All green, every publish. The gap is in what "parses the front matter" actually means. The parser isn't a real YAML parser. It's closer to line.partition(":") — split each line on the first colon, take the right-hand side as the value. That works fine for tags: testing, devops where there's nothing to unwrap. It does not work for title: "Some Title" , because the quote characters are part of the string on the right-hand side of the colon, and a partition-based parser has no concept of "this

Most A/B testing happens in the browser: a script swaps a headline or button color after the page loads. That works for surface-level UI changes, but it cannot test the logic that runs before a page is ever rendered — a pricing algorithm, a search ranking model, a checkout flow, or a backend API response. Server-side A/B testing moves the experiment decision into your application code, where you control the full request lifecycle. This guide explains when to test server-side, how it differs from client-side testing, and how to implement it with Optimizely Feature Experimentation, including working SDK code for Node.js and Python. What Server-Side A/B Testing Is In a server-side A/B test, your application server decides which variation a user sees and renders the response accordingly. Instead of shipping the control experience and patching it in the browser, the server already knows the assignment by the time it builds the HTML, the JSON payload, or the rendered component. The decision is deterministic: a given user ID is consistently bucketed into the same variation, so the experience stays stable across requests and devices. Your code branches on that assignment, serves the corresponding experience, and reports a conversion event when the user does something that matters — a purchase, a signup, a search that returns a click. This is the model Optimizely calls Feature Experimentation . If you have used Optimizely before, you may know this product by its former name, Full Stack — the SDKs, datafile, and decision model are the same lineage, now under the Feature Experimentation name. Searchers still look for "Optimizely full stack," but the current product and documentation use Feature Experimentation. When to Test Server-Side Server-side testing is the right tool when the thing you are changing is not a cosmetic, post-render tweak. Reach for it in these situations: Backend logic and algorithms. Recommendation engines, search ranking, fraud scoring, feed ordering, and

I wrote a cleanup routine that matched processes by command line with a wildcard pattern. It reported success on every run. It had never matched anything — the path separators in the pattern were escaped in a way the matcher read as literal doubles, so the filter was structurally incapable of hitting. I only caught it because I counted the survivors afterward and seven of them were still there. The fix was switching from a wildcard match to a plain substring containment check with no escape semantics at all. A filter that cannot fail loudly will lie to you politely forever. Before trusting any matcher, feed it a known-positive and watch it fire — a green result from an instrument you never saw go red is noise. What's the equivalent lesson your worst bug taught you?

The moment we realized our staging environment was broken It was 3 PM on a Thursday, and our team was scrambling. A critical API change had just been merged to main, but the staging environment—our supposed safety net—was showing false positives. The integration tests passed, but the mobile app was completely broken in production. That's when we knew: our shared staging environment was failing us. The Problem: Shared Staging Is Broken by Design Like many engineering teams, we operated with a single, shared staging environment. Every developer deployed their changes to the same place, leading to: Deployment conflicts: "Who deployed that breaking change?" Cascading failures: One broken PR would block the entire team Test contamination: Data from one test would leak into another Delayed feedback: You'd only discover issues after merging your PR and deploying to staging The "works on my machine" syndrome, now at scale The worst part? Our API contracts were changing constantly, but we only discovered breaking changes during integration testing—often too late. The Solution: Ephemeral Environments per PR We made a radical change: every PR gets its own isolated, short-lived environment. Here's our architecture: Our Implementation Stack Infrastructure: Kubernetes (EKS) with namespace-per-PR Orchestration: Custom GitHub Action workflow Database: Isolated RDS instance per environment Contract Testing: Pact flow + OpenAPI validation Cleanup: AWS Lambda that runs every hour, destroying environments older than 2 hours The Game Changer: Automated Contract Testing The magic wasn't just in isolated environments—it was in what we did with them. Every time a PR deployed to its ephemeral environment, we ran: Consumer-Driven Contract Testing (Pact) Our mobile and web clients would verify their expectations against the actual deployed API. If a change broke what the client expected, the PR would fail. Provider Contract Validation We'd automatically verify that the deployed API matched ou