标签：#Testing

The .NET 10 JSON console logging change is small enough to miss during an upgrade: the formatted message still exists, but a typical record no longer duplicates it at State.Message . A collector, script, or snapshot test that reads only that nested property can start returning null while the application continues logging normally. I treat console JSON as a schema whenever another process parses it. That means a runtime upgrade deserves a contract test, not just a visual check in a terminal. The practical fix is to read the top-level Message , keep State for structured values, and retain a narrow fallback for older records. Why .NET 10 JSON console logging breaks nested-message parsers Before .NET 10, a normal AddJsonConsole record commonly repeated the rendered text: { "Message" : "Order 42 moved to ready." , "State" : { "Message" : "Order 42 moved to ready." , "OrderId" : 42 , "Status" : "ready" , "{OriginalFormat}" : "Order {OrderId} moved to {Status}." } } In .NET 10, the typical shape keeps one rendered message at the top level: { "Message" : "Order 42 moved to ready." , "State" : { "OrderId" : 42 , "Status" : "ready" , "{OriginalFormat}" : "Order {OrderId} moved to {Status}." } } Microsoft documents this as a behavioral breaking change and recommends that parsers use the top-level property. The official compatibility note also gives an essential caveat: State.Message may still appear when its content differs from the top-level value. I therefore do not reject a record merely because both properties exist. This is not a loss of structured logging data. OrderId , Status , and {OriginalFormat} remain useful fields inside State . The part that changed is where a consumer should get the rendered sentence. Prefer the top-level Message and keep State structured A legacy-only extractor is brittle because it assumes the duplicate is the contract: static string ? ReadLegacyOnly ( JsonElement root ) => root . TryGetProperty ( "State" , out var state ) && state . TryGetPro

Traditional test automation frameworks often carry heavy maintenance costs, slow release cycles, and high knowledge dependency. By transitioning from standard script creation to a governed AI Test Automation Factory , engineering teams can shift their focus from writing boilerplate code to high-value validation and architectural optimization. Here is an architectural breakdown of how multi-agent AI systems, governed telemetry, and Model Context Protocol (MCP) transform enterprise quality engineering. The Problem: The 45-Hour Manual Bottleneck Building a end-to-end BDD automation suite manually requires significant time per user story—often taking up to 45 hours across five distinct steps: Context Generation & Requirements Review (~8 hrs) Manual Test Case Design (~9 hrs) Cucumber Feature File Creation (~8 hrs) Page Object Model Generation (~8 hrs) Step Definition Implementation (~10 hrs) This traditional workflow creates coverage gaps, inconsistent code quality, and defect leakage. The Solution: Multi-Agent AI Automation Pipeline Instead of relying on single prompts, an AI Test Automation Factory routes requirement artifacts (BRDs / User Stories) through specialized agents: [BRD / User Story] │ ▼ [Context Agent] ──► [Test Case Agent] ──► [Feature File Agent] │ [Automation Suite] ◄── [Step Definition Agent] ◄── [Page Object Agent] Context Agent: Parses acceptance criteria and enterprise domain knowledge. Test Case Agent: Auto-generates exhaustive test scenario matrices. Feature File Agent: Drafts standardized BDD Cucumber feature files. Page Object & Step Def Agents: Constructs clean design patterns (POM) and matching step implementations. Measurable ROI: Before vs. After AI By replacing manual generation with agentic workflows, the effort to automate a scenario drops from 45 hours to 9.5 hours: Phase Manual Effort AI-Driven Effort Time Saved Context Generation 8 hrs 2 hrs 75% Test Design 9 hrs 2 hrs 78% Feature File Creation 8 hrs 0.5 hrs 94% Page Object Creation 8 h

1. What is a Test Case? A Test Case is a set of steps, test data, conditions and expected results used to check whether a particular functionality is working correctly or not. Example: For a login page, enter a valid username and password and click Login. The expected result is that the user should successfully log in. 2. What is a Test Scenario? A Test Scenario is a high-level functionality or condition that needs to be tested. Example: "Verify Login Functionality" is a Test Scenario. Under this scenario, we can create multiple test cases like valid login, invalid password, empty username, empty password, etc. 3. What are Negative Test Cases? Negative Test Cases are used to check how the application behaves when invalid or unexpected data is given. Example: Entering an incorrect password or leaving the username field empty. The application should not crash and should show the proper error message. 4. What are Positive Test Cases? Positive Test Cases check whether the application works correctly with valid and expected input. Example: Entering a valid username and password should allow the user to log in successfully. 5. Relationship Between Test Case and Test Scenario A Test Scenario is a high-level requirement or functionality, while a Test Case contains detailed steps to test that scenario. Example: Test Scenario: Verify Login Functionality. Test Cases: Login with valid username and password. Login with invalid password. Login with empty username. Login with empty password. So, one Test Scenario can have multiple Test Cases. 6. What is Unit Testing? Unit Testing is testing individual units or components of software separately. Usually, developers perform Unit Testing. Example: If there is a function that calculates the total price, we can test that function separately to check whether it returns the correct result. 7. What is Integration Testing? Integration Testing is used to check whether two or more modules work correctly after they are combined. It mainly foc

I founded Krasyn, an outpatient EMR with an AI scribe inside it. Krasyn has run a working outpatient clinic's real patient records since March 2026, so what our scribe drafts ends up in charts that real clinicians sign. This post covers two things we shipped in August: a published benchmark of how faithful those drafts are to the transcript, and Note Check, a tool that reads any scribe's note against its transcript and lists what the transcript does not support. Why a fluent note is the problem A faithful note and a note with one invented blood pressure look the same on the screen, and the clinician who signs it owns every sentence. Published evaluations put ambient-scribe hallucination at about 1 to 3 percent of notes. A March 2026 analysis of 71,173 AI-drafted and finalized note sections found a confirmed edit in 5.8 percent of them. The drafting got automated. The checking did not. I wanted a number for our own scribe that I could defend, with the definitions printed next to it. A benchmark without definitions is marketing. The unit: a clinical assertion We measure at the level of a clinical assertion, one atomic statement about the patient that could be true or false on its own. "Denies fever, chills, and nausea" is three assertions. A measurement and its value are one. Hedging is kept verbatim. Every assertion gets exactly one label against the transcript: Supported: the transcript says it, or it is a faithful paraphrase or clinical translation. Inferred: not stated, but a reasonable clinical inference with a basis in the transcript. Tracked separately because it is the contested category. Unsupported: no basis in the transcript at all. Contradicted: the transcript says the opposite, including a symptom the patient denied, a treatment the clinician declined, or another person's symptom attributed to the patient. Hallucination rate is unsupported plus contradicted over all assertions. Coverage is measured separately against key facts per case, because a note tha

If you are building AI-powered test automation, you may eventually run into this question: Should we use RAG or MCP? The question sounds reasonable, but it is slightly misleading. RAG and MCP solve very different problems. In testing, you will probably need both. The Problem With AI-Generated Tests LLMs can already generate Selenium, Cypress, and Playwright tests from natural-language prompts. Ask: Test the login flow with valid credentials. and an AI can produce a reasonable script. But there is a problem. The AI does not automatically know: Your actual business rules Existing test cases Previous defects Test data API behaviour High-risk workflows Team-specific automation standards It knows how testing works , but not necessarily how your product works . That is where RAG becomes useful. What RAG Actually Solves RAG gives the AI access to project-specific information. Instead of working from a generic prompt, the model can retrieve relevant: Requirements Test Cases API Docs Bug History Business Rules Existing Automation Test Data Now consider the same request: Test the checkout flow. Without RAG, the AI may create a fairly standard checkout process. With RAG, it could first learn: Which payment methods are supported Whether guest checkout is allowed Which validations are required Which checkout bugs appeared previously Which scenarios already exist The generated test becomes much more relevant. But there is still a limitation. Knowing what should happen does not mean the AI can actually test it. That Is Where MCP Comes In MCP gives an AI system access to external tools. For browser testing, that could mean allowing an AI agent to use Playwright capabilities to: Open Page ↓ Inspect UI ↓ Enter Data ↓ Click ↓ Observe Result ↓ Validate So the difference is simple: RAG gives the AI context. MCP gives the AI capabilities. Or even shorter: RAG = What does the AI know? MCP = What can the AI do? Why This Matters for Test Automation Imagine an AI receives this instruction: C

A manual psql check answers exactly one question: does this policy work right now, against today's schema, with today's roles. It says nothing about tomorrow. Three ordinary changes are enough to quietly break tenant isolation without anyone noticing at review time. A migration that drops and recreates a table loses RLS entirely, since it's a per-table flag, not something that travels with column definitions. A new service role for a background job can skip the policy if nobody remembers to apply it. And the most common one: someone grants BYPASSRLS during an incident and never revokes it. Most guides point you at pgTAP here and stop. pgTAP is fine, but it's a separate SQL-based framework with its own runner. If your backend is already on Jest, you don't need a second test framework, you need a Jest test that actually proves a leak can't happen. The core pattern: seed a row as tenant A, query as tenant B, assert the result is empty. Run it through a dedicated low-privilege role, since table owners and superusers bypass RLS by default even with FORCE enabled for the owner. I break down the full pattern, the queryAsTenant helper, testing WITH CHECK on INSERT/UPDATE, catching accidental BYPASSRLS grants, and wiring it into GitHub Actions here: https://devencyclopedia.com/blog/postgres-rls-testing-jest If you're doing this across more than one or two tables, I also built RLSBuilder, a browser tool that generates the CREATE POLICY SQL and a matching Jest test from the same three inputs so they can't drift apart: https://devencyclopedia.com/tools/rls-builder

I had 87% coverage, and we still broke the billing flow on launch day. Not because of a gap in the percentage. Because 87% was covering the wrong things. The tests were written to pass a gate, not to catch a failure. That is a more common story than most teams admit. And the reason it keeps happening is not that engineers are careless. It is that the incentive structure you created made it the rational outcome. The series checkpoint The first three articles in this series built the investment case for testing and then dismantled the received wisdom about how to execute it. We've made the economic argument for automation. We've restructured when quality checks happen across the SDLC. We've replaced the pyramid model with something shaped by risk rather than by code hierarchy. Now, when someone asks: how do you know if it is working? The answer most teams give is their coverage percentage. This article is about why that answer is structurally broken, and why fixing it is a management decision before it is a tooling decision. What coverage percentage actually measures Coverage percentage tracks which lines of your code were executed during a test run. If a line ran, it counts as covered. That is the complete definition. It does not measure whether the test asserted anything meaningful about that line. It does not measure whether both branches of a conditional were exercised. It does not measure whether the specific inputs that cause failures were ever tried. A test that calls a payment function and checks assert response is not None covers the same lines as a test that validates the transaction ID, amount, currency, error code, and retry behaviour. The coverage tool treats them identically. The research on this is unambiguous. A 2017 study by Kochhar et al. examined the correlation between code coverage and actual bug rates across 100 large open-source Java projects. The finding: the coverage of existing test suites has an insignificant correlation with the number of b

Originally published on hexisteme notes . I run pre-registered checks against a live system, read the verdict, and move on — that's the whole point of pre-registering them, so I don't get to argue with the result after the fact. Most of the time the discipline pays for itself. This time it passed, and the pass was wrong, and the reason it was wrong is more interesting than the failure itself: the check could not have returned anything else, whatever had actually happened to the file under test. The question I was probing something narrow: does a hand-made audio crossfade survive a round trip through DaVinci Resolve? Build a timeline with a crossfade sitting on a cut, export it to FCPXML 1.10, re-import it, and see whether the crossfade is still there. Third-party documentation says transitions are invisible to and unmodifiable by the scripting API. Believing that, I pre-registered a judgment method that never looks at timeline structure at all: render audio around the splice and classify it by waveform shape. The judge, exactly as pre-registered: render two seconds either side of the cut, downsample to 8 kHz mono, compute a 20 ms sliding-window RMS envelope — 202 windows across the render — and take the largest normalized step between adjacent windows. Above 0.5, call it a hard cut: the fade is gone. Below 0.5, call it a gradual ramp: the fade survived. The probe came back pass — gradual ramp, max step 0.4761, under the 0.5 threshold. Exit 0, all green. The crossfade had actually been lost at the export step. The pass was a false confirm, and I only found that out by going back in with a second, read-only inspection after the fact. Why the check could not fail The prep instructions for this probe — which I also wrote — said the easiest way to get two adjacent audio items with enough handle to build a crossfade is to take one continuous clip and blade-split it in the middle. That's a completely reasonable instruction on its own. A crossfade needs overlap media on bot

I built a pytest harness that runs the same set of questions through five language models at once - a free local Llama, plus GPT, DeepSeek, and two Claude models - and compares them on the three things a team pays for: cost per query, speed, and answer quality. The plan was simple. Run the grid, read the scoreboard, say which model to use. The scoreboard came back clean and easy to read. This is the story of why I didn't trust it, and what I found when I checked. The thing I stopped trusting wasn't any of the models. It was the tool I was using to score them. It's also the first project in this series that spends real money. Every one before it ran locally, for free. Here each call costs something, and the whole comparison came to about 21 cents. That price is small, but it changed how I tested, and not in the way I expected. The scoreboard, and why I didn't stop there Five models, the same ten questions, twice each, every call measured. Here is the run, ordered by quality score (a second model grades each answer on correctness and relevance, combined into a 0-1 score, pass line 0.7): model quality mean $/query mean latency out-tokens deepseek-v4-pro 0.970 $0.000138 2713 ms 113 claude-haiku-4-5 0.967 $0.000537 1597 ms 104 gpt-5.6-luna 0.962 $0.000082 1323 ms 65 claude-sonnet-5 0.937 $0.002426 4093 ms 239 llama3.2 (local) 0.922 $0.000000 7859 ms 130 Read it straight and it looks finished. The whole quality column sits in a tiny band, 0.92 to 0.97. The cheapest, fastest paid model scores right in there with the rest. The most expensive one, Sonnet, at about thirty times the price per query, sits no higher than the others - its answers are just longer (239 tokens to GPT's 65), which costs more and takes longer without scoring better. So the easy takeaway is: use the small cheap model, skip the expensive one. I want to be careful with that, because it's the kind of tidy result I've learned to distrust. The gaps between the top models are tiny, and a ranking built on tin

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. The...

The Forked History: Byzantine Witness and the 3-of-4 Quorum — Tested Agent Determinism Illusions (Part 19) 2026-08-20 Where this fits: Part 18 closed the runtime face of C3's boundary at capability isolation — the oracle reads from a surface the producer cannot write. Part 18's §6 named the residual this part answers: Byzantine authority . Sealing one honest oracle's history says nothing about whose view is the truth when a compromised authority can present forked views to different observers. This part maps the witness layer — the answer is not a stronger single authority, but a witness set with three separable properties, an explicit fault bound, and a governed membership surface. Part 18 ended with the oracle isolated from the producer's writable surface. Isolation answers "can the producer fake the read?" It does not answer "whose read is the truth when the authority itself equivocates?" A trusted parent that closes the verdict channel still presents the approval history. If that authority is compromised — or the CI choosing the harness is — it can show job A a signed checkpoint and job B a different fork; each job sees a locally valid tree head with an inclusion proof, and no single history exists. This part tests that shape, then the witness machinery that answers it. 1. Sealed floor ≠ global history Peter's pin-rollback reply sealed a monotonic minimum-version floor: CI could no longer resurrect an older harness with a known false-green channel. The sealed floor is honest for what it claims. It is not what it looks like at first. The split into two predicates. The sealed floor proves this job did not go backwards on the view it was shown. It does not prove the approval history itself is one global append-only log. A compromised authority can hand job A a signed checkpoint whose minimum is 2 and job B a fork whose minimum is still 1; each view carries a locally valid signature and an inclusion proof while no single history exists. cell setup result A local sea

I gave my LLM a 29-question order-reading exam. Last time was how to build the exam. Today: grading. Grading gets its own post for a reason. Build the grading wrong, and the score lies to you. 5 wrong out of 29 — can I ship? No idea. Because "which 5" is missing. If it missed 5 typo-riddled questions, ship it. But if one of those 5 was reading "please cancel my order" as a NEW order? Then even with everything else perfect, you can't ship. That program sends goods to a customer who just cancelled. So don't grade by count. Grade by severity. Severity = "can a human undo this?" My grader has 4 grades. One criterion — is it reversible? In this program, the irreversible moment is when the wrong goods get loaded onto a truck. FATAL Wrong goods on the truck. Cannot be undone RISKY Confirmed something ambiguous without asking. Right this time — fatal next time MISSED Dropped an order. The customer calls. Fixable HARMLESS Over-asked "please confirm." Just slower One principle falls out of this: A wrong confirmation is worse than no confirmation. Sounds obvious. In production you'll be tempted to flip it. Someone complains "it asks for confirmation too often," so you lower the confidence bar. The screen gets cleaner. And the accidents start happening off-screen. The same 28/29 splits two ways FATAL 0 · MISSED 1 → Ship it. Humans catch what it drops FATAL 1 · everything else perfect → Don't ship. You don't know when that 1 comes back Same score. Opposite fates. Two accidents my grader caused The grader is code I wrote. Like all code I write, it had bugs. Accident one — zero points over formatting. A model answer was perfect in content, but the JSON wrapper arrived with the tail cut off. The grader ruled "broken format = fatal." A 100-point answer, zeroed over one missing brace. The fix is simple: count the open brackets and close what's missing (ignoring brackets inside strings). The actual code is in parse_json in the repo . Accident two — penalizing a good answer. For "250 b

Last time I gave my LLM an order-reading exam and lost 5 times as the exam author. Today: how that exam was built. Conclusion first — nice questions are a waste of paper. You'll want to start with the happy path Ask anyone to write a test and they start with the case that works. "5 boxes of the 250 shipping boxes please" → shipping box 250, 5 boxes. It passes. Feels good. Reassuring. But that's wasted points. Models rarely fail the normal cases. What fails is everything that isn't normal. My 29 questions broke down like this: Normal orders 4 Things that aren't orders 6 ← the biggest group Changes & cancellations 4 Ambiguous ones 5 Typos & extreme shorthand 3 After learning kicks in 7 Normal is the smallest group. On purpose. Why "not an order" gets the most questions The worst accident for this program is shipping something nobody ordered. So the exam should aim at that accident more than anything else. What are the dimensions of the 250 shipping box? Product name: present. Number: present. But it's not an order. It's a question. A program that treats "product name spotted" as "order detected" calls the truck right here. So I planted six of these: price inquiries, stock inquiries, delivery questions, greetings, a tax-invoice request. Changes and cancellations are nastier. I ordered 5 boxes of the 250 — please send only 3 Two numbers. Read only the first half and it's a perfect order. Treat it as a new order and the goods ship twice. Plant traps in the catalog too It's not just about hard questions. Make the data itself messy. Two kinds of clear tape — 48mm and 60mm Five products starting with "250" Different pack sizes per box — 50, 40, 25, 10 sheets A few loose items with no box unit at all One reason: real data already looks like this. A real product catalog always has near-twins. Run the exam on a clean catalog and here's what happens — everything passes. Then you plug in production data and it collapses. If the exam passed but production has accidents, that's no

The MCP C# SDK hybrid sessions option solves an awkward upgrade boundary: some clients still use the 2025-11-25 initialize handshake and depend on sessions, while clients on 2026-07-28 expect every HTTP request to stand alone. I want both groups to reach one ASP.NET Core endpoint without making modern clients downgrade or stripping useful behavior from legacy clients. The stable C# SDK 2.2.0 release added exactly that path with HttpServerSessionMode.StatefulForInitializeClients . The release notes describe it as hybrid stateful/stateless serving, and the official session-mode guide spells out the per-request behavior. Why one global session switch fails The 2026-07-28 MCP revision removed the initialize handshake and Mcp-Session-Id from its wire format. Client identity, capabilities, and protocol version travel with each request instead. The final specification announcement explains why the core moved toward request/response statelessness. That creates a migration choice for an existing server. With HttpServerSessionMode.Stateful , initialize-era clients receive full sessions. A modern request is refused so a dual-path client can fall back to the older handshake. Compatibility is preserved, but the client does not use the new protocol natively. With HttpServerSessionMode.Stateless , every request is independent. That is the right default for servers that do not need session state, unsolicited notifications, resource subscriptions, or older server-to-client flows. It may be too abrupt when deployed clients still rely on those features. Hybrid mode makes the decision from the incoming request instead of applying one choice to the endpoint. Configure MCP C# SDK hybrid sessions The server configuration is deliberately small: builder . Services . AddMcpServer () . WithHttpTransport ( options => { options . SessionMode = HttpServerSessionMode . StatefulForInitializeClients ; }) . WithTools < DemoTools >(); app . MapMcp ( "/mcp" ); An initialize-era client sends an initial

You picked a free model because the answers looked good. Good answers are not an endpoint. An endpoint is the model plus the server plus the network. Demos pass. Pipelines stall. The model was rarely the problem. So why do we keep benchmarking only the model? Because it is easy. You paste a prompt. You read the output. You declare a winner. The server never gets a vote. This post is a reproducible benchmark. It measures the pair, not the model. Run it before you wire any free endpoint into CI. The Pair, Not the Model Most evaluations compare answers. You paste a prompt. You judge the output. You pick a winner. That measures the model. It ignores the server. Free model access usually means a shared endpoint. A free server option means shared tenancy. Other users share the CPU, memory, and network. Your latency is their latency. Your timeout is their timeout. Here is the scenario I keep seeing. A team evaluates a free model on Friday. The answers look great. They wire it into CI on Monday. By Wednesday, the pipeline is red. The model did not change. The server did. A neighbor started a batch job. Now every request queues behind it. I applied the same harness to MonkeyCode's free model access and their free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not trust the demo. I built a harness instead. The Harness A benchmark needs three things. A fixed prompt set. A concurrency ladder. A pass/fail table. Here is the harness I use. #!/usr/bin/env python3 """ Benchmark a model endpoint as a pair: model + server. """ import argparse import asyncio import json import statistics import time import httpx PROMPTS = [ " Say OK. " , " Classify this log line: ERROR disk full " , " Return one word: is 429 a retryable status? " , ] async def fire ( client , url , payload , sem , timeout = 30 ): async with sem : start = time . perf_counter () try : r = await client . post ( url , json = payload , timeout = timeout ) return r . sta

Conclusion first: a free model drafted a working C++17 directory hasher in one pass. The draft compiled, ran, and was still wrong. A differential test against standard system tools found three real bugs before the tool ever touched a production cache. Generation was the cheap part. Verification was the deliverable. Background I needed a deterministic hash of a directory tree. The use case was cache invalidation for a small build pipeline: if any file content, name, or symlink target changes, the cache key must change. If nothing changes, the key must stay identical across machines and across checkouts. Hand-writing the tool is maybe 200 lines of std::filesystem code. The happy path is easy. The risk lives in ordering, symlinks, and metadata leaking into the hash. I turned the task into an experiment. MonkeyCode's free model access and free server option meant the model ran on a remote server while I kept verification on my laptop. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The plan: let the model write the first version, then prove or disprove it against a reference oracle. The Contract The goal was not "a tool that compiles." The goal was a tool that matches a reference implementation on every input I could generate. I wrote the contract in three sentences: Same tree → same hash, on any machine. Different content, name, or symlink target → different hash. File metadata (mtime, inode) must not affect the hash. Implementation Step 1: the prompt. I gave the model the contract, the C++17 standard, and one constraint: a single file with no dependencies beyond the standard library. Step 2: the draft. The model returned one .cpp file in a single response. It compiled on the first try. That is the exact moment where most workflows stop. This one did not. Step 3: the reference oracle. Instead of reviewing the code line by line, I built a harness that compares the tool against a shell pipeline: find " $tree " -printf '%P\0' | sort -z | wh

MCP x-mcp-header validation is easy to miss because the annotation looks like ordinary JSON Schema metadata. On the 2026-07-28 Streamable HTTP transport, it is a wire contract: the client copies selected tool arguments into Mcp-Param-* headers, intermediaries can act on those headers, and the server checks them against the JSON-RPC body. I treat that contract as something to test before a tool reaches tools/list . A bad suffix, an unsupported type, or an unreachable annotation makes the whole tool definition invalid. Silently accepting it only moves the failure to a harder place to diagnose. Why the same value travels twice The final Streamable HTTP specification mirrors request metadata into HTTP headers so a load balancer, gateway, or WAF does not need to parse JSON-RPC. A server can add x-mcp-header to a tool property: { "type" : "object" , "properties" : { "region" : { "type" : "string" , "x-mcp-header" : "Region" } } } A call with "region": "us-west1" then carries: Mcp-Param-Region: us-west1 The official C# SDK can generate that schema from a parameter attribute: [ McpServerTool ] public static string ExecuteSql ( [ McpHeader ( "Region" )] string region , string query ) => $"Queued for { region } " ; Current C# SDK v2 tool documentation describes both schema generation and automatic header projection. The feature is on the stable v2 line; it is not necessary to pin an earlier preview or release candidate. MCP x-mcp-header validation rules The final tool definition rules are deliberately narrow. The annotation value must be a non-empty HTTP field-name token and must be unique without regard to case. Region and region therefore collide. Control characters, spaces, and separators such as a colon are not valid suffix characters. Only string , integer , and boolean properties can be mirrored. JSON Schema number is excluded, and integer values must stay between -(2^53 - 1) and 2^53 - 1 so every conforming implementation can represent the value exactly. Reachability i

Free model endpoints are seductive. Zero cost. Zero setup. Zero reason to trust them. I don't trust demos. I trust failure modes. So I built a small evaluation harness. It tests one thing: can a free model endpoint gate a pull request for secrets? This is not a benchmark. It's a repeatable experiment. You can run it in an afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model endpoint and the free server option for the test. No quotas. No hardware claims. Just a harness and a rubric. Why I stopped trusting free endpoints Free endpoints look great in a demo. You paste a diff. The model finds the secret. Everyone claps. Then you wire it into CI. The JSON breaks. The latency spikes. The model misses a private key. The demo didn't show that. An evaluation will. The experiment I designed a 40-minute test. It answers one question: where does the free endpoint perform well, and where does it break? The dataset is 30 synthetic diffs. Fifteen contain real-looking secrets. Fifteen are clean. Each diff is small. Each diff has one clear change. The prompt is strict. The model must return JSON. No prose. No apologies. Just a verdict. # eval_secret_gate.py # Simplified harness. Adapt to your client SDK. import json , time def classify ( client , diff : str ) -> dict : prompt = f """ You are a secret scanner for code review. Return ONLY JSON with this shape: {{ " contains_secret " : true, " line " : 12, " type " : " aws_access_key " }} Diff: { diff } """ start = time . time () response = client . complete ( prompt , model = " free " , server = " free " , # free server option ) latency = time . time () - start return { " latency " : latency , " raw " : response } def evaluate ( client , diffs , runs = 3 ): for i , diff in enumerate ( diffs ): for run in range ( runs ): yield i , run , classify ( client , diff ) The harness is deliberately small. It measures five things. Accuracy. JSON validity. Latency. Variance. Fa

I'd already done a lot right by the time I started writing tests for Beaver-Auth . Every module had gone through multiple rounds of deliberate review. Enumeration protection, hashed tokens, refresh rotation, TOTP replay defense — the design was solid, and I knew it was solid, because I'd thought hard about every piece of it. Then I wrote 238 tests against the actual code, and found 8 real bugs. Some of them were the kind that would have silently broken production on day one. This post isn't about the bugs specifically — it's about the gap between "I reviewed this carefully" and "this is shippable," and why that gap is bigger than most of us assume, even when the reviewing was genuinely careful. "Passing tests" and "shippable" are different claims Here's the trap I nearly walked into: I'd built a solid test suite covering the core auth flows — registration, login, verification — and every test passed. It felt done. But passing tests only tell you the code does what the tests expect. If the tests were written from the same mental model as the code, they'll happily confirm a bug is correct behavior, because both the code and the test agree on the same wrong assumption. The fix wasn't "write more tests." It was testing against the real, integrated system — not a hand-built mock of my own logic, and not testing modules in isolation from what actually calls them. A few of the bugs below only surfaced because a test exercised the real dependency chain instead of assuming it worked. Bug 1: TypeScript let an argument-shift bug compile clean This is the one that scared me most. Beaver-Auth dispatches background work (like sending a verification email) through a TaskDispatcher interface: interface TaskDispatcher { dispatch ( taskName : string , payload : unknown , handler : () => Promise < void > , onFailure ?: ( error : unknown ) => Promise < void > | void , ): Promise < void > } The default implementation had drifted to a different signature — missing the payload parameter e

Successfully onboarding an automated test suite to Harness CI requires configuring infrastructure placeholders, secrets, pipelines, and branch protection rules. Here is a 10-step checklist to help you onboard your end-to-end (E2E) automation pipelines seamlessly. Step 1: Replace Infrastructure Placeholders Ensure your pipeline YAML definitions (e.g., .harness/e2e-poc.yaml and .harness/e2e-regression-parallel.yaml) contain your specific environment values: ORG_ID: Harness Organization Identifier PROJECT_ID: Harness Project Identifier GIT_CONNECTOR: Harness Git Connector for GitHub Enterprise access APP_REPO_NAME: Target repository in owner/repo format K8S_CONNECTOR: Kubernetes connector for build infrastructure K8S_NAMESPACE: Kubernetes namespace where build pods run Step 2: Configure Environment Secrets In Harness, set up the following runtime secrets: CONNECT_URL CONNECT_USERNAME CONNECT_PASSWORD Step 3: Setup PR Validation Pipeline Import your short-run pipeline YAML into Harness. Save it as your PR Validation Pipeline. Run a manual validation test using runtime overrides: TargetEnv = qa cucumberTags = @smoke Step 4: Verify Artifact Generation Confirm that the initial execution correctly generates and uploads all required outputs: JUnit Report: reports/junit-report.xml Test Reports: reports/** Failure Artifacts: test-results/** (screenshots, traces) Step 5: Setup Nightly Parallel Pipeline Import your parallel pipeline YAML into Harness. Save it as your Nightly Regression Pipeline. Run a manual validation test with target concurrency parameters: TargetEnv = qa cucumberTags = @regression cucumberParallel = 4 Step 6: Configure Automated Triggers & Branch Protection PR Trigger: Configured on pull requests with cucumberTags= @smoke . Nightly Schedule Trigger: Configured on a nightly cron schedule with cucumberTags=@regression and cucumberParallel=4. GitHub Branch Protection: Enable branch protection on target branches requiring the Harness PR pipeline status check to p