标签：#validation

Running LLM jobs over hundreds of items, the obvious shortcut is to collapse execution and verification into one model pass: one call, one output, ship it. It fails at scale, and a scoring system for 146 countries across 11 categories shows exactly why. Each score runs 0 to 100 on a single canonical dataset, the overall rating is the arithmetic mean of those 11, and there are no per-country exceptions. One yardstick, applied identically everywhere. Ask a cheap model to generate all 146 in one pass and you get speed with a hidden cost: drift. One country's "friendliness" score reads high because the model read it as social warmth rather than visa bureaucracy. Another's culture score inflates after the prompt happened to emphasize food over history. None of these are bugs, they're quiet inconsistencies, and at 146 items a 5% drift rate means seven countries silently failing the canonicity requirement while every individual score still looks reasonable. The pattern Step one: a cheap executor runs the mechanical pass. Fixed ruleset, all 146 countries in parallel batches, structured JSON out. No judgment calls, just apply rule X to field Y. Step two: a stronger gate verifies before anything ships. Same scale everywhere? Any statistical outlier? Did a category get reweighted mid-run? This is judgment work, holding many items in view at once, and it's what a single combined pass can't do reliably. A model doing both jobs at once optimizes for the wrong thing: it second-guesses the ruleset mid-run, adds nuance where the spec demanded consistency, and marks cases "exceptional" that shouldn't be. Splitting the two roles is faster and cheaper than one model trying to hold both contexts simultaneously. Where the consistency requirement bites The Country Comparison Tool's best-travel-months field works the same way: a month qualifies if it scores 70 or higher on a fixed weather index built from Open-Meteo data, no editorial override, no "tourists usually go in December anyway."

A VAST wrapper with no AdSystem, no VASTAdTagURI and no Impression validates against the VAST 4.4 draft schema. The same document has been invalid in every version from 2.0 through 4.2. It is one line of XSD, and it is almost certainly a side effect of the CTV Ad Portfolio restructure rather than a decision anyone made on purpose. I have filed it with IAB Tech Lab. This post is the working, because the reproduction is short enough that anyone can check it in about a minute. The change In vast_4.4.xsd on master, both vastInLine_type and vastWrapper_type wrap their children in a single compositor: an xs:choice with minOccurs zero and maxOccurs unbounded. That looks harmless. It is the idiom people reach for when they want to say "these children may appear in any order". What it actually says is stronger than that. In XSD, the cardinality on the compositor governs the content model, and the minOccurs on the individual child elements only describes a single selection from the choice. Set the choice itself to zero-or-more and every constraint underneath it stops binding. So the children still declare minOccurs="1". They are still, in effect, optional. The compositor in question <!-- vast_4.4.xsd, vastWrapper_type and vastInLine_type --> <xs:choice minOccurs= "0" maxOccurs= "unbounded" > <xs:element name= "AdSystem" type= "vastAdSystem_type" /> <xs:element name= "VASTAdTagURI" type= "vastURIElement_type" /> <xs:element name= "Impression" type= "vastImpression_type" /> <xs:element name= "Creatives" type= "vastCreatives_type" /> <!-- ... --> </xs:choice> Three consequences, not one The empty wrapper is the headline, but the compositor gives up three separate guarantees at once. Each is reproducible with xmllint against the published schema. What now validates in 4.4 Everything is optional. An empty <Wrapper/> validates. So does an empty <InLine/> , with no AdSystem, no AdTitle, no Impression and no Creatives. Everything repeats. maxOccurs="unbounded" on the choice means any

The last lesson was about validating what a model hands you. The story behind it: a set of prompts that had returned real, criteria-matched vendors for weeks came back in staging with placeholder junk, literally the words Vendor A, Vendor B, Vendor C. So I built the validation layer, and the last gate in it is a model checking a model. Then FromZeroToShip asked three questions in the comments, and all three were about the gate rather than the model. That's the harder thing to look at, and I hadn't written all of it down. Here's the long version. What was on the fail list that I hadn't already been burned by? More than the question assumes, and not because I got clever about imagining failures. The placeholder output changed what I do with a failure . I stopped fixing the instance and asked what class it belonged to, and that class is a lot wider than "the model emitted example data." It's a suggestion that looks fine and isn't usable. Two of those I had never hit went in on the back of it: A vendor that's wrong for the category. A vendor that's no longer in business. Neither has anything to do with placeholder text, and both would sail through a schema check looking like a perfectly real answer. They also changed the prompt that produces the suggestions, not just the gate. Fixing only the failure I actually met would have left both of them live. So the list isn't purely retrospective. It grows by generalizing from the one failure you hit to the class it sits in, and it keeps growing from what the running system actually throws at me rather than from what I remembered to imagine. Is it foolproof? No. What's left is the case worth worrying about: results that read as real, pass the schema, satisfy every criterion I gave, and are still wrong. You can't validate the truth of a guess from inside the system. You can only lower the cost of it being wrong. That means a human in the loop at the stage where being wrong is expensive, the confidence surfaced so the answer is ch

A portfolio can look good on the usual scorecard and still answer the wrong question. One line says return was high. Another says risk-adjusted performance was acceptable. A third says drawdown stayed inside a tolerable range. Then the market turns, the benchmark starts recovering, and the thing I actually care about is different: how efficiently did the portfolio catch up? That is where a new metric can fool its own author. If I build a recovery measure and it moves almost exactly like an existing ratio, I have created a longer name for the same signal. The right acceptance test is geometric: a useful metric should cast a different shadow. This is the rule I used while validating Hyperlogarithmic Benchmark Catch-Up Ratio (HBCR): orthogonality to existing measures is a first-class test, not a chart for the appendix. 1. A new metric has to earn its axis HBCR was built to measure benchmark-relative recovery dynamics. The research page states the motivation plainly: traditional benchmark-relative metrics often fail to capture the true dynamics of investment performance, especially during market recoveries [ A New Metric for Private Equity Risk Adjusted Returns , Calibration of Risk and Correlation in Private Equity ]. That framing matters because the obvious validation path is tempting and weak. You compare the new number with familiar performance measures, find a comforting relationship, and declare victory. But a high correlation with a well-known score can be a warning. If HBCR strongly tracked Sharpe Ratio, it would probably be an expensive synonym for risk-adjusted return. The acceptance test I wanted was sharper. HBCR should have some relationship with performance, because recovery has economic content. It should also avoid collapsing into the same direction as Sharpe Ratio, Beta, Volatility, Alpha, Total Return, or Max Drawdown. Written as a predicate, the test has two sides. Let $\mathcal{T}$ be the set of metrics already on the scorecard, $\rho_{n,m}$ the corr

Express does not validate request input for you. Without a check at the edge, handlers get raw req.body , req.query , and req.params - strings where you expected numbers, missing fields, and shapes that only blow up deep in business logic. Zod is a TypeScript-first schema library. You declare the shape once, infer types with z.infer , and parse at the HTTP boundary so route handlers only see valid data. Invalid input becomes HTTP 400 before your code runs. This post covers Zod 4 schemas ( z.email() , z.uuid() , z.coerce ), Express validation middleware, error formatting and pitfalls. Prerequisites Node.js version 26 Zod 4: npm i zod Express: npm i express and npm i -D @types/express Zod 3 method forms like z.string().email() still work but are deprecated in v4. Prefer the top-level APIs below. Schemas // schemas.ts import { z } from ' zod ' ; export const createUserSchema = z . object ({ email : z . email (), name : z . string (). min ( 1 ). max ( 100 ), age : z . number (). int (). min ( 0 ). max ( 150 ). optional () }); export type CreateUserInput = z . infer < typeof createUserSchema > ; export const userIdParamSchema = z . object ({ id : z . uuid () }); export const listUsersQuerySchema = z . object ({ limit : z . coerce . number (). int (). min ( 1 ). max ( 100 ). default ( 10 ), q : z . string (). trim (). min ( 1 ). optional () }); z.coerce.number() is useful for query strings - HTTP query values arrive as strings. Prefer safeParse over parse at the edge so you control the HTTP status and response body. Format errors once Map ZodError.issues into a stable JSON body, or use Zod 4 helpers z.flattenError() / z.treeifyError() when you want field-keyed or nested shapes. // format-zod-error.ts import { ZodError } from ' zod ' ; export function formatZodError ( error : ZodError ) { return { message : ' Validation failed ' , issues : error . issues . map (( issue ) => ({ path : issue . path . join ( ' . ' ) || ' (root) ' , message : issue . message , code : issue . c

Stale data và cache stampede: vì sao TTL một mình không đủ và vì sao origin sập khi key hết hạn Stale data là dữ liệu trong cache đã lỗi thời so với source of truth. Nó xuất hiện vì cache và origin là hai bản sao, và bất kỳ cơ chế đồng bộ nào — TTL, event-based invalidation, versioning — đều có cửa sổ giữa lúc origin đổi và lúc cache biết chuyện. Cái giá phải trả trong production không chỉ là "user nhìn thấy giá cũ vài giây". Khi một key hot vừa hết hạn, hàng nghìn request cùng miss, cùng đâm xuống DB để tính lại — đó là cache stampede (dogpile, thundering herd), và nó đủ sức đưa origin xuống trong vài chục giây. Cơ chế hoạt động Có bốn cơ chế invalidation dùng thật: TTL (time-to-live). Mỗi entry gắn một hạn dùng. Hết hạn coi như miss, đọc lại từ origin. Đơn giản, không cần coordination giữa writer và cache. Nhược điểm: staleness bounded bởi TTL, và tất cả replica của cùng một key hết hạn cùng lúc. Event-based invalidation. Khi origin thay đổi, phát một event (thường qua pub/sub, CDC như Debezium, hoặc gọi trực tiếp DEL) để cache xoá hoặc cập nhật entry. Fresh gần như realtime, nhưng đòi hỏi coupling giữa write path và cache — writer phải biết mọi key phái sinh từ dữ liệu vừa đổi. Versioning (cache key có version). Key gắn version của dữ liệu, ví dụ user:123:v42 . Đổi dữ liệu thì tăng version, key cũ tự nhiên bị bỏ qua, không cần xoá gì. Kỹ thuật này còn được gọi là generational caching; Rails cache dùng cách tương tự với cache_key_with_version . Single-flight (request coalescing). Không phải invalidation, mà là cách xử lý miss: khi N request cùng miss cùng một key, chỉ một trong số đó được phép gọi origin, các request còn lại đợi kết quả của nó. Go có golang.org/x/sync/singleflight implement sẵn pattern này; Facebook memcache dùng "leases" (paper của Nishtala et al., NSDI 2013) cho cùng ý tưởng ở scale phân tán. Kết hợp điển hình: TTL để bounded staleness, single-flight để chặn stampede khi key hết hạn, event-based invalidation để cắt TTL sớm khi có write. Ví dụ si

JWTs are one of those technologies that feel wonderful right up until you hit your first "log me out" requirement. Then you discover the awkward truth: the very property that makes JWTs attractive — statelessness — is also what makes logout hard. This post walks through what JWTs actually are, why "invalidating" one is a design problem rather than a one-liner, and the practical methods available to revoke an access token on logout, along with the bottleneck each one introduces. A quick refresher on JWTs A JSON Web Token (JWT) is a signed, self-contained token. It carries a JSON payload of claims — who the user is, when the token was issued, when it expires, and often a unique token id ( jti ) — and a cryptographic signature over that payload. Because the token is signed with a secret (or a private key), any server holding the corresponding key can verify it is authentic and untampered without calling a database . That last part is the entire point. When a request arrives with a JWT, the server checks the signature and the expiry, reads the claims, and proceeds. No lookup, no shared session store, no round trip. This is what people mean when they call JWT auth stateless : the server keeps no per-user session record. The token itself is the session, and it's valid until it expires. Access tokens and refresh tokens In practice you rarely use a single token. The common pattern splits responsibility across two: The access token is the short-lived workhorse. It's sent on every API request and typically expires in minutes (5–15 is common). Because it's checked statelessly on every call, you want its lifetime short — if it leaks, the damage window is small. The refresh token is long-lived (days or weeks) and does one job: obtain new access tokens when the current one expires. It is not sent on every request — only to a dedicated token endpoint. This lets the access token stay short and stateless while the user avoids logging in every ten minutes. The refresh token is easy —

Book: Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You have two copies of the same rules. One lives in a StoreUserRequest on the server. The other lives in a Zod schema, or a Yup object, or a pile of required attributes, on the front end. They started identical. Then someone bumped the password minimum from 8 to 12 on the backend and forgot the client. Now the form says the password is fine, the user clicks submit, and a 422 bounces back with an error the UI never predicted. That drift is the whole reason live client-side validation is annoying to maintain. You are keeping two rulesets in sync by hand, and the sync breaks quietly. Laravel Precognition removes the second copy. The front end asks the server "would this pass?" before the user submits, and the server answers using the exact same validation rules the real request will run. What a precognitive request actually is A precognitive request is a normal HTTP request to your real endpoint, tagged with a Precognition: true header. Laravel sees the header, runs the route's middleware and validation, and then stops before your controller does any real work. It never writes a row. It never sends an email. It runs the rules and returns the verdict. Success comes back as 204 No Content with a Precognition-Success: true header. Failure comes back as a normal 422 with the same JSON error bag your form submit would produce. Same rules, same messages, same field names. There is no second schema to drift. The lifecycle is worth holding in your head: Front end sends the form state to the real URL with Precognition: true . Middleware runs. FormRequest validation runs. Laravel short-circuits: your controller body never executes. Response is