标签：#pagi

Middleware in PAGI A port of the sample app from What Is Middleware? — which builds the same three-layer stack in Plack/PSGI (Perl) and Starlette/ASGI (Python) — to PAGI , an async, ASGI-style application interface for Perl. The app is deliberately tiny but exercises the three things middleware exists to do: Logger — wrap the request, time it, log method/path in and status/duration out. Authenticator — inspect a header, inject context for downstream layers on success, or short-circuit with a 401 on failure. ProfileRouter — answer one specific route from inside the stack, reading the context the Authenticator injected. All code below was run under perl-5.40.0 with PAGI::Test::Client ; the log lines and responses shown in Running it are the actual captured output, not hand-written. The PAGI middleware contract A PAGI application is, in the spec's words, "a single coderef returning a Future": an async sub over the ($scope, $receive, $send) triple — the same shape as ASGI. $scope is the per-connection metadata hash ( type , method , path , headers , …), $receive pulls inbound events, $send pushes outbound ones ( http.response.start , then http.response.body ), and the Future it returns resolving is what tells the server the response is complete. Middleware is just as plain: a subroutine that takes an application and returns a new application, wrapping the inner one. That is the whole spec-level contract — app in, app out: sub middleware { my ( $app ) = @_ ; return async sub ($scope, $receive, $send) { # ... before ... await $app -> ( $scope , $receive , $send ); # call the inner app # ... after ... }; } A middleware propagates the inner app's Future — its completion and any exception flow straight through — and never reads its return value, which the spec defines as inert; to observe or rewrite the response it wraps $send instead, and to add per-request context it clones $scope (top-level edits stay visible downward only). PAGI::Middleware , from PAGI-Tools rather than

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected. No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count. TL;DR A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early. Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records. The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page. Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows) . This is a completeness check, not a correctness check. Different layer, different bug. What actually goes wrong You write the loop everyone writes. Walk ?page=1 , ?page=2 , keep going until a page comes back empty. Stop. Save. Done. The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean. You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows an