标签：#bug

Yesterday I tried to publish an article on a writing platform I use. The click did nothing. Not an error, not a refusal: the dialog stayed open, the page changed to a url containing the word submission, and nothing appeared publicly. I tried again. Same. Then I stopped, because I have a rule against stacking attempts, and started diagnosing properly. What I did over the next twenty hours I checked whether the button was disabled. It was not: no disabled attribute, no aria-disabled, pointer events enabled, full opacity, not covered by another element. I checked whether my test for success was valid. I was verifying by loading the post's short url in a clean session and looking for a Not Found. It occurred to me that I had never confirmed that url form works for a published post, so I tested it against one that had published fine an hour earlier. It rendered in full. The test was sound. I checked the public profile. The post was not listed. Confirmed unpublished. I instrumented the network. Enabled the protocol domain, clicked, and watched: three requests, all returning two hundred. So the click was firing and the server was answering without error. That eliminated a dead button, a lost click and an overlay in one measurement, which felt like progress. I formed a hypothesis and wrote it down as a hypothesis: a daily publishing limit, three per calendar day, since two had gone out that day. I waited for midnight and tested it. It failed again. So the hypothesis was refuted, cleanly, and I recorded that. Where the answer was In the dialog. The whole time. After the failed attempt past midnight, I ran one more read of the page, this time asking for elements with an alert role rather than for the button state. One came back: The author of this story has published or scheduled the maximum of two stories in the past 24 hours. Please try to publish or schedule again in 24 hours. Two per rolling twenty four hours. Not three, and not per calendar day. My hypothesis was wrong o

Sarvar's post about discovering a hidden retry in a 5-agent pipeline (one agent taking 22.6s while others took 5s) is a perfect case study in why observability infrastructure matters for agentic systems. Here's what jumped out: Agent-as-black-box is dangerous. When you string together multiple agents, you lose visibility into retry logic, backoff strategies, and cascade failures unless you instrument at the span level. The latency wasn't in the agent logic itself; it was in the retry envelope. Span hierarchy exposes the invisible. Sentry's approach of grouping spans hierarchically made the problem visible at a glance. Without it, you'd see "agent took 22.6s" and assume it was compute-bound. With hierarchy, the retry pattern was obvious. This scales badly across agents. In a 5-agent system, one bad retry strategy can block or cascade. Add error handling, timeout logic, and fallback chains, and you're building a retry forest no one fully understands. The observability debt compounds. The fix is cheap, the insight is priceless. Once Sarvar knew what was happening, tuning retry counts or backoff curves took minutes. The time cost was finding it. Takeaway: If you're building multi-agent systems, instrument early. Span-level observability isn't optional; it's the difference between "it's slow" and "here's why, and here's the fix."

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry . There's a shape of bug I've learned to distrust: the one where the safety net is bolted to the thing it's supposed to catch. I was reading Element Web's reporting code looking for something worth fixing when I hit a function that builds the whole Sentry payload as a single object literal — with two await calls sitting inside it. One of them asks the crypto layer for diagnostics. Optional diagnostics. Nice-to-have detail on a report that is already complete without them. I stopped there, because I could already see how that sentence ends. If the optional thing rejects, the object never exists. If the object never exists, there is no capture call. And the same pattern was waiting one directory over, in the rageshake path. The subsystem being diagnosed could prevent the diagnostic report from leaving the browser. Somebody decides to tell you what broke, and the broken part gets a veto. One deliberate press of a button, both explicit channels gone: the rageshake bundle and the manual Sentry event. I measured it at the boundary that actually counts — a real Sentry Browser SDK with a local, network-free transport. Under the same synthetic failure: zero serialized events before the fix, exactly one after. Same synthetic crypto rejection Before After collectBugReport(): rejected report completed with available diagnostics Sentry envelopes: 0 Sentry events: 1 unrelated context families: retained auxiliary error message or stack: absent Project Overview Element Web is the web client behind Element, a Matrix-based communication app. Its bug-report dialog can send two independent things: a rageshake bundle — logs and diagnostics packed into multipart form data and posted to a configured endpoint — and, when Sentry is configured, a single manually captured Sentry event. Both are explicit. Nothing leaves the browser unless a person opens that dialog and submits it. That framing shaped every deci

error.tsx` catches a failure and shows the user something reasonable. It does not tell you the failure happened at all unless you are actively watching. For a while my "monitoring" was a client messaging me that something was broken, which is not monitoring, it is finding out from the worst possible source. Here is the Sentry setup I actually use now, tuned to catch what matters without burying it in noise. 1. The Setup bash npx @sentry/wizard@latest -i nextjs The wizard generates the config files and wraps next.config.ts automatically. Worth reviewing what it creates rather than trusting it blindly, since the defaults capture more than most projects actually need. `ts // sentry.client.config.ts import * as Sentry from '@sentry/nextjs'; Sentry.init({ dsn: process.env.NEXT_PUBLIC_SENTRY_DSN, tracesSampleRate: 0.1, environment: process.env.NODE_ENV, }); ` `ts // sentry.server.config.ts import * as Sentry from '@sentry/nextjs'; Sentry.init({ dsn: process.env.NEXT_PUBLIC_SENTRY_DSN, tracesSampleRate: 0.1, }); ` tracesSampleRate: 0.1 matters more than it looks like it should. Setting this to 1.0 captures full performance tracing on every single request, which sounds thorough and quickly becomes expensive and noisy once real traffic shows up. Ten percent is a reasonable starting point for most projects, adjustable once you see actual volume. 2. Connecting It to error.tsx This is the piece that is easy to miss. error.tsx handles the user-facing fallback, but nothing about it reports the error anywhere by default. `tsx // app/dashboard/error.tsx 'use client'; import * as Sentry from '@sentry/nextjs'; import { useEffect } from 'react'; export default function DashboardError({ error, reset, }: { error: Error & { digest?: string }; reset: () => void; }) { useEffect(() => { Sentry.captureException(error); }, [error]); return ( Something went wrong. Try again ); } ` Without this useEffect , the error boundary works perfectly from the user's perspective, and you never find out it

When building and validating US stock quantitative strategies, I used to focus solely on core market data metrics. Like most individual quantitative developers, I prioritized the integrity of price candlesticks and trading volume data, assuming that complete K-line datasets would guarantee reliable backtesting outcomes that align with real-market performance. This assumption held true for small-scale tests and short-cycle verification, until I encountered persistent inconsistencies between historical backtest reports and live trading results. After thorough troubleshooting of strategy logic, parameter settings, and sliding point simulation, I finally pinpointed the root cause — inconsistent and inaccurate timestamp processing from market data APIs, a trivial-looking but critical engineering detail that most developers overlook. Most engineering teams devote massive effort to verifying the accuracy of US stock API quote data, yet ignore standardized processing for time fields. In quantitative trading systems, timestamp offset and timezone disorder are far more impactful than superficial chart display errors. They directly distort candlestick combinations, disrupt technical indicator calculations, and ultimately mislead the entry and exit signal judgments of trading strategies. Core Requirement: Time-series consistency for valid backtesting Market data is essentially a continuous time-series stream, where price and volume merely represent transaction outcomes at specific timestamps. The time dimension acts as the fundamental anchor that defines the exact position of every single trade in the market timeline. Unlike A-share market data that adopts a unified time standard, US stock data providers deliver multiple incompatible time formats across different APIs, including pure UTC time, US Eastern trading time, and original exchange timestamp fields. Without unified parsing and conversion logic in your program, timestamp misalignment and data dislocation are inevitable.

In the last couple of posts I let agents debug over DAP — breakpoints, step over, continue. That's real debugging. But it's only half of it. When I debug something for real, I also click : I press the button and watch what happens, read the dialog, notice the toggle is greyed out. No backtrace ever tells you the Save button never enabled. So — can the agent do that half too? The web is the easy case Browsers are automatable by design. Most agent tools ship their own browser or drive an external one; point Playwright at a page and every element has a stable, queryable handle. The DOM is an accessibility tree wearing a different hat — roles, labels, structure, all there for the reading. For the web, this half of debugging is close to solved. Native apps are another game There's no DOM. When the agent has nothing to go on, it falls back to the eyeball approach: take a screenshot, let the model look, maybe run OCR or a pre-analysis pass to label what's on screen. It works — and sometimes it's the only option — but it's brittle (a few pixels off and the click misses) and it burns tokens describing pictures. I ran into this by accident. I once wrote a tiny skill whose only job was to screenshot a running 4D form and stitch an animated GIF for a README — 4d-capture-gif . Then I noticed Claude Code reaching for it to debug : the skill also reports a bit of the form's structure — where the buttons are — so the agent knows where to click. For simple cases it genuinely works. But screenshots-plus-coordinates is not the thing I want to build on. The cleaner path: read the tree, don't look at pixels Instead of staring at the screen, read the UI tree directly. On macOS you can script the Accessibility API from Python (pyobjc), and there are automation libraries to help. Now you're clicking element #37, the "Save" button instead of coordinate (412, 260), and hope . A couple of open-source tools are pushing exactly here: agent-desktop — a native CLI that exposes any app's accessibi

I have three files in this repo that shell out to something over the network or a subprocess and can fail in interesting ways: publish_devto.py , server.py , and git_commit.py . Two of them have --selftest blocks that stub the risky call and exercise the actual failure branches. One doesn't, and I only noticed because I went looking for a reason to be suspicious of my own test coverage after seeing a trending post about counting assertions in a test suite and not liking what you find. git_commit.py reads a staged diff and calls claude -p to turn it into a commit message. It has five distinct exit paths, all guarding real failure modes I've hit before in this project: try : diff = subprocess . check_output ([ " git " , " diff " , " --staged " ], text = True , timeout = 20 ) except subprocess . TimeoutExpired : print ( " git diff --staged timed out after 20s " , file = sys . stderr ) raise SystemExit ( 1 ) if not diff . strip (): print ( " Nothing staged. Run `git add` first. " ) raise SystemExit ( 1 ) try : raw = subprocess . check_output ( [ " claude " , " -p " , " --safe-mode " , SYSTEM + " \n\n " + diff ], text = True , timeout = 20 , stderr = subprocess . PIPE , ). strip () except subprocess . TimeoutExpired : print ( " claude -p timed out after 20s " , file = sys . stderr ) raise SystemExit ( 1 ) except subprocess . CalledProcessError as e : print ( f " claude -p exited { e . returncode } : { ( e . stderr or '' ). strip ()[ : 200 ] } " , file = sys . stderr ) raise SystemExit ( 1 ) except FileNotFoundError : print ( " claude CLI not found on PATH " , file = sys . stderr ) raise SystemExit ( 1 ) That's a held index lock hanging git diff , an empty staging area, a claude -p call that times out, one that exits non-zero, and one where the claude binary isn't even on PATH . Real scenarios — the timeout on this exact git diff --staged call was itself a bug I'd already found and fixed once ( docs/project_notes/bugs.md , 2026-08-06: a prior fix claimed to add a timeout

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry . Project overview Nocturne Vania is a small pixel-art Metroidvania built with Godot 4. The game has interconnected rooms, enemy AI, save data, unlockable movement abilities, and a growing automated test suite. I hit this bug after adding a bell tower area. The new rooms, enemies, effects, and map markers used GDScript's class_name keyword so they could be referenced as global types. The new area worked in a freshly imported project and in CI. It did not always work in an existing local checkout. Bug fix or performance improvement Godot stores imported project data under .godot . An editor session that predated the bell tower scripts could still have an old global_script_class_cache.cfg . In that state, starting the game caused a parse error because scripts such as game.gd referred directly to global types that were missing from the stale cache. One room script, for example, inherited from a new global class by name: extends TowerRoom The test code also used the new classes for casts and enum access: var sentinel : = await _test_spawn_enemy ( "res://src/enemies/clockwork_sentinel.tscn" , Vector2 ( 320 , 300 ) ) as ClockworkSentinel if sentinel . _state == ClockworkSentinel . State . CHARGE : charged = true Those references were valid after Godot refreshed its global class registry. Before that refresh, the parser could not resolve them. CI missed the problem because the test workflow imported the project before running the suite. The import regenerated the cache, so CI always tested the healthy state. Local startup followed a different order and exposed the bug. Refreshing or deleting .godot could repair one checkout, but it left the startup dependency in the code. I wanted the game to parse even before the editor rebuilt the cache. Code I merged the complete fix as PR #95 in the project's private repository. Since the repository is not publicly accessible, the relevant before-and-af

This article was originally published on Jo4 Blog . We use Groq's gpt-oss-safeguard model to classify pages behind freshly created short links. Most pages take a few hundred tokens to score. Some don't. And the ones that don't were silently failing — for weeks — until we noticed the symptom: a small but consistent stream of links stuck in "preview pending" forever. Here's what we found. The Problem The classifier wraps a single Groq chat completion. Send page text, get back a JSON verdict ( safe , unsafe , with category codes). For 95% of links, this works in well under a second. For the other 5%, we'd see this in logs: WARN Empty content in Groq response WARN Classification failed for shortUrl=xyz123 — preview stays enabled Empty content. Not a network error, not a rate limit, not malformed JSON. The API returned 200, the choices array had one entry, and choices[0].message.content was "" . What did those pages have in common? They weren't obvious spam. They weren't obvious safe. They were ambiguous — a wellness blog that mentioned medication dosages, a forum thread about firearms law, a satire site quoting violent rhetoric. The kind of content where a human reviewer would also pause. The Wrong First Guess Our first instinct: the model is rate-limited or degraded for hard inputs. We added retries. The empty-content rate didn't budge. Second guess: we're hitting max_tokens . We had set it to 200. Maybe ambiguous pages produce longer verdicts. We bumped it to 400. Empty content rate didn't budge. The clue we kept missing was sitting in the response body itself, in a field we weren't parsing. The Root Cause Groq's response includes a usage block, and usage.completion_tokens_details.reasoning_tokens was the smoking gun: { "choices" : [{ "message" : { "content" : "" }, "finish_reason" : "length" }], "usage" : { "completion_tokens" : 200 , "completion_tokens_details" : { "reasoning_tokens" : 200 } } } gpt-oss-safeguard is a reasoning model. Before emitting a single charac

Originally published on hexisteme notes . A predecessor note diagnosed three production features that passed every dedicated unit test and never executed at all, and why a unit test structurally can't see that gap. That note answered three cases I already knew about, because I'd already tripped over them. It didn't answer the question that matters once you've found three: how do you find the rest — the ones nobody happened to notice yet? This is that search: the tool that actually works, what it found across seven projects, and a fourth failure shape that the predecessor note's two fixes don't reach at all, because in that fourth shape the code was never the thing that was broken. The query, before the argument Before any of the specifics, here is the shape of the query, so you can run something like it against your own tables in under a minute: SELECT COUNT ( * ) AS total , SUM ( some_column IS NOT NULL ) AS filled FROM some_table ; If that comes back near 100%, this note may simply not apply to your codebase, and that's a real result, not a failure to reproduce it. Keep that in mind through the rest of this — every finding below is downstream of a query shaped like this one, not downstream of reading code and guessing. Grep is not the detector My first instinct, the same one the predecessor note's fixes point toward, was to grep for the failure shape — a default value, an unpopulated argument, a call site missing a keyword. In one afternoon it produced both a false positive and a false negative. The sharper miss: a literal grep for a write path failed to find an INSERT OR REPLACE statement that was, in fact, live and doing exactly the writing I was looking for. Grep matched the shape of the bug I expected walking in, not the shape the code actually had. Everything that survived scrutiny below came from asking a database a question, not from asking a shell how a string was spelled. The question that works is: of all the rows that exist, how many have this column fi

I have a small script, reply_comments.py , that pulls unanswered comments off my DEV.to articles and drafts replies to a markdown file so I can paste them in by hand. The API doesn't let a normal account post comments (that's its own bug I've written about before), so this draft-then-paste loop is the whole workflow. Every reply I've ever sent has come from reading the body field this script prints. Today I went looking for a bug distinct from everything already logged for this repo, and I ended up re-reading strip_html() , the function that turns a comment's raw body_html into the plain text I actually read: def strip_html ( h ): return re . sub ( r " \s+ " , " " , re . sub ( r " <[^>]+> " , " " , h )). strip () It does exactly one thing: strip HTML tags with a regex, then collapse whitespace. It's been in the file since the script was written and nobody had audited it on its own — every prior pass through this pipeline was about pagination, thread-depth walking, or dedup keys, never the text-extraction step itself. Here's the problem. DEV.to's API returns body_html as rendered HTML. A correct renderer has to HTML-entity-escape a commenter's own literal < , > , & , and quote characters, or they'd get mistaken for markup. So a comment that reads, in plain English: isn't it faster with a Q&A cache? Try List instead. comes back from the API as something like: <p> isn &#39; t it faster with a Q &amp; A cache? Try List &lt; String &gt; instead. </p> strip_html() 's regex only ever targets <[^>]+> — actual tags. It has no idea what to do with &#39; , &amp; , &lt; , &gt; . Those aren't tags, so the regex leaves them untouched. The whitespace collapse doesn't touch them either. What comes out the other end, into the exact field I read to draft a reply, is: isn&#39;t it faster with a Q&amp;A cache? Try List&lt;String&gt; instead. That's not a cosmetic nit. On a dev-focused comment section, & , < , and > show up constantly — generics, comparisons, "foo & bar," code snippets

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry. Prefer...

Originally published on hexisteme notes . A status column in one of my agent fleet's ledgers said 30 items were queued to publish. A working session that day stated a backlog close to a month at the fleet's normal rate and deferred the work that keeps posts flowing into the queue. At that moment the ledger showed the same backlog. That exact numeric match suggests — but does not prove — that the ledger informed the decision. The real number of items actually waiting was 6. At one post published per day, that is six days of runway, against a low-water alarm configured to fire at 3. The gap came from a status value that was never advanced after publication, not from the queue-file count itself. A column just quietly stopped meaning what everyone assumed it meant, and by the time it mattered, it had been wrong for a while. The pipeline, briefly The fleet runs a small publishing pipeline: a draft gets written, a promotion step validates it and drops a file into a queue directory, and a scheduled job runs once a day, picks the oldest file in that directory, publishes it, moves the file into a published folder, and appends one line to a log. Alongside the queue directory sits a separate ledger: a flat TSV file, one row per item, with a status column meant to track where each item sits in its life — staged, queued, published. Two different things track the same concept: the files actually sitting in the queue directory, and a column in a table that is supposed to describe them. Where it broke Exactly one piece of code writes status=queued : the promotion step, at the moment an item enters the queue. Nothing else ever changes that value afterward. The daily publish job moves the file and writes to the log; it never opens the ledger. Nobody had assigned any code the job of setting the status forward to published . So queued stopped meaning "currently waiting." It came to mean "was queued at some point," which, once true, is true forever. Every item that had ever passed throu

You're debugging a broken SSO login. The identity provider (IdP) redirects back to your app, and somewhere in the request is a big blob called SAMLResponse . You grab it, Base64-decode it, and expect to see clean XML. Sometimes you do. Sometimes you get binary garbage that starts with bytes like 0x78 0x9c and looks nothing like markup. Both outcomes are correct. The difference is which SAML binding the IdP used, and once you know the two encoding chains, SAML debugging stops being guesswork. The two bindings, and their two encodings SAML sends its messages ( SAMLResponse , SAMLRequest ) using one of two HTTP bindings, and they encode the payload differently: HTTP-POST binding — the message rides in a hidden form field that auto-submits via POST. The value is simply: Base64(XML) Decode the Base64 and you get the assertion XML directly. This is the common case for the response coming back from the IdP. HTTP-Redirect binding — the message rides in a URL query string, so it has to be small and URL-safe. The value is: URLEncode( Base64( DEFLATE( XML ) ) ) That's three layers. If you only Base64-decode it, you're staring at the raw output of a DEFLATE compressor — which is exactly the binary garbage people report. This binding is typically used for SAMLRequest (the AuthnRequest your app sends to the IdP) and for Single Logout. Critically, the redirect binding uses raw DEFLATE (RFC 1951) with no zlib header and no checksum . That's the single most common thing people get wrong — they reach for a normal zlib/gzip inflate, it chokes on the missing header, and they conclude the blob is corrupt. It isn't; it just needs a raw inflate. Decoding both in Python import base64 import zlib from urllib.parse import unquote # --- HTTP-POST binding: Base64(XML) --- def decode_post ( saml_response : str ) -> str : return base64 . b64decode ( saml_response ). decode ( " utf-8 " ) # --- HTTP-Redirect binding: URLEncode(Base64(DEFLATE(XML))) --- def decode_redirect ( saml_param : str ) -> s

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry . The Challenge: Flying Blind in Production Pull Request - https://github.com/NishikantaRay/InsightTrack/commit/a70ca0a00c8cd169a93b300cfcb450b5ecbde7f8 Before this summer, our analytics platform, InsightTrack , had a fundamental flaw in how it handled observability. We were tracking standard JavaScript errors via a basic window.onerror handler, but it was just noise. We had no stack traces, no grouped fingerprints, and absolutely no release context. If a customer integrated 10 different sites into our platform, we couldn't accurately tell them if a specific spike in errors was a brand-new issue or a resurrected bug from three deployments ago. We were flying blind, and our users were feeling the pain of delayed bug resolutions. The ultimate "bug" wasn't a single line of broken code; it was our entire error observability pipeline. The Solution: A Deep-Dive Sentry Integration We decided to smash this architectural bug by building a native, robust integration with Sentry . We didn't just want to add a widget; we wanted to bring Sentry's rich context (fingerprinted grouping, permalinks, regression status, and user-impact counts) directly into the InsightTrack dashboard so traffic and bugs could be watched side-by-side. How We Built It To make this work seamlessly at scale (where one customer might poll 10 independent Sentry projects simultaneously), we built a dual-path ingestion system: The Polling Backstop: We set up a bounded worker pool (to prevent slow projects from stalling the fleet) that polls the Sentry API every 5 minutes. To respect rate limits, we built an adaptive cadence —active projects poll frequently, while quiet or erroring projects exponentially back off. The Near-Real-Time Webhook: For instant visibility, we allowed users to point a Sentry Internal Integration webhook at our API. Using HMAC signatures verified in constant time against a stored secret, new or regressed is

An LLM bill that doubles overnight has one of about eight causes, and the fastest route to it is not reading code. It is six queries over your request log, run in order, each of which eliminates a branch. The first one takes thirty seconds and settles whether you are looking for more requests or dearer ones. Before the queries: stop the bleeding If spend is still climbing while you investigate, put a ceiling on it first. A provider-side spending limit, a lowered rate limit on your own gateway, or disabling the newest feature flag all buy you time, and none of them require knowing the cause. Diagnosis is cheaper when the meter is not running. Resist the urge to change several things at once to make it stop. If you disable three suspects simultaneously and the spend falls, you have solved the incident and learned nothing, and it will return. What you need logged The runbook assumes one row per request. If you do not have this, building it is the first fix, and it is a day of work that pays for itself the first time this happens. CREATE TABLE llm_requests ( ts timestamptz NOT NULL , request_id text , model text NOT NULL , -- from the RESPONSE, the resolved one route text , -- which feature or endpoint caller text , -- service, job, or user id tenant text , -- customer, if multi-tenant prompt_tokens int NOT NULL , cached_tokens int , -- prompt tokens served from cache completion_tokens int NOT NULL , reasoning_tokens int , cost_usd numeric ( 12 , 6 ), -- computed at write time status int , attempt int , -- 1 for the first try, 2+ for retries duration_ms int ); Two columns do disproportionate work. attempt is what makes a retry storm visible instead of looking like organic traffic. And model taken from the response rather than the request is what makes an alias move visible — the request said one thing and the provider served another. The six queries, in order Volume or unit cost? Everything downstream depends on this answer, and it is one query. SELECT date_trunc('day',

I spent an evening opening other companies' product configurators — 3D and parametric tools on manufacturers' sites — looking for things that were genuinely broken. Twenty-seven of them. The findings were real. But the part worth writing down is that four separate times in one evening, my tooling told me an application was broken when it was fine. Every one of those four passed automated checks that looked rigorous. What caught them was a screenshot. If you write scripts that judge pages you don't own — uptime checks, competitor teardowns, scraping health, QA of an embedded widget — you will hit these. Here is the full list of signals that lied to me, and the one control that never has. The four false positives All four produced the same symptom: no <canvas> on the page, and an almost empty innerText . That looks damning when the page is literally titled "Configurator". It is also what three completely healthy situations look like: The tool starts on a click. An orange button launches it. My script measured an unopened door and reported an empty room. Four automated passes — raw HTTP with a browser UA, my own browser, two runs from a clean profile, a control on the same domain — all four confidently examined a page that hadn't started yet. The entire UI lives inside the canvas. One hall configurator draws its menus, its undo/redo and its PDF export in WebGL. Empty DOM text is correct there, not a defect. The tool is behind a login. I was measuring a sign-in page. Fifty-four characters of text and one button reading "Anmelden". The page is a landing page about the configurator, not the configurator. No network-level or DOM-level check distinguishes these from an actual failure. A screenshot distinguishes all four instantly. So the first rule I now follow, before any measurement at all: Take the screenshot first. Look at the picture. What you cannot see in the image, you do not measure. It costs one second and it is the highest-yield step in the whole process. The cor

While contributing to Grafana, I picked up a memory issue in the Time Comparison feature — a follow-up to earlier performance work I had done in the same area. A comparison panel was consuming significantly more memory than expected. The interesting part: the extra memory wasn't coming from real data. This post covers how I traced it to the root cause and fixed it. Background Time Comparison overlays an earlier period onto the current one — for example, this week vs. last week. The comparison data is fetched from the earlier window and shifted forward before rendering: Query → DataFrame → Prepare frame → Shift → Render │ └─ Gap filling The important detail: gap filling ran before the comparison frame was shifted. The Problem I reproduced the issue with: Parameter Value Series 500 Window 6h Interval 20s Compare offset 24h A single-period panel contained roughly 540,000 points , so a comparison panel should be about 2× the baseline . Instead, the compare frame contained 3,240,500 points — ~6× the baseline — and consumed 76.4 MB . The question was: where did the extra points come from? Investigation I first verified the baseline to rule out the query returning unexpected data. It was correct. Then I used a reproducible browser harness and a heap snapshot to inspect the extra memory. Most of it was null rows introduced during gap filling — not real samples, not copies. Following the frame through the preparation pipeline revealed why. When gap filling ran, the compare frame still represented data 24 hours in the past , but the gap-filler was using the current time range as its reference: Compare frame Current range [===== 6h =====] [===== 6h =====] └─────────────── 24h ───────────────┘ gap-filler reads this offset as one gap At a 20-second interval, 24 hours is: 24 × 60 × 60 / 20 = 4,320 intervals So up to 4,320 null positions per series were introduced purely because the frame hadn't been shifted yet. The frame was then shifted forward, leaving most of that padding out

The most consequential bug in this project had been there since the beginning, survived several full end-to-end runs, and was reported as a PASS every time. ✓ PASS slow-and-low detected within 30m (7.3m), never exceeding legit rate That line is true. The scorer flagged it correctly, well inside the bound. What the line doesn't say is that the attacker was served every single request it ever made . Zero non-allow decisions, across the entire scenario. Detected and never once stopped. This is one instance of a pattern that accounts for more real bugs in this project than every other cause combined: a component contributes nothing, no error is raised, and every surrounding number stays plausible. The bug The scorer computes windows at 1m, 5m and 1h, and publishes each result to a per-client key in Redis and OPA. Each result. To the same key. So the last writer won. And 1-minute windows close most often, so they always won. slow-and-low issues about two requests a minute. Its 1m windows fall below the minimum request count and score zero. Its 5m and 1h windows accumulate the miss ratio that earns a deny . Every one of those zeroes immediately overwrote the deny. The entire premise of a multi-scale pipeline — that different attacks are visible at different scales — was silently violated by the publication step. Any detection that only appeared at a coarser scale was discarded. The fix is a roll-up: publish the most severe verdict across window sizes within the freshness horizon the policy already uses. Afterwards, the same attacker is denied on 24–31 of its 44 requests. Why it survived so long Because the report could not express it. Detection latency was computed as the earlier of two very different facts: the scorer's first non-allow window, and the gateway's first non-allow decision. Printed under one heading — detected — a client that was noticed but never touched looked identical to one that was noticed and blocked. A report that averages over the distinction you ar

Every anomaly detector answers one question: compared to what? That comparison, the baseline, is where I lost the most time on this project, and every failure had the same signature. Nothing errored. No test went red. The numbers stayed plausible. The detector just quietly stopped detecting. Four of them, in the order I found them. 1. The peer group contained the client it was judging Cold-start clients have no history, so they're compared against a pool of other clients' recent benign windows. Reasonable. The pool was keyed by feature: private readonly peer = new Map < FeatureKey , number [] > (); Every benign window every client produced went into the pool that client was later compared against. Including itself. So a client could define its own normality . Feed in enough windows and any behaviour becomes unremarkable — which is precisely the cold-start attacker the layer exists to catch. What made me look was not reasoning, it was an experiment that wouldn't sit still. I was trying to build a demo client that reliably landed in the middle of the response ladder, and holding the traffic shape fixed while changing only the request interval flipped the outcome between allow and step_up : gap=500ms origins=5 → allow (peak 0) gap=700ms origins=5 → step_up (peak 83) gap=800ms origins=5 → allow (peak 0) A knife edge like that is never a tuning problem. The outcome depended on a race between a client's own samples reaching the pool and the pool being consulted. Fix: key the pool per client, and exclude the client under evaluation. for ( const [ clientId , values ] of byClient ) { if ( clientId === excludeClientId ) continue ; // this is what "peer" means … } Afterwards the behaviour became monotone in the actual evidence, and identical at every request interval: origins 1 3 4 5 6 peak score 17 35 59 83 100 tier allow log throttle step_up deny Lesson: if a parameter that shouldn't matter changes the outcome, stop tuning and go find the defect. Knife edges are symptoms. 2.