标签：#MachineLearning

Using a model to grade another model’s output is the only approach that scales to open-ended text. It is also the point at which your measurement device becomes a second stochastic system with opinions, and the difference between a useful judge and a number-generator is entirely in whether you validated it. A judge is an instrument, not an oracle Think of the judge the way a lab thinks about a thermometer. It has a reading, a bias, a precision, and a range over which it is trustworthy — and none of those are known until you check it against a reference. The reference is human labels. There is no way around this: a judge whose agreement with humans on your task is unknown produces numbers whose meaning is unknown, however many decimal places the harness prints. The good news is that the calibration is a one-off cost of a few hundred human labels, after which the judge runs for essentially free on every subsequent evaluation. That trade is what makes judges worth the trouble. What the published agreement figures say The standard reference is Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. On their setup, a strong judge model agreed with human expert preferences at a rate above 80% — which the authors note is comparable to the agreement rate between two human experts on the same comparisons. That framing is the important part: the ceiling for a judge is not perfect agreement, it is human-human agreement, because the humans disagree with each other on genuinely ambiguous items. The same paper documents the failure modes that come with it — position bias, verbosity bias, self-enhancement bias, and weakness on maths and reasoning items where the judge must itself solve the problem to grade it. So the honest summary of the literature is: a well-constructed judge on general chat quality can approach human-level agreement, and it does so while carrying systematic biases that you have to design around. It is not evidence that your judge, on your

You ask for every line item on the invoice. There are ten. You get seven, the JSON validates, and nothing anywhere reports a problem. This is the single most reported structured-output bug and at least half the time the model is not the cause. Four causes, wildly different fixes Cause Description Truncation finish_reason == 'length'. The list was cut off mid-flight. Your max_tokens, not the model's recall. Chunk boundary Items 8-10 were on a page you did not send. Check what text actually reached the model. Dropped constraint You set minItems and the provider ignored it, so nothing enforced anything. Genuine omission Everything was present and the model stopped early. The only one that is actually about the model. Diagnose in that order, because the first three are cheap to rule out and the fourth is the expensive one to work on. Log finish_reason , usage.completion_tokens and the length of the text you sent on every extraction call and the first two answer themselves. The constraint you thought you set minItems and maxItems sit outside the documented supported keyword set for hosted strict modes. Depending on the stack, sending them either gets you a 400 naming the keyword — fine, you learn immediately — or a 200 where the keyword was quietly discarded. The second case is the trap, because your schema is now a comment. You believe a floor is enforced, the API returned success, and the array is short. Nothing in any log says the constraint was never applied. Find out which of the two your endpoint does before you rely on it — and either way, put “every item, do not summarise or skip” in the array’s description , since that reaches the model whether or not the keyword survives. Why counting is hard for a decoder There is no counter. Each token is produced from the context, and the context contains the items already emitted — so “have I got them all” is not a lookup, it is a judgement the model re-makes at every array element from what it can see. Two structural conse

An inference key is a payment instrument with an API. That is the property that makes it different from most credentials you manage: the person who steals it does not need your data to profit, because the key itself buys something they want. What makes an inference key different It is directly monetisable. A stolen database credential needs a buyer for the data. A stolen inference key is resold as capacity within hours, and automated scanners harvest public repositories continuously looking for exactly this. The loss accrues while you sleep. Usage-based billing means the damage is a function of elapsed time and rate limit, not of a single event. This is the argument for hard caps over careful monitoring. It is passed around more than most secrets. Notebooks, evaluation scripts, a colleague’s laptop, a CI job, an agent’s own environment. Every one of those is a copy you do not control. The blast radius is often the whole account. Where a provider offers one key with full access, a leak is total. Where it offers scoped keys, use them — this is the single biggest lever available. Leak paths specific to AI applications Generic advice — do not commit secrets, use a manager — is correct and widely published. These are the paths that only exist because there is a model in the system, and they are the ones that survive a conventional review: The key in the context. A key pasted into a system prompt so a tool “has access to it”. It is now one paraphrase from the transcript, and the transcript is stored. Traces and observability. LLM tracing tools capture full request bodies by default. If a header, a tool argument or an environment dump ends up in a span, your key is in a third-party dashboard with a broader access list than your secret manager. Prompt and response logs. Same problem, your own infrastructure. A logger that prints the request object on error will print the Authorization header. Evaluation datasets. Captured production traffic reused as an eval set, then share

Most LLM alerting starts as a threshold on latency and a threshold on error rate, fires nine times in the first week, and is muted by the second. The fix is not better thresholds. It is a different trigger model and a much shorter list of things allowed to page. Level-triggered, not edge-triggered An edge-triggered alert fires on a transition: latency crossed 3 seconds, error rate spiked. It is easy to write and it is why your phone buzzed at 03:00 about a condition that resolved itself in forty seconds. A level-triggered alert asks a different question — is the system currently in a bad state, and has it been for long enough to matter? Concretely, the difference is that the alert condition is evaluated over a window and describes a sustained state, and it clears when the state clears rather than when someone acknowledges it. Every rule below is of that shape. Anything that fires on a single scrape does not belong in a paging policy; put it in a dashboard. Page on symptoms, ticket on causes The reliable partition, straight out of ordinary SRE practice and entirely applicable here: Page when users are being harmed now, and a human can do something about it in minutes. That is a small list: the feature is failing, the feature is unusably slow, or money is leaving the building at an unplanned rate. Ticket when something is degraded, trending wrong, or will bite in days. Rising retry rate. One provider slower than usual while failover is absorbing it. Attribution coverage slipping. Neither for everything else. If nobody would act on it, it is a chart. The distinction matters more for LLM features than for a normal service because so many of the interesting signals are causes : a provider 429 rate, a fallback rate, a cache-hit drop. If failover is working, none of those are user-visible and none of them should wake anyone. They are exactly what you want in the morning ticket queue. Burn-rate alerts, with the numbers The standard design — described in Google’s Site Reliab

Most discussion of AI in peer review argues about whether the reviews are any good. That is the second question. The first one is that a manuscript under review is somebody else’s confidential unpublished work, and pasting it into a service is a disclosure you were not entitled to make. The argument that comes first When you accept a review invitation you accept a confidentiality undertaking. The manuscript is unpublished, it usually contains results the authors have not yet established priority on, and in the case of grant review it contains an unfunded research plan — arguably the most commercially and academically sensitive document in the whole system. You agreed not to share it. Sending it to a third-party service is sharing it. That is true whether or not the provider trains on it, whether or not it is retained, and whether or not anyone ever reads it. The undertaking was not “do not let this be trained on”; it was “do not disclose this”, and transmission to a party the authors never agreed to is disclosure. Retention and training policies affect how bad the breach is, not whether one occurred. Notice what this argument does not depend on. Not model quality, not hallucination, not bias. It would apply identically to a perfect system, which is why it is the argument that has actually driven policy, and why it will not be resolved by better models. It can only be resolved by changing where the computation happens — a model running on infrastructure already covered by the confidentiality arrangement raises a different question from a consumer chat interface, and any serious policy will distinguish them. The second argument: accountability A review is a named expert’s judgement. Its value to an editor is not the prose; it is that a person who knows the field read the paper and formed a view they are willing to stand behind. Generated text can simulate the prose and cannot supply the judgement. Editors describe the resulting artefact recognisably: fluent, correctly

Two quite different things are discussed under one heading, and almost all the confusion comes from that. One is a researcher using a model to draft, edit or translate work they did. The other is fabricated content submitted to inflate a publication record. The first is a disclosure question. The second is fraud, and it is not new. Two problems wearing one name A non-native English speaker using a model to make their methods section readable has done nothing wrong and has improved the literature. A paper mill generating plausible manuscripts at volume has committed fraud, and would have done so with or without a language model — mills existed, using image manipulation, template text and fabricated data, long before this technology arrived. Keeping them apart matters because they call for opposite responses. The first needs a disclosure norm and nothing else. The second needs content verification, and content verification does not care what tool produced the content. Any policy built around detecting machine text will punish the first group and miss most of the second, because fabricated research that has been lightly rewritten is indistinguishable from careful assisted writing. It is also worth being clear about where the demand comes from, because it explains why no technical measure will resolve this. Paper mills exist because publication counts are used as a proxy for research contribution in hiring, promotion and institutional ranking, in systems large enough that buying an authorship is a rational purchase for some buyers. Generative tools lowered the cost of supplying that demand; they did not create it. A detector, even a perfect one, sits downstream of an incentive that would simply route around it — which is why the interventions with the best track record are the ones that attack verifiability, such as requiring data and code, rather than the ones that attack production. What the artefacts look like Leftover interface text. Phrases that belong to a chat in

Whether your clinical software is a regulated medical device is decided by what you claim it does, not by how it is built. The same model can be an unregulated administrative tool with one intended-use statement and a class III device with another, and the statement is yours to write. Information, not legal advice, and not clinical or regulatory advice. Reviewed 4 August 2026. Device classification is fact-specific and the consequences of getting it wrong include enforcement action and product withdrawal. Use a regulatory professional. This page describes the questions that decide the answer; it does not answer them for your product. The line, in one paragraph per jurisdiction European Union. Software is a medical device if the manufacturer intends it for a medical purpose — diagnosis, prevention, monitoring, prediction, prognosis, treatment or alleviation of disease — as set out in the definition in Regulation (EU) 2017/745, the Medical Device Regulation, or the corresponding definition in the In Vitro Diagnostic Regulation (EU) 2017/746 where it works on specimens. There is no clinical decision support carve-out. Software that drives or influences the use of a device, or provides information used to take decisions for diagnostic or therapeutic purposes, is in. United States. Software is a device under the Federal Food, Drug, and Cosmetic Act if it is intended for use in the diagnosis, cure, mitigation, treatment or prevention of disease — but section 520(o), added by the 21st Century Cures Act in 2016, excludes certain clinical decision support software from the device definition entirely, on four cumulative conditions. That carve-out has no EU equivalent and it is the single biggest structural difference between the two regimes. EU: qualification then classification Two questions in order. Qualification asks whether it is a device at all. Classification asks which class, which determines the conformity assessment route and whether a notified body is involved. Qua

One rule settles nearly every case: a model may help you say what is true about you, and it may not decide what is true about you. Drafting is help. Supplying the content of a claim about your own experience is not. The line, and why it is there An application is a set of representations about a person, made by that person, on which somebody else will rely. That is what makes fabrication in one different in kind from fabrication in an essay: there is a party who acts on it, and there are consequences downstream for colleagues, clients and sometimes patients. So the test is not “did a machine touch this”. It is “does the document assert something the applicant does not know to be true”. A cover letter drafted from your notes and edited by you asserts nothing you did not supply. A cover letter that describes a project you did not run asserts something false regardless of who typed it, and would be equally dishonest written by a friend. Case Description rewriting: fine Rewriting your own bullet points more clearly. Fixing grammar. Translating your industry's jargon into the target industry's. Cutting 900 words to 300. Generating ten possible openings so you can choose one. decoding: fine Asking what a job advert is actually asking for, then checking your own experience against that list yourself. metrics: not fine Letting it fill in achievements, metrics or responsibilities you have not verified. 'Increased conversion by 32%' is a fact about the world; if you do not know the number, it is a fabrication with a number in it. motivation: not fine Any statement of motivation you have not read and would not say out loud. 'I have long admired your work in X' when you have not is a small lie that is very cheap to expose in an interview. assessments: not fine Completing an assessment designed to measure your unaided ability, where the employer has said not to, or where the whole point of the task is the thing you outsourced. What genuinely helps The honest uses are also the ef

Automatic top-up — the feature that charges a saved card when a customer’s balance falls below their threshold — could never have succeeded for anybody. The invoice was constructed in the wrong currency, and every attempt would have failed in a way that told the customer their card was bad. This is the whole write-up, in the shape we would want any incident written in. Summary An invoice does not take its currency from the line items attached to it. It takes it from the customer’s default currency, or failing that from the Stripe account’s — which is EUR for a Dutch business. Every price in this product is denominated in USD. Finalising the invoice therefore failed with a currency-conflict error, on every automatic top-up, unconditionally. The manual top-up path was never affected, because a Checkout Session takes its currency from the first line item rather than from the customer record. That difference is why the bug could exist in a product whose payment flow demonstrably worked. Impact Dimension Description Customers affected None. The defect was found before the path carried real traffic. This is stated plainly rather than omitted, because a postmortem that lets a near miss read as an outage is as dishonest as one that hides an outage. What would have happened Every automatic top-up fails. The failure surfaces as a payment error, which the failure counter records as a strike, and after three strikes the customer's automatic top-up is switched off entirely. What the customer would have concluded That their card was declined. The message they receive says the saved card could not be charged. They would have gone and fixed a card that was working perfectly. Secondary effect A customer relying on automatic top-up to keep a production integration serving would have run out of credit silently, at whatever hour their traffic happened to cross the threshold. The second and third rows are what make this worth writing up. A defect that fails loudly and correctly is a bug

“AI discovered a new material.” “AI found a drug candidate.” “AI solved protein folding.” Each of those sentences can be true, badly misleading, or flatly wrong depending on one thing the sentence does not tell you: how far the result got from the model before somebody wrote it down. The sentence that hides four different claims Take a single headline: a model proposed a molecule that binds a protein implicated in a disease. That sentence is compatible with at least four very different states of the world. The molecule might exist only as a string in a file. It might have been synthesised. It might have bound the protein in a test tube. Or it might have improved an outcome in a person. Those four are separated by years, by orders of magnitude in cost, and by a probability of success that drops at every step — and press coverage routinely reports the first as though it were the fourth. This is not a complaint about journalism. It is the single most useful thing to internalise about the whole field, because once you have the ladder in your head you can grade a claim in about ten seconds, and you can do it for a subject you know nothing about. The ladder The rungs are the same in every discipline. Only the names of the instruments change. Rung Description 1 · Output The model emitted something: a structure, a score, a candidate, a forecast. Nothing has been checked. Everything downstream is conditional on this being worth checking. 2 · Retrospective The output was compared against data that already existed — held-out structures, historical weather, known compounds. This is where nearly all published numbers live, and it is entirely dependent on the held-out set resembling the future. 3 · Prospective The prediction was made first and the answer arrived afterwards. A forecast verified against what the weather then did. A candidate synthesised after being proposed. This rung is qualitatively stronger than rung 2 and much rarer. 4 · Confirmed An independent method establis

Every team already knows how to put a feature behind a flag. What is different here is that the thing most likely to need changing at three in the morning is not whether the feature is on — it is which model it calls, which prompt it uses, and how much it is allowed to do without asking. Why the usual flag is not enough A conventional feature flag answers one question with a boolean, and it is the right shape because a conventional feature has one failure mode: it is broken. An AI feature has several, and they want different responses. The provider is degraded — you want a different model, not the feature off. A prompt change regressed quality — you want the previous prompt, which is not a code deploy. The feature is fine but a specific customer’s data is producing bad output — you want it off for them and on for everyone else. Spend is running above forecast — you want the cheap model or the degraded path, not an outage. A single boolean answers none of these, so the response to each becomes a deploy, and a deploy is the slowest tool available at the moment you most need speed. There is a second reason, specific to this dependency. The behaviour you are flagging can change without you deploying anything, because the model is somebody else’s and it can be updated underneath you. Flags are usually a mechanism for controlling your own changes; here they are also the mechanism for reacting to changes you did not make, which is why detecting a provider-side behaviour change and having a flag to respond with are two halves of one control. Four things to flag separately Axis Description Feature on/off The ordinary flag. Per-tenant and per-segment, because the common case is a problem confined to one customer's data rather than a global outage. Model selection Which model each call site uses, as configuration. This is what lets you switch providers during an incident, run a canary on a new model, or drop to a cheaper one under budget pressure — without shipping code. Promp

“Something went wrong. Please try again.” is correct for about a third of AI failures and actively harmful for the rest, because for the rest, trying again cannot possibly help and you have just told the user to spend money finding that out. Everything that can go wrong Errors arrive from at least four layers, and the user-facing consequences differ enough that collapsing them into one message destroys the only information you had. Failure Description Transport Connection dropped, DNS, TLS, the stream died mid-token. Retryable, usually transient, and the user did nothing wrong. This is the only class where 'try again' is straightforwardly true. Rate limited (429) Yours or the provider's capacity, not the request. Retryable but only after a wait, and the wait is often stated in a header. Telling the user to retry immediately guarantees a second 429. Provider 5xx / overloaded Retryable with backoff, and the single best case for automatic failover to another provider rather than for any message at all. Timeout Ambiguous by construction: the request may have completed on the provider's side and been billed. Retrying may duplicate a side effect, which is why idempotency matters more here than anywhere. Context length exceeded Deterministic. Retrying the identical request fails identically. The only fix is fewer tokens, and the interface knows that — so the message should offer the fix, not the retry. Content filter The provider blocked the input or the output. Not retryable unchanged. Distinct from a model refusal, and users experience the two very differently. Truncated output The generation hit max_tokens. Not an error at the transport layer at all — status 200, a finish reason of 'length', and an answer that stops mid-sentence. Silently the most common broken experience. Malformed structured output Valid HTTP, invalid JSON or a schema violation. Retryable and often succeeds on a second sample, because it is a sampling accident rather than a capability failure. Empty o

Estimates of the energy in one model query differ by orders of magnitude across credible sources. Most of that spread is not disagreement about physics. It is disagreement about where the system boundary is drawn, and a figure quoted without its boundary is not a figure. Why the published figures disagree Before comparing two numbers, establish which of these each one includes. Any of them can change the answer by more than the model choice does: Which model, and how much output. A short answer from a small model and a long answer from a large reasoning model differ by several orders of magnitude on their own. A single “per query” figure averages over a distribution nobody specifies. Batch size and utilisation. The dominant engineering term. Serving many requests concurrently amortises the weight read across all of them; the same hardware at low occupancy spends nearly the same power for a fraction of the tokens. Facility overhead. Cooling, power conversion and distribution, captured as power usage effectiveness. It multiplies everything, and whether a figure includes it is frequently unstated. Training amortisation. Some analyses divide training energy across expected lifetime queries. Defensible, and it produces a different quantity from marginal serving energy. They are not comparable. Embodied energy. Manufacturing the accelerators, the building and the power infrastructure. Usually excluded, occasionally included, rarely flagged. Idle and provisioned capacity. Capacity is held for peak. Charging queries only for the seconds they compute understates the system; charging them for provisioned capacity overstates the marginal query. Both are used. Building the estimate yourself The marginal serving calculation is not complicated, and doing it once makes every published figure legible. Serving side, per accelerator: E_per_token = (P_device · n_devices · PUE) / R_tokens_per_second P_device average power draw under load, from the spec sheet (below the rated maximum in

Economists modelling AI reach conclusions ranging from a modest productivity bump to a change in the growth regime. They are not using different data. They are using different values for three or four parameters, and the parameters are where the argument should be. Two families of model Task-based automation models Associated most closely with Daron Acemoglu and Pascual Restrepo, these treat production as a continuum of tasks, each performed by labour or by capital. Automation moves tasks from labour to capital, which raises productivity and displaces workers; new task creation moves the boundary back. Growth and distributional effects both fall out of the movement of that boundary. The framework’s virtue is that it makes the aggregate effect an explicit function of quantities you can in principle measure: what share of tasks is exposed, how much cost is saved on each, and how fast new tasks appear. Acemoglu’s own applications of it to AI produce deliberately conservative aggregate numbers, and the reasoning is transparent — the effect is bounded by the exposed share times the saving on that share, so a large aggregate effect requires both terms to be large. Idea-production models The semi-endogenous growth tradition, associated with Charles Jones, models growth as driven by ideas, with ideas produced by researchers. Its central empirical observation is that ideas are getting harder to find: research effort has risen dramatically while growth has not, so productivity per researcher is falling. Aghion, Jones and Jones applied this framework to AI directly, and the key move is that AI enters not as a better tool but as a substitute for researchers themselves. That changes the mathematics qualitatively rather than quantitatively. If the population of effective researchers can be expanded by producing more compute rather than by waiting for demographic growth, the constraint that keeps growth steady in these models is loosened, and under some parameter values the models

This page is about method, not about any particular medicine, and nothing here is medical advice. It is written to answer one question: when a company says a drug was discovered with AI, which part of a decade-long process is that sentence about? The pipeline, and where the years go Roughly, and with enormous variation: pick a target, find molecules that do something to it, optimise those molecules into something drug-like, test in animals and in safety assays, then run the clinical stages — first for safety in a small number of people, then for efficacy in patients, then in a large confirmatory trial — and then apply to a regulator. Start to finish is usually over a decade. Two facts about that pipeline determine everything else on this page. The first is that the calendar and the money are dominated by the clinical stages, not the discovery ones. The second is that failure is the normal outcome, and it is concentrated where the drug first meets human biology: a candidate can be a beautiful molecule, hit its target exactly as designed, and still not help anyone, because the target was the wrong thing to hit. Where models are genuinely used Application Description Virtual screening Score enormous make-on-demand chemical libraries against a target site far faster than physics-based docking can. The output is a shortlist to synthesise and assay, and it replaces a search, not an experiment. Generative chemistry Propose molecules conditioned on a target, a scaffold or a set of property constraints, rather than picking from a catalogue. Whether the molecule can be made at all is a separate model. Property prediction Solubility, permeability, metabolic stability, cardiac ion channel liability. These filter a list early and cheaply. They are trained on assay data and inherit its coverage: they are most reliable on chemistry that resembles what has been tested. Retrosynthesis Plan a route from purchasable starting materials. This is the application closest to a solved probl

I’m curious whether there is now a theoretical or empirical “sweet spot” for LLM quantization, preferably research done using open-source formats like GGUF Suppose you have a fixed memory/compute budget and can choose the model size freely. For example, instead of a smaller model at 8-bit or 4-bit, you could fit a progressively larger model at 3-bit, 2-bit, 1.5-bit, etc. A few years ago, I remember 4-bit often being described as roughly the practical sweet spot because it preserved most model quality while giving a large memory reduction. But with newer methods, I’ve seen surprisingly strong 3-bit, 2-bit, and even ~1.5-bit results. So if the goal is maximum model capability for a fixed memory budget , rather than preserving one particular pretrained model as faithfully as possible, what does current research suggest is the optimal bits-per-weight? Is there evidence that, for example, a 2-bit 70B model generally beats a 4-bit 35B model, or does quantization degradation eventually outweigh the gains from additional parameters? I’m especially interested in recent theoretical/scaling-law work or large empirical studies from 2025–2026. If no one is studying this, then could any of you do this work? I feel like it could be immensely useful for the community. submitted by /u/takuonline [link] [留言]

To all those in the US: Are you planning to go Sydney or Atlanta this year for NeurIPS? submitted by /u/rsesrsfh [link] [留言]

Table of Contents Overview AI Didn't Remove the Skill, It Relocated the Skill Vibe Coding...

It's an MLP architecture with around 500K total parameters. Top1 Training accuracy: 5.11% Validation accuracy 4.59% Detailed Validation accuracy numbers: Top-1 Acc: 4.59% Top-3 Acc: 9.44% Top-5 Acc: 12.68% Top-10 Acc: 18.53% The model was trained on a downscaled version of the Imagenet-1k dataset (32x32) for 5 epochs. I used pytorch for the training and pyarrow for the dataset, all within termux. Before anyone comes at me for using an MLP instead of a CNN or similar it's mainly because on my phone an MLP was just more stable, and trained 10-30x faster/step (could be my fault but I'm not too sure). This model specifically took around 30 minutes to train (6 minute/epoch) The training was entirely on the CPU which is a Dimensity 9300+ and I used 4 of the Arm Cortex-X4 cores. I might make an improved version later on as this one isn't very accurate. submitted by /u/Tall_Abrocoma_3533 [link] [留言]

Every anomaly detector answers one question: compared to what? That comparison, the baseline, is where I lost the most time on this project, and every failure had the same signature. Nothing errored. No test went red. The numbers stayed plausible. The detector just quietly stopped detecting. Four of them, in the order I found them. 1. The peer group contained the client it was judging Cold-start clients have no history, so they're compared against a pool of other clients' recent benign windows. Reasonable. The pool was keyed by feature: private readonly peer = new Map < FeatureKey , number [] > (); Every benign window every client produced went into the pool that client was later compared against. Including itself. So a client could define its own normality . Feed in enough windows and any behaviour becomes unremarkable — which is precisely the cold-start attacker the layer exists to catch. What made me look was not reasoning, it was an experiment that wouldn't sit still. I was trying to build a demo client that reliably landed in the middle of the response ladder, and holding the traffic shape fixed while changing only the request interval flipped the outcome between allow and step_up : gap=500ms origins=5 → allow (peak 0) gap=700ms origins=5 → step_up (peak 83) gap=800ms origins=5 → allow (peak 0) A knife edge like that is never a tuning problem. The outcome depended on a race between a client's own samples reaching the pool and the pool being consulted. Fix: key the pool per client, and exclude the client under evaluation. for ( const [ clientId , values ] of byClient ) { if ( clientId === excludeClientId ) continue ; // this is what "peer" means … } Afterwards the behaviour became monotone in the actual evidence, and identical at every request interval: origins 1 3 4 5 6 peak score 17 35 59 83 100 tier allow log throttle step_up deny Lesson: if a parameter that shouldn't matter changes the outcome, stop tuning and go find the defect. Knife edges are symptoms. 2.