Alerting on LLM Metrics Without Alarm Fatigue
Most LLM alerting starts as a threshold on latency and a threshold on error rate, fires nine times in the first week, and is muted by the second. The fix is not better thresholds. It is a different trigger model and a much shorter list of things allowed to page. Level-triggered, not edge-triggered An edge-triggered alert fires on a transition: latency crossed 3 seconds, error rate spiked. It is easy to write and it is why your phone buzzed at 03:00 about a condition that resolved itself in forty seconds. A level-triggered alert asks a different question — is the system currently in a bad state, and has it been for long enough to matter? Concretely, the difference is that the alert condition is evaluated over a window and describes a sustained state, and it clears when the state clears rather than when someone acknowledges it. Every rule below is of that shape. Anything that fires on a single scrape does not belong in a paging policy; put it in a dashboard. Page on symptoms, ticket on causes The reliable partition, straight out of ordinary SRE practice and entirely applicable here: Page when users are being harmed now, and a human can do something about it in minutes. That is a small list: the feature is failing, the feature is unusably slow, or money is leaving the building at an unplanned rate. Ticket when something is degraded, trending wrong, or will bite in days. Rising retry rate. One provider slower than usual while failover is absorbing it. Attribution coverage slipping. Neither for everything else. If nobody would act on it, it is a chart. The distinction matters more for LLM features than for a normal service because so many of the interesting signals are causes : a provider 429 rate, a fallback rate, a cache-hit drop. If failover is working, none of those are user-visible and none of them should wake anyone. They are exactly what you want in the morning ticket queue. Burn-rate alerts, with the numbers The standard design — described in Google’s Site Reliab