Stop Treating LLM API Errors Like Normal HTTP Errors

Most backend engineers already know how to handle HTTP errors. 400 means the request is bad. 401 means auth failed. 429 means rate limited. 500 means something broke upstream. Retry a few times, add exponential backoff, log the response body, move on. That works fine for many APIs. It works badly for LLM APIs. LLM providers may use normal HTTP status codes, but the operational meaning behind those errors is different enough that treating them like ordinary REST failures can make your app slower, more expensive, and harder to debug. The mistake I kept making Early on, I handled LLM failures the same way I handled every other external API: if ( response . status === 429 || response . status >= 500 ) { retryWithBackoff (); } Simple. Familiar. Dangerous. That logic misses the actual question your app needs to answer: What kind of LLM failure happened, and what should the product do next? Because an LLM API failure is rarely just "one HTTP request failed." It can break: a user-facing chat response a background agent run a document generation job a tool-calling workflow a batch evaluation pipeline a structured JSON generation step And each one needs different handling. Not all 429s mean the same thing For a normal API, 429 Too Many Requests usually means: Slow down and retry later. With LLM APIs, 429 can mean several different things. It might be a temporary rate limit: { "error" : { "message" : "Rate limit reached" , "type" : "rate_limit_error" } } Retrying with backoff may help here. But it might also mean quota exhaustion: { "error" : { "message" : "You exceeded your current quota" , "type" : "insufficient_quota" } } Retrying this does not help. It just adds latency, noisy logs, and a worse user experience. It could also be model-specific pressure. One model may be overloaded while another model from the same provider, or a different provider, would work fine. So your handler should distinguish between: temporary rate limit hard quota exhaustion model-level capacity is