Your Retry Loop Is a Token Incinerator: A Cascade Router for Mixed-Tier Endpoints

When a free endpoint returns 429, most agents do the most expensive thing possible: retry. Retrying looks harmless. A 200-millisecond request becomes a 2-second wait, then another attempt. But under peak load, that loop becomes a 30-second stall while your agent clicks refresh on an empty response. If the quota window resets during the stall, every retry burns tokens you could have spent on actual work. The retry loop assumes the failure is temporary. For rate limits, that assumption is usually wrong. Quota counters reset on a fixed schedule, not on your convenience. You are not just waiting; you are burning wall-clock time that could have gone elsewhere. The Cascade Pattern A cascade router is the alternative. It sends requests to the free endpoint, backs off on rate-limit signals, then degrades gracefully to a backup endpoint. The free tier carries the load; the backup exists only when needed. You get the cost advantage of the free tier and the reliability of the paid tier. The design has three parts: an endpoint abstraction layer, a rate-limit detector, and a circuit breaker that trips when the free endpoint fails repeatedly. Here is the core code: # cascade_router.py — free tier first, paid/self-hosted as fallback. import json import os import time import urllib.error import urllib.request from dataclasses import dataclass @dataclass class Endpoint : name : str url : str api_key : str model : str cooldown_until : float = 0.0 consecutive_failures : int = 0 def available ( self ) -> bool : return time . time () >= self . cooldown_until class CascadeRouter : def __init__ ( self , endpoints : list [ Endpoint ]): self . endpoints = endpoints def _call_one ( self , ep : Endpoint , messages : list [ dict ]) -> tuple [ int , dict ]: body = json . dumps ({ " model " : ep . model , " messages " : messages , " max_tokens " : 256 }). encode () req = urllib . request . Request ( ep . url , data = body , headers = { " Content-Type " : " application/json " , " Authorization "