标签：#localinference

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000. Cost optimization isn't about cutting corners. It's about spending tokens where they matter. Every token you waste is a token you could have spent on a better answer. Token budgeting The simplest way to control costs is to set limits. Per session, per task, or per day. Strategy 1: Per-Session Budgets Per-session budgets are straightforward: class SessionBudget : def __init__ ( self , budget_tokens : int = 10000 ): self . budget = budget_tokens self . used = 0 def allocate ( self , tokens : int ) -> bool : if self . used + tokens <= self . budget : self . used += tokens return True return False def remaining ( self ) -> int : return self . budget - self . used Strategy 2: Per-Task Budgets Per-task budgets are more useful. Different tasks need different amounts of context: task_budgets : classify : max_tokens : 100 model : qwen2.5-1.5b summarize : max_tokens : 500 model : qwen2.5-7b code_review : max_tokens : 2000 model : qwen2.5-coder-7b reason : max_tokens : 4000 model : qwen2.5-32b Strategy 3: Adaptive Budgets Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100: class AdaptiveBudget : def __init__ ( self ): self . task_history = {} def allocate ( self , task_type : str ) -> int : if task_type in self . task_history : return int ( self . task_history [ task_type ] * 1.5 ) return 1000 def record ( self , task_type : str , tokens_used : int ): if task_type not in self . task_history : self . task_history [ task_type ] = tokens_used else : self . task_history [ task_type ] = ( 0.9 * self . task_history [ task_type ] + 0.1 * tokens_used ) The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are. API vs local inference Local inference

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in. It matches task complexity to model capability. The tradeoffs are real, but the savings are too. The routing problem People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request. Four strategies work in practice: Capability-based — route by what the model can do Cost-aware — route by what you're willing to spend Latency-aware — route by how fast you need it Hybrid — combine them Each optimizes something different. Picking one is usually a decision about what hurts most. Capability-based routing The simplest approach. Classify the task, send it to the model that handles it. Task Model size Examples Classification, tagging 1-3B Qwen2.5-1.5B, Gemma-2-2B Summarization, extraction 3-7B Qwen2.5-7B, Llama-3.1-8B Code generation 7-14B Qwen2.5-Coder-7B, DeepSeek-Coder-V2 Complex reasoning 14-32B Qwen2.5-32B, Llama-3.1-70B Creative writing, analysis 32B+ Qwen2.5-72B, Claude, GPT-4 If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay. Implementation is straightforward: ROUTING_RULES = { " classify " : { " model " : " qwen2.5-1.5b " , " max_tokens " : 100 }, " summarize " : { " model " : " qwen2.5-7b " , " max_tokens " : 500 }, " code_review " : { " model " : " qwen2.5-coder-7b " , " max_tokens " : 2000 }, " reason " : { " model " : " qwen2.5-32b " , " max_tokens " : 4000 }, " creative " : { " model " : " claude-sonnet-4 " , " max_tokens " : 8000 }, } def route_request ( task_type : str ) -> dict : return ROUTING_RULES . get ( task_type , ROUTING_RULES [ " reason " ]) The catch is classification itself. If you get the task type