标签：#inference

ZML, a hot French AI startup endorsed by Turing Award winner Yann LeCun, has now released ZML/LLMD, software that could make running AI less costly.

GPU programming usually asks Rust developers to surrender the borrow checker at the launch boundary: references collapse into raw pointers, and aliasing, synchronization, and stream lifetimes become hand-managed invariants. A new NVIDIA Labs paper argues that trade is unnecessary. How cuTile Rust Extends the Borrow Discipline to GPU Dispatch cuTile Rust is a tile-based DSL that carries Rust's ownership and borrowing rules across the host-to-GPU launch boundary — not just through host code. Introduced in "Fearless Concurrency on the GPU" (arXiv:2606.15991), submitted by NVIDIA researchers Melih Elibol, Jared Roesch, Isaac Gelado, Eric Buehler, and Michael Garland , it lets you author the kernel itself in idiomatic, memory-safe Rust rather than wrapping hand-written unsafe CUDA. The mechanism is type construction, not a runtime lock. Before launch, mutable output tensors are partitioned into provably disjoint tiles; each tile program then receives an exclusive &mut view of its slice, while inputs arrive as shared & references . Because the partitions cannot overlap, the kernel is single-threaded in its semantics and data-race-free by construction, yet still compiles to massively parallel GPU code. As Melih Elibol put it, "each tile program gets an exclusive &mut view of its memory, plus the inputs as shared references" (source: users.rust-lang.org ). Explicit unchecked types remain available for local opt-out when you need lower-level control. The safety story would be academic if it cost throughput, but the reported numbers say otherwise. On an NVIDIA B200, cuTile Rust reaches 7 TB/s on memory-bound element-wise operations and 2 PFlop/s on GEMM — roughly 96% of cuBLAS, and within measurement noise of cuTile Python . End to end, the companion Qwen3 inference engine Grout reaches 171 generated tokens/s for Qwen3-4B on an RTX 5090 and 82 tokens/s for Qwen3-32B on a B200 in batch-1 decode . Those are the authors' own measurements on specific hardware — independent reprod

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000. Cost optimization isn't about cutting corners. It's about spending tokens where they matter. Every token you waste is a token you could have spent on a better answer. Token budgeting The simplest way to control costs is to set limits. Per session, per task, or per day. Strategy 1: Per-Session Budgets Per-session budgets are straightforward: class SessionBudget : def __init__ ( self , budget_tokens : int = 10000 ): self . budget = budget_tokens self . used = 0 def allocate ( self , tokens : int ) -> bool : if self . used + tokens <= self . budget : self . used += tokens return True return False def remaining ( self ) -> int : return self . budget - self . used Strategy 2: Per-Task Budgets Per-task budgets are more useful. Different tasks need different amounts of context: task_budgets : classify : max_tokens : 100 model : qwen2.5-1.5b summarize : max_tokens : 500 model : qwen2.5-7b code_review : max_tokens : 2000 model : qwen2.5-coder-7b reason : max_tokens : 4000 model : qwen2.5-32b Strategy 3: Adaptive Budgets Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100: class AdaptiveBudget : def __init__ ( self ): self . task_history = {} def allocate ( self , task_type : str ) -> int : if task_type in self . task_history : return int ( self . task_history [ task_type ] * 1.5 ) return 1000 def record ( self , task_type : str , tokens_used : int ): if task_type not in self . task_history : self . task_history [ task_type ] = tokens_used else : self . task_history [ task_type ] = ( 0.9 * self . task_history [ task_type ] + 0.1 * tokens_used ) The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are. API vs local inference Local inference

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in. It matches task complexity to model capability. The tradeoffs are real, but the savings are too. The routing problem People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request. Four strategies work in practice: Capability-based — route by what the model can do Cost-aware — route by what you're willing to spend Latency-aware — route by how fast you need it Hybrid — combine them Each optimizes something different. Picking one is usually a decision about what hurts most. Capability-based routing The simplest approach. Classify the task, send it to the model that handles it. Task Model size Examples Classification, tagging 1-3B Qwen2.5-1.5B, Gemma-2-2B Summarization, extraction 3-7B Qwen2.5-7B, Llama-3.1-8B Code generation 7-14B Qwen2.5-Coder-7B, DeepSeek-Coder-V2 Complex reasoning 14-32B Qwen2.5-32B, Llama-3.1-70B Creative writing, analysis 32B+ Qwen2.5-72B, Claude, GPT-4 If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay. Implementation is straightforward: ROUTING_RULES = { " classify " : { " model " : " qwen2.5-1.5b " , " max_tokens " : 100 }, " summarize " : { " model " : " qwen2.5-7b " , " max_tokens " : 500 }, " code_review " : { " model " : " qwen2.5-coder-7b " , " max_tokens " : 2000 }, " reason " : { " model " : " qwen2.5-32b " , " max_tokens " : 4000 }, " creative " : { " model " : " claude-sonnet-4 " , " max_tokens " : 8000 }, } def route_request ( task_type : str ) -> dict : return ROUTING_RULES . get ( task_type , ROUTING_RULES [ " reason " ]) The catch is classification itself. If you get the task type

Startup Baseten is reportedly close to finalizing a $1.5 billion round at a $13 billion as the “inference gold rush" marches on.

Crescent Island is an air-cooled chip that uses LPDDR5 memory.