标签：#inference

Researchers from UC Berkeley and MIT have developed FreeToken, an open-source inference engine that enhances the utility of Mixture-of-Experts models on consumer hardware. By implementing a dynamic scheduling policy and optimising weight management, FreeToken improves decoding speeds and execution efficiency in edge AI applications, fostering self-hosted reasoning systems. By Olimpiu Pop

I want to be upfront about something: this whole project runs on free Kaggle T4 notebooks, an AWS EC2 t3.micro relay that costs almost nothing, and public internet. No A100s. No private datacenter network. No budget. And yet, ShardFlow v2.1 hits 28.10 TPS peak on Qwen2.5-7B across two separate cloud regions over WAN. This is the story of how that happened, and specifically the one fix in v2.1 that I did not see coming. The Problem: Running a 7B Model When You Have No Money A 7B parameter model in FP16 needs roughly 15 GB of VRAM. A single Kaggle T4 has 16 GB. Technically it fits, barely, with nothing left over for a KV cache. The solution is tensor parallelism: split the model across two machines. Node 0 (Iowa) handles layers 0 to 14. Node 1 (Oregon) handles layers 14 to 28, plus the LM head and final verification. They talk to each other through a TCP relay running on an EC2 t3.micro in Ohio. The baseline throughput with this setup and no tricks: 4.92 TPS. Usable, but not fast. Speculative Decoding: The Idea LLM inference is slow because it's sequential. You generate one token, wait, generate another, wait. Each round trip across WAN costs you ~86ms RTT. At 1 token per round trip, you're fighting the network the whole time. Speculative decoding flips this. Instead of sending one token at a time, you run a tiny draft model locally to guess the next K tokens ahead. Then you send all K guesses to the verifier in one shot. If the big model agrees with M of them, you've committed M tokens in a single round trip instead of one. ShardFlow uses Qwen2.5-0.5B as the draft model, running on cuda:1 of Node 0 while the 7B target slice runs on cuda:0. Zero VRAM contention. The drafter proposes 8 candidates, Node 1 verifies them all in parallel, and you get an average of 4.07 tokens per round trip instead of 1. With speculative decoding in eager mode: 14.3 TPS peak. 3x better. The Wall I Hit I thought 14.3 was the ceiling. The network was the obvious bottleneck: two Kaggle instan

I saw "MTP round-trip" on a checklist for a Megatron conversion pipeline and had no idea what it meant. Two acronyms, one hyphen, apparently important enough that someone had listed it as a thing to verify. Working out what it meant took me somewhere I didn't expect. The interesting part turned out not to be MTP at all — it was the reason speculative decoding works in the first place, which rests on a fact about hardware that I had backwards. TL;DR Generating text is slow because it's sequential: one full forward pass per token. But a forward pass over five tokens costs about the same as over one. Generation is bottlenecked by moving weights, not by arithmetic. Speculative decoding exploits that: something cheap drafts k tokens, the big model verifies all of them in one pass. It is exact , not an approximation. Same output distribution as normal decoding. MTP (Multi-Token Prediction) is one way to produce those drafts — a small module trained into the model itself. MTP has two separate lives: a training-time auxiliary loss you can throw away, and an inference-time draft head you can't. Why generating text is slow To produce token N+1, the model needs token N. There's no way around that ordering — it's what "language model" means. So generating 100 tokens means 100 full passes through the network. For a model like GLM-5.2, that's 78 layers, 100 times over. The obvious conclusion is that generation is 100 times as expensive as reading the prompt. The obvious conclusion is wrong, and the way it's wrong is the whole point. The part that got me A forward pass processing one token and a forward pass processing five tokens take roughly the same wall-clock time. I had assumed compute scaled with tokens. It doesn't, because compute isn't the bottleneck. Every forward pass has to read the model's weights out of memory and into the compute units. That's hundreds of gigabytes moving across a memory bus, and it happens whether you're processing one token or fifty . The actual ar

The idea that GPUs are poorly suited for agentic workflows may be a misconception, according to French startup Kog.

Meta AI Research has introduced Muse Glimmer, a 30-billion-parameter open-weight model under the Apache 2.0 license, designed for local workflows. It enables autonomous agents and complex task execution on consumer GPUs without relying on cloud APIs. The model employs a multi-stage training approach for efficient performance and supports multimodal inputs, enhancing coding and automation tasks. By Olimpiu Pop

Anthropic and OpenAI are racing to scale up while reducing dependence on Nvidia.

Telecom expects AI revenue from dark fiber deals and retrofitted data centers.

Netflix has described the production lessons behind bringing LLM inference into its internal serving platform, including the challenges of supporting different model sizes, hardware requirements, and rapidly evolving inference engines. By Matt Foster

ZML, a hot French AI startup endorsed by Turing Award winner Yann LeCun, has now released ZML/LLMD, software that could make running AI less costly.

GPU programming usually asks Rust developers to surrender the borrow checker at the launch boundary: references collapse into raw pointers, and aliasing, synchronization, and stream lifetimes become hand-managed invariants. A new NVIDIA Labs paper argues that trade is unnecessary. How cuTile Rust Extends the Borrow Discipline to GPU Dispatch cuTile Rust is a tile-based DSL that carries Rust's ownership and borrowing rules across the host-to-GPU launch boundary — not just through host code. Introduced in "Fearless Concurrency on the GPU" (arXiv:2606.15991), submitted by NVIDIA researchers Melih Elibol, Jared Roesch, Isaac Gelado, Eric Buehler, and Michael Garland , it lets you author the kernel itself in idiomatic, memory-safe Rust rather than wrapping hand-written unsafe CUDA. The mechanism is type construction, not a runtime lock. Before launch, mutable output tensors are partitioned into provably disjoint tiles; each tile program then receives an exclusive &mut view of its slice, while inputs arrive as shared & references . Because the partitions cannot overlap, the kernel is single-threaded in its semantics and data-race-free by construction, yet still compiles to massively parallel GPU code. As Melih Elibol put it, "each tile program gets an exclusive &mut view of its memory, plus the inputs as shared references" (source: users.rust-lang.org ). Explicit unchecked types remain available for local opt-out when you need lower-level control. The safety story would be academic if it cost throughput, but the reported numbers say otherwise. On an NVIDIA B200, cuTile Rust reaches 7 TB/s on memory-bound element-wise operations and 2 PFlop/s on GEMM — roughly 96% of cuBLAS, and within measurement noise of cuTile Python . End to end, the companion Qwen3 inference engine Grout reaches 171 generated tokens/s for Qwen3-4B on an RTX 5090 and 82 tokens/s for Qwen3-32B on a B200 in batch-1 decode . Those are the authors' own measurements on specific hardware — independent reprod

LLM costs scale linearly with usage. A system processing 10,000 requests a day at $0.01 per request costs $100 daily — $365 a year. At enterprise scale, that's over $10,000. Cost optimization isn't about cutting corners. It's about spending tokens where they matter. Every token you waste is a token you could have spent on a better answer. Token budgeting The simplest way to control costs is to set limits. Per session, per task, or per day. Strategy 1: Per-Session Budgets Per-session budgets are straightforward: class SessionBudget : def __init__ ( self , budget_tokens : int = 10000 ): self . budget = budget_tokens self . used = 0 def allocate ( self , tokens : int ) -> bool : if self . used + tokens <= self . budget : self . used += tokens return True return False def remaining ( self ) -> int : return self . budget - self . used Strategy 2: Per-Task Budgets Per-task budgets are more useful. Different tasks need different amounts of context: task_budgets : classify : max_tokens : 100 model : qwen2.5-1.5b summarize : max_tokens : 500 model : qwen2.5-7b code_review : max_tokens : 2000 model : qwen2.5-coder-7b reason : max_tokens : 4000 model : qwen2.5-32b Strategy 3: Adaptive Budgets Adaptive budgets adjust based on what actually happens. If classification tasks consistently use 80 tokens, stop allocating 100: class AdaptiveBudget : def __init__ ( self ): self . task_history = {} def allocate ( self , task_type : str ) -> int : if task_type in self . task_history : return int ( self . task_history [ task_type ] * 1.5 ) return 1000 def record ( self , task_type : str , tokens_used : int ): if task_type not in self . task_history : self . task_history [ task_type ] = tokens_used else : self . task_history [ task_type ] = ( 0.9 * self . task_history [ task_type ] + 0.1 * tokens_used ) The exponential moving average (0.9 weight) means recent usage matters more than history. Adjust the weight based on how volatile your workloads are. API vs local inference Local inference

Running a 70B parameter model to summarize a 200-word email is wasteful. Running a 3B model to review production code is reckless. Most systems live somewhere in between — and that's where model routing comes in. It matches task complexity to model capability. The tradeoffs are real, but the savings are too. The routing problem People usually start with one model and stick with it. That works until you notice the cost, or the latency, or both. The alternative is building a router — something that decides which model handles which request. Four strategies work in practice: Capability-based — route by what the model can do Cost-aware — route by what you're willing to spend Latency-aware — route by how fast you need it Hybrid — combine them Each optimizes something different. Picking one is usually a decision about what hurts most. Capability-based routing The simplest approach. Classify the task, send it to the model that handles it. Task Model size Examples Classification, tagging 1-3B Qwen2.5-1.5B, Gemma-2-2B Summarization, extraction 3-7B Qwen2.5-7B, Llama-3.1-8B Code generation 7-14B Qwen2.5-Coder-7B, DeepSeek-Coder-V2 Complex reasoning 14-32B Qwen2.5-32B, Llama-3.1-70B Creative writing, analysis 32B+ Qwen2.5-72B, Claude, GPT-4 If the task doesn't need the bigger model, don't use it. A 1.5B model handles sentiment classification fine. It just won't write a coherent essay. Implementation is straightforward: ROUTING_RULES = { " classify " : { " model " : " qwen2.5-1.5b " , " max_tokens " : 100 }, " summarize " : { " model " : " qwen2.5-7b " , " max_tokens " : 500 }, " code_review " : { " model " : " qwen2.5-coder-7b " , " max_tokens " : 2000 }, " reason " : { " model " : " qwen2.5-32b " , " max_tokens " : 4000 }, " creative " : { " model " : " claude-sonnet-4 " , " max_tokens " : 8000 }, } def route_request ( task_type : str ) -> dict : return ROUTING_RULES . get ( task_type , ROUTING_RULES [ " reason " ]) The catch is classification itself. If you get the task type

Startup Baseten is reportedly close to finalizing a $1.5 billion round at a $13 billion as the “inference gold rush" marches on.

Crescent Island is an air-cooled chip that uses LPDDR5 memory.