I Got 28 TPS Out of Free Kaggle GPUs. Here's What It Took.

I want to be upfront about something: this whole project runs on free Kaggle T4 notebooks, an AWS EC2 t3.micro relay that costs almost nothing, and public internet. No A100s. No private datacenter network. No budget. And yet, ShardFlow v2.1 hits 28.10 TPS peak on Qwen2.5-7B across two separate cloud regions over WAN. This is the story of how that happened, and specifically the one fix in v2.1 that I did not see coming. The Problem: Running a 7B Model When You Have No Money A 7B parameter model in FP16 needs roughly 15 GB of VRAM. A single Kaggle T4 has 16 GB. Technically it fits, barely, with nothing left over for a KV cache. The solution is tensor parallelism: split the model across two machines. Node 0 (Iowa) handles layers 0 to 14. Node 1 (Oregon) handles layers 14 to 28, plus the LM head and final verification. They talk to each other through a TCP relay running on an EC2 t3.micro in Ohio. The baseline throughput with this setup and no tricks: 4.92 TPS. Usable, but not fast. Speculative Decoding: The Idea LLM inference is slow because it's sequential. You generate one token, wait, generate another, wait. Each round trip across WAN costs you ~86ms RTT. At 1 token per round trip, you're fighting the network the whole time. Speculative decoding flips this. Instead of sending one token at a time, you run a tiny draft model locally to guess the next K tokens ahead. Then you send all K guesses to the verifier in one shot. If the big model agrees with M of them, you've committed M tokens in a single round trip instead of one. ShardFlow uses Qwen2.5-0.5B as the draft model, running on cuda:1 of Node 0 while the 7B target slice runs on cuda:0. Zero VRAM contention. The drafter proposes 8 candidates, Node 1 verifies them all in parallel, and you get an average of 4.07 tokens per round trip instead of 1. With speculative decoding in eager mode: 14.3 TPS peak. 3x better. The Wall I Hit I thought 14.3 was the ceiling. The network was the obvious bottleneck: two Kaggle instan