标签：#latency

Recently, I've added a bunch of hype-monsters to my AI Werewolf : Kimi K3 Qwen 3.8 Max, Qwen 3.7 Plus, Qwen 3.7 Flash MiniMax M3 Plus the ones I've had for a while DeepSeek V4 Pro and Flash GLM-5.2 Sakana Fugu base and Ultra The last one is Japanese, not Chinese, but it was in the news a month or two ago and it belongs in this story. I sat down to write about how slow Chinese models are. Because they are all annoyingly slow even with a small context. I had the numbers, the thesis wrote itself, and then... I discovered something I didn't expect. Okay, let me go straight to the problem All Chinese official APIs are extremely slow. DeepSeek got better with their v4, the rest are just terrible. So slow that they are barely usable in a text game. No, seriously, take a look. Time to produce one four-sentence vote: Kimi K3 : 29 to 34 seconds MiniMax M3 : 25 to 30 seconds Qwen 3.8 Max : 25 to 27 seconds, and only because I capped its thinking. Uncapped it hit 100. DeepSeek V4 Pro : 14 to 22 seconds, the best of the group Same prompt, same afternoon: Claude 5 Opus answers in 5.9 seconds. I have a test which simulates the voting at the end of a game day. Some chat has happened, some players have already cast their votes, and now the model under test has to do the same. The prompt contains all of that: 36,000 characters, which works out to 8-13k tokens depending on the model's tokenizer. Not much, assuming those models have 1M contexts. More US/EU models: Model Time Input Output tokens Avg cost GPT-5.6 Luna 3.2-3.6s 12,124 190-231 $0.0020 Mistral Large 3 2.9-5.2s 8,175 73-134 $0.0042 Claude Fable 5 4.8-6.0s 13,027 183-209 $0.1399 Gemini 3.6 Flash 4.6-7.1s 7,979 297-532 $0.0152 GPT-5.6 Sol 6.7-7.3s 12,124 150-156 $0.0471 Claude 5 Opus 5.9-7.8s 13,027 220-387 $0.0726 Gemini 3.1 Pro 11.6-12.1s 7,979 821-940 $0.0263 Claude 5 Sonnet 6.8-14.8s 13,027 252-821 $0.0464 Grok 4.5 10.0-14.8s 8,190 428-607 $0.0149 Mistral Medium 3.5 2.9-35.0s 8,175 57-109 $0.0128 Worth staring at the input

GitHub redesigned GitHub Issues navigation using a client-side architecture that combines caching, predictive prefetching, and service workers to reduce perceived latency. The approach uses IndexedDB, in-memory caching, and background synchronization to serve data faster. GitHub reported instant navigation improvements from 4% to 22%, with latency reductions across multiple navigation By Leela Kumili

When a series of regional outages forced a rethink of a multi-region AWS API, the team discovered that an obstacle to global failover was hiding in plain sight: a pre-flight discovery call baked into every client session years earlier as the only available option. This article describes what it took to remove it, and what the rollout actually cost. By Suresh Gururajan

Building an AI agent is fun. Fixing its production latency when it's juggling live data, RAG, and text-to-speech? Not so fun. In the latest episode of the AI Agent Clinic, we sat down with developer Sami Maghnaoui to debug PlaybackIQ, a football / soccer agent he built to provide pre and post match analysis with text to voice, and minute-by-minute match insights with interactive UI. The app was awesome, but under heavy "match day" data loads, the wait times were killing the UX. Here’s how we fixed it: The Bottleneck: We implemented OpenTelemetry on the Agent Platform to trace exactly where the LLM calls and data retrieval were hanging up. The Scale: We shifted the deployment to Cloud Run to properly handle concurrent traffic. The Result: We managed to slash the agent's latency by 80%. If you're dealing with sluggish LLM response times in your own apps and want to see what a production-grade fix looks like, we recorded the whole teardown and rebuild. 🎥 Watch the teardown here: [ https://youtu.be/G7olcqETSn8 ] (Let me know in the comments what your go-to stack is for tracing LLM latency!)

Uber introduced a high-throughput financial ledger processing system designed to handle hot account write contention at scale. Using 250ms batching, Redis coordination, and optimistic atomic updates, the system supports 30+ updates per second per account while preserving consistency and auditability, reducing multi-hour processing pipelines to minutes in its distributed accounting infrastructure. By Leela Kumili

Shopify introduced GraphQL Cardinal, a new execution engine replacing depth-first traversal with breadth-first execution. The redesign improves large-scale GraphQL performance with up to 15x faster field execution, 6x lower GC overhead, and +4s P50 latency gains. It focuses on execution-layer efficiency and batched resolver processing for high-cardinality commerce queries. By Leela Kumili