Latency vs. Tokens: What I Learned Optimizing an Agent with Gemma (and What Didn't Work)
I'd been waiting for more than 30 minutes. The terminal just sat there, blinking, without returning a single word. I'd launched Gemma2 in its 9-billion-parameter version on my laptop (a regular Mac, the kind any professor or student would use) and the model simply wasn't responding. It wasn't a bug. It was the most honest answer the experiment could have given me. That frustrating wait ended up being, without exaggeration, the most interesting finding of the whole process. Because the question that brought me there wasn't "how big can a model get?" — it was a much more practical one: what actually happens when an agent you built in a tutorial has to survive in production? I've been working with Gemma as a case study to understand that jump — from an educational prototype to something that can hold up under long conversations, limited hardware, and real users. This post is the honest summary of that process: what worked convincingly, what didn't work the way I expected, and why that "didn't work" turned out to be more useful than a clean result would have been. The real problem: why tutorials are a little dishonest Almost every conversational agent tutorial does the same thing, without saying so out loud: on every turn, it sends the model the entire previous history, all over again. Imagine that every time you added a sentence to a conversation, you had to repeat everything said before it — every message, every reply — before you could say the new one. At first you don't notice. But if the conversation runs 30 or 50 turns, you're repeating an entire novel just to add one sentence. This pattern is called linear context stacking , and it causes three concrete problems: Memory saturation — every call to the model processes an increasingly large context. Risk of hitting the token limit — every model has a maximum context window; sooner or later, you hit it. Quality degradation — there's a documented phenomenon in NLP literature called "lost in the middle" : when context