Put the LLM last: I replaced a 7B model with a tiny Go classifier
TL;DR : most production AI tasks are not LLM tasks. To triage my email, I replaced a 7-billion-parameter model with a tiny classifier in Go. The rule fits in one sentence. Rules first, a small model next, the LLM only as a last resort. The result: no GPU, sub-millisecond inference, and a cloud call that became rare. Here is how, with the real numbers. This article is for developers who put an LLM in production and pay the bill. Not a demo. Most AI tasks are not LLM tasks In 2026, the default reflex is to wire a big model into everything. A question comes in, you call the LLM. But many tasks do not need it. Filing an email under "work" or "newsletter" is classification. A problem solved for twenty years, long before LLMs. To classify is to pick a label from a short, stable list. To generate text is something else. The first job needs a small model. The second earns a big one. The rule I defend fits in one sentence. Put the LLM last. The setup I built an agent that triages my inbox. It is a daemon. It reads new messages and files each one into a category: work, notification, newsletter, promo, and a few more. Nothing secret, just my real mailbox, with years of mail. The first version handed every email to a local LLM. A 7-billion-parameter model, Qwen 2.5 7B, served by Ollama on a GPU. Ollama is a tool that runs an LLM on your own machine. It worked. But the price was heavy. A GPU on all the time. One more container to watch. And an absurd slowness for the question asked. One day I asked myself: does deciding "is this a newsletter?" really need 7 billion parameters? No. The answer sits in two or three words from the sender and the subject. So I rethought the whole thing. Three layers, from cheapest to most expensive Every email goes through three layers, in order. It stops at the first one that can answer. Deterministic rules. Instant, exact, no cost. A small model. Sub-millisecond, on CPU. The LLM. Only if the small model is unsure. The routing code fits in a few lin