Route by Task, Not by Hype: A Budget-Aware Harness for Trying New Coding Models

Every few weeks a new checkpoint drops and the timeline fills up with claims that it's cheaper, smarter, and about to change everything. Some of those claims hold up. Many don't. And even when a model genuinely is better on public leaderboards, that tells you almost nothing about whether it's better on your codebase, your tasks, and your budget . I wrote previously about building a reproducible harness before wiring any model into your workflow. This article is the sequel nobody asked for but everybody needs: once you have a harness, how do you evaluate a steady stream of new models without spending a steady stream of money? The answer I keep coming back to is routing by task difficulty : don't run your whole eval suite against every candidate. Tier your tasks, send the cheap ones to cheap models, and reserve expensive runs for the cases that actually discriminate between models. The problem with "run everything against everything" If your eval suite has 60 tasks and a new model appears every two weeks, naive evaluation costs scale linearly forever. Worse, most of those runs are wasted signal: Easy tasks (rename a variable, write a docstring, fix an obvious off-by-one) are solved by almost every current model. Running a frontier-priced model on them tells you nothing. Medium tasks (implement a small feature against an existing test, refactor across two files) are where models actually diverge. Hard tasks (multi-file reasoning, subtle concurrency bugs, unfamiliar framework internals) discriminate strongly but are few — and they're where failures are expensive to verify. So the harness should spend its budget where the signal is. A concrete artifact: a tiered router in ~80 lines of Python Here's a minimal, runnable sketch. It assumes your eval tasks are JSON files with a tier field ( easy , medium , hard ) and a verify command you can execute (a test suite, a diff check, whatever your harness already uses). # router.py — tiered evaluation router (working sketch, adapt