You Benchmarked the Model. Now Benchmark the Server.
You picked a free model because the answers looked good. Good answers are not an endpoint. An endpoint is the model plus the server plus the network. Demos pass. Pipelines stall. The model was rarely the problem. So why do we keep benchmarking only the model? Because it is easy. You paste a prompt. You read the output. You declare a winner. The server never gets a vote. This post is a reproducible benchmark. It measures the pair, not the model. Run it before you wire any free endpoint into CI. The Pair, Not the Model Most evaluations compare answers. You paste a prompt. You judge the output. You pick a winner. That measures the model. It ignores the server. Free model access usually means a shared endpoint. A free server option means shared tenancy. Other users share the CPU, memory, and network. Your latency is their latency. Your timeout is their timeout. Here is the scenario I keep seeing. A team evaluates a free model on Friday. The answers look great. They wire it into CI on Monday. By Wednesday, the pipeline is red. The model did not change. The server did. A neighbor started a batch job. Now every request queues behind it. I applied the same harness to MonkeyCode's free model access and their free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I did not trust the demo. I built a harness instead. The Harness A benchmark needs three things. A fixed prompt set. A concurrency ladder. A pass/fail table. Here is the harness I use. #!/usr/bin/env python3 """ Benchmark a model endpoint as a pair: model + server. """ import argparse import asyncio import json import statistics import time import httpx PROMPTS = [ " Say OK. " , " Classify this log line: ERROR disk full " , " Return one word: is 429 a retryable status? " , ] async def fire ( client , url , payload , sem , timeout = 30 ): async with sem : start = time . perf_counter () try : r = await client . post ( url , json = payload , timeout = timeout ) return r . sta