I Built a 40-Minute Evaluation for Free Model Endpoints. Here's the Scorecard.

Free model endpoints are seductive. Zero cost. Zero setup. Zero reason to trust them. I don't trust demos. I trust failure modes. So I built a small evaluation harness. It tests one thing: can a free model endpoint gate a pull request for secrets? This is not a benchmark. It's a repeatable experiment. You can run it in an afternoon. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model endpoint and the free server option for the test. No quotas. No hardware claims. Just a harness and a rubric. Why I stopped trusting free endpoints Free endpoints look great in a demo. You paste a diff. The model finds the secret. Everyone claps. Then you wire it into CI. The JSON breaks. The latency spikes. The model misses a private key. The demo didn't show that. An evaluation will. The experiment I designed a 40-minute test. It answers one question: where does the free endpoint perform well, and where does it break? The dataset is 30 synthetic diffs. Fifteen contain real-looking secrets. Fifteen are clean. Each diff is small. Each diff has one clear change. The prompt is strict. The model must return JSON. No prose. No apologies. Just a verdict. # eval_secret_gate.py # Simplified harness. Adapt to your client SDK. import json , time def classify ( client , diff : str ) -> dict : prompt = f """ You are a secret scanner for code review. Return ONLY JSON with this shape: {{ " contains_secret " : true, " line " : 12, " type " : " aws_access_key " }} Diff: { diff } """ start = time . time () response = client . complete ( prompt , model = " free " , server = " free " , # free server option ) latency = time . time () - start return { " latency " : latency , " raw " : response } def evaluate ( client , diffs , runs = 3 ): for i , diff in enumerate ( diffs ): for run in range ( runs ): yield i , run , classify ( client , diff ) The harness is deliberately small. It measures five things. Accuracy. JSON validity. Latency. Variance. Fa