I Stopped Trusting the LLM With the Score: Building an Honest AI Portfolio Reviewer

Ask a language model to score a developer portfolio out of 100 and you get a confident number back. Hand it a near-empty page with a name and a broken avatar, and it will often still tell you something like 92. Nice layout. Strong personal branding. The model is being polite, not accurate. That was the first wall I hit building Leon, the reviewer inside getfolio. If the score is not trustworthy, nothing downstream matters: the critique, the suggestions, and the fix button all hang off a number the model invented to sound encouraging. This is the build log of how I stopped letting the model hold the pen. Short version: a deterministic rules engine owns the score, and the language model only owns the words around it. The failure mode: an LLM judge wants to be liked If you have shipped anything with an LLM evaluator you have probably seen this. You hand it a rubric, a JSON schema, even worked examples, and it still drifts upward. Empty inputs get encouraging scores. Weak inputs get the benefit of the doubt. Strong inputs land in the same band as the weak ones, just with longer praise. A few reasons, roughly in order of how much they hurt: Tuning rewards a helpful, encouraging tone. Harsh scoring reads as unhelpful, so the model softens it. The model has no ground truth for what a 70 versus an 85 looks like in your specific domain. It is scoring on vibes. Scoring and explaining are entangled. The model writes the kind explanation first, then picks a number to match the nice things it just said. Run it twice on the same input and you get two different numbers. There is no anchor. For a portfolio reviewer that real recruiters and developers would act on, that was a non-starter. If Leon says 64, an empty page should not be able to reach 64 by accident, and a strong portfolio should not get talked down to it either. The number has to mean something. The fix: rules engine owns the score, model owns the language The architecture splits responsibilities hard. A deterministic e