标签：#llmevaluation

LLM Evaluation System Prompts Scored Rubrics Runtime Guardrails: A Practical Guide for Production Learn how to evaluate LLM outputs in production using system prompts, scored rubrics, and runtime guardrails to prevent hallucinations and ensure quality. TL;DR: To evaluate LLM outputs in production, combine system prompts that define evaluation criteria, scored rubrics using LLM-as-a-judge for dimensions like correctness and relevance, and runtime guardrails that filter or flag unsafe outputs. This approach scales better than human review, adapts via prompt changes, and catches failures that status codes miss, as seen in the Air Canada chatbot case. Why Production LLM Evaluation Demands More Than Status Codes A 200 status code only confirms the server processed the request—it says nothing about whether the generated text is factual, safe, or useful. The Air Canada chatbot that invented a non-existent bereavement discount returned perfectly valid HTTP responses, yet the hallucinated policy led to a tribunal ruling against the airline. Production evaluation must therefore separate operational health (latency, error rates) from output quality (correctness, relevance, harmlessness). Consider a typical API call that succeeds operationally but fails qualitatively: import requests response = requests . post ( " https://api.example.com/v1/chat " , json = { " model " : " gpt-4o " , " messages " : [{ " role " : " user " , " content " : " What is Air Canada ' s bereavement policy? " }]}, headers = { " Authorization " : " Bearer $KEY " } ) print ( response . status_code ) # 200 print ( response . json ()[ " choices " ][ 0 ][ " message " ][ " content " ]) # Output: "Air Canada offers full refunds for bereavement-related cancellations..." A 200 status code and a well-formed JSON body mask a completely fabricated policy. To catch this, you need a separate evaluation layer that scores the output against a rubric. LLM-as-a-judge is a common approach, using a second model to assess the

Learning how to add evals to an LLM feature is the difference between shipping a demo and shipping a reliable product. When you embed an LLM into a real feature — a chatbot, a voice agent, a document summarizer — you’re not just calling a model. You’re betting your user’s experience on a non‑deterministic system that can silently break with every prompt tweak, model update, or edge case. That’s why we instrument every LLM feature we build with a purpose‑built eval suite. Here’s how we did it for an outbound AI calling agent and how you can do the same. Why Evals Are Not Optional LLMs are non‑deterministic: give them the same input twice, and you’ll get two different responses. That means unit tests that check for exact string matches are useless. As Pragmatic Engineer notes , you need evals to verify that the solution works well enough — because there’s no guarantee it will. When you’re building a feature that speaks to real customers, like the AI Calling Agent dashboard we built, a regression in tone or missed booking intent can cost revenue immediately. Evals turn that uncertainty into signal. How to Add Evals to an LLM Feature: A 4‑Step Workflow We’ll walk through the exact process we followed, from defining success to automating checks in CI, using the DeepEval framework as an example. You can swap in Evidently AI or build your own, but the pattern is the same. Step 1: Define Success for Your Feature Takeaway: Before you pick a metric, write down the one thing that makes the feature “done” — usually a business outcome, not a technical measure. For the AI Calling Agent, the core feature was an outbound call that books a meeting. The success criterion wasn’t “the LLM replied politely.” It was “the agent scheduled a meeting with the right time and date.” This is a reference‑based evaluation: you compare the output to a known ground truth. Evidently AI’s guide calls this pattern out as essential for regression testing and experimentation. From that criterion, we der