标签：#foundry

This closes out the series' capstone: the multi-agent customer support system built across Parts 6-9, now hardened with evaluation, cost governance, and observability so it can actually run in production with an on-call rotation behind it, not just in a demo environment. Continuous evaluation pipeline Evaluation: measuring quality continuously, not just at launch A one-time eval before launch tells you nothing about drift once real traffic — and real edge cases — start hitting the system. Set up a continuous evaluation pipeline using a G-Eval-style approach, where a separate model scores production outputs against explicit criteria: eval_criteria = { " correctness " : " Does the response accurately reflect the order/refund status retrieved from the tools? " , " escalation_appropriateness " : " If the case was ambiguous or high-risk, did the agent escalate to a human rather than resolving it alone? " , " tone " : " Is the response professional and appropriately empathetic given the customer ' s stated frustration level? " , } def geval_score ( response , context , criterion_name , criterion_description , eval_model_client ): prompt = f """ Evaluate the following response against this criterion: { criterion_description } Context: { context } Response: { response } Score from 1-5 and give one sentence of reasoning. Return JSON: {{ " score " : int, " reasoning " : str}} """ result = eval_model_client . complete ( prompt ) return json . loads ( result ) def run_continuous_eval ( sample_of_production_traffic ): scores = { crit : [] for crit in eval_criteria } for interaction in sample_of_production_traffic : for crit_name , crit_desc in eval_criteria . items (): result = geval_score ( interaction . response , interaction . context , crit_name , crit_desc , eval_model_client ) scores [ crit_name ]. append ( result [ " score " ]) return { crit : sum ( vals ) / len ( vals ) for crit , vals in scores . items ()} Sample a percentage of real production traffic daily (not just s