Workflow Series (05): Evaluation Framework — Three-Layer Testing and Trace Tracking

Why Workflows Need a Dedicated Evaluation Framework Traditional software testing covers code correctness. Workflows add two layers of uncertainty: LLM output is non-deterministic : the same input can produce different results across runs Cross-step dependencies : a Phase 3 problem may only surface at Phase 7, making the debugging chain long Without an evaluation framework, every workflow change requires a full end-to-end run: slow, expensive, incomplete coverage. Three-layer testing decomposes the problem. Three-Layer Evaluation Structure Layer 3: End-to-end tests (Workflow level) Full pipeline from trigger to completion Test cases: eval/cases.yaml Metrics: completion rate, Phase 4 avg rounds, gate trigger rate Layer 2: Integration tests (Phase level) Cross-step data flow is correctly passed Cross-phase routing logic fires correctly Layer 1: Unit tests (Step level) Each subagent's output matches its output contract No real LLM calls — validates JSON schema only Test priority: Layer 1 should be the most numerous and fastest — catches contract violations in seconds. Layer 3 is the slowest and most expensive — run it only when changes affect the main pipeline. Layer 1: Step-Level Unit Tests Unit tests verify that subagent output files match the declared schema. No real LLM calls needed. # tests/unit/test_phase3_output.py import json from pathlib import Path def test_analysis_output_schema (): """ Phase 3 output must conform to analysis_final.json schema """ output = json . loads ( Path ( " test_fixtures/phase3/analysis_final.json " ). read_text ()) assert " passed " in output assert isinstance ( output [ " passed " ], bool ) assert " confidence " in output assert 0.0 <= output [ " confidence " ] <= 1.0 assert " root_cause " in output assert isinstance ( output [ " root_cause " ], str | type ( None )) assert " evidence " in output assert isinstance ( output [ " evidence " ], list ) # on failure, error field must be present and non-empty if not output [ " passed " ]: ass