标签：#trace

Why Workflows Need a Dedicated Evaluation Framework Traditional software testing covers code correctness. Workflows add two layers of uncertainty: LLM output is non-deterministic : the same input can produce different results across runs Cross-step dependencies : a Phase 3 problem may only surface at Phase 7, making the debugging chain long Without an evaluation framework, every workflow change requires a full end-to-end run: slow, expensive, incomplete coverage. Three-layer testing decomposes the problem. Three-Layer Evaluation Structure Layer 3: End-to-end tests (Workflow level) Full pipeline from trigger to completion Test cases: eval/cases.yaml Metrics: completion rate, Phase 4 avg rounds, gate trigger rate Layer 2: Integration tests (Phase level) Cross-step data flow is correctly passed Cross-phase routing logic fires correctly Layer 1: Unit tests (Step level) Each subagent's output matches its output contract No real LLM calls — validates JSON schema only Test priority: Layer 1 should be the most numerous and fastest — catches contract violations in seconds. Layer 3 is the slowest and most expensive — run it only when changes affect the main pipeline. Layer 1: Step-Level Unit Tests Unit tests verify that subagent output files match the declared schema. No real LLM calls needed. # tests/unit/test_phase3_output.py import json from pathlib import Path def test_analysis_output_schema (): """ Phase 3 output must conform to analysis_final.json schema """ output = json . loads ( Path ( " test_fixtures/phase3/analysis_final.json " ). read_text ()) assert " passed " in output assert isinstance ( output [ " passed " ], bool ) assert " confidence " in output assert 0.0 <= output [ " confidence " ] <= 1.0 assert " root_cause " in output assert isinstance ( output [ " root_cause " ], str | type ( None )) assert " evidence " in output assert isinstance ( output [ " evidence " ], list ) # on failure, error field must be present and non-empty if not output [ " passed " ]: ass

TL;DR We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Dynatrace via the Log Ingest API v2. The real value: Dynatrace's Davis AI can automatically correlate file access anomalies with application performance degradation — answering "why is the app slow?" with "because 500 users hit the same NFS share simultaneously." FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Dynatrace Log Ingest API v2 │ ▼ Davis AI ┌───────────────────┐ │ Correlates: │ │ • File access │ │ anomalies │ │ • APM metrics │ │ • Infrastructure │ │ health │ │ │ │ → Root cause │ │ in seconds │ └───────────────────┘ Verified on Dynatrace SaaS Trial (Tokyo-equivalent region). Logs visible in Logs Viewer within 1-2 minutes. This is Part 11 of the Serverless Observability for FSx for ONTAP series. Why Dynatrace for FSx for ONTAP? Most observability tools treat storage logs as isolated data. Dynatrace is different — it builds a topology map of your entire stack and uses Davis AI to find causal relationships through time-window correlation and entity connectivity: Scenario Without Dynatrace With Dynatrace App latency spike "Check the logs" Davis AI detects temporal correlation: file access to /vol/data/ increased 10x within the same 5-minute window as app response time degradation, connected via topology (app → NFS mount → SVM) Storage I/O anomaly Manual investigation Automatic correlation via shared topology entities — Davis identifies which services are affected based on entity relationships User reports slow file access Grep through audit logs DQL query + topology view showing the full dependency path from user request to storage operation The key differentiator: Davis AI correlates events across entities that share topology connections within overlapping time windows — not just keyword matching or manual dashboard correlation. Architecture ┌─────────────────────────────────────────────────────────┐ │ Event Sources │ ├─────────────────────────────────────────