标签：#hitl

When you're building an AI evaluation platform with multiple microservices, the "core" services get all the attention — the evaluation engine, the scoring system, the RAG pipeline. But a platform doesn't work without the connective tissue: the workflow orchestration that keeps humans in the loop, the taxonomy engine that classifies tasks intelligently, the platform service that ties authentication together, and the evaluation suites that ensure models actually remember context. These four services don't make headlines, but they're what turned a collection of microservices into an actual platform. Here's what went into each one and why the engineering decisions mattered. Workflow Orchestration: The Human-in-the-Loop Engine AI evaluation is not fully automated — and it shouldn't be. Certain decisions require human judgment: Is this model response harmful? Does this evaluation rubric make sense for this domain? Is this edge case a genuine failure or acceptable behavior? The workflow orchestrator manages these decision points. It coordinates multi-step evaluation workflows where some steps are automated (LLM scoring, data validation) and others require human approval before the pipeline continues. The Architecture The core is a state machine built on FastAPI and PostgreSQL. Each workflow is a DAG (directed acyclic graph) of tasks, where each node can be: Automated: Runs immediately, calls another service (scoring, data enrichment), stores the result Human gate: Pauses the workflow, notifies the assigned reviewer via the notification service, waits for approval/rejection Conditional: Routes to different branches based on previous step outcomes (e.g., if confidence score < threshold, escalate to senior reviewer) State transitions are persisted in PostgreSQL with Alembic-managed migrations. Every transition is logged — who approved what, when, and with what context. This audit trail turned out to be critical for client reporting. Real-Time Updates with WebSocket The origin