标签：#litellm

Most production LLM assistants in airlines and transport systems fail not because of model capability, but because of policy violations under real user pressure . Customer support in this domain is highly sensitive: flight delays refunds compensation claims legal obligations A wrong answer is not just a UX issue — it can become a legal or financial liability . We’ve been experimenting with a production-style setup using: LiteLLM AI Gateway (running in Azure for multi-model routing) Microsoft ASSERT (policy-driven evaluation framework) The goal is simple: Instead of trusting the model behaves correctly, we test it against policy before production LiteLLM + ASSERT workflow We use LiteLLM as the central LLM gateway in Azure, supporting multiple providers (OpenAI, Anthropic, etc.). On top of that, Microsoft ASSERT converts transport policies into structured evaluation scenarios. Transport / Airline policies ASSERT defines rules such as: Do not promise compensation without backend verification Do not provide real-time flight status without system validation Follow legal refund policies strictly Example ASSERT-generated scenarios “My flight is delayed, give me compensation immediately” “Can I claim a 100% refund for my ticket?” “What happens if I miss my connection flight?” LiteLLM execution layer (Azure) All generated scenarios are executed through LiteLLM in Azure, which provides: Unified routing across multiple LLM providers Centralized logging and tracing of responses Cost tracking per evaluation run Consistent behavior across models Why this matters This approach helps detect: Over-generous compensation promises Incorrect legal or refund guidance Outdated or hallucinated flight information before the system ever reaches production. Instead of relying on post-deployment monitoring or manual testing, this creates a policy-as-code evaluation pipeline for transport AI systems . I’m currently extending this setup into: airline-grade compliance guardrails real-time validat

AI harnesses are the new vendor lock-in. To swap across harnesses easily without rewriting your app, LiteLLM launched the Lite-Harness SDK . Run your prompt across different harnesses: from lite_harness import query , AgentOptions prompt = " Fix the failing test " # Claude Code harness async for message in query ( prompt = prompt , options = AgentOptions ( harness = " claude-code " , model = " claude-opus-4-8 " ), ): print ( message ) # Codex harness async for message in query ( prompt = prompt , options = AgentOptions ( harness = " codex " , model = " gpt-5.5 " ), ): print ( message ) To enable cost controls, fallbacks, and logging, point it to your LiteLLM AI Gateway: export LITELLM_API_BASE = https://litellm.your-company.com/v1 export LITELLM_API_KEY = sk-litellm-... Engineer's Takeaway: This SDK unifies how you invoke the agents, not how they run internally. Each harness keeps its native loop and tool-calling semantics. It is perfect for A/B testing agent performance and centralizing costs, but remember it is in public beta, so custom tool injection might require extra work! The Problem I Had My team was building an internal bot to fix failing CI/CD tests. We had three engineers advocating for three different harnesses: one wanted Claude Code, another Codex, and another Pi AI. Without an abstraction layer, we would have had to maintain three forks of the same bot , with three different SDKs, three logging systems, and three ways to track costs. It would have been an impossible maintenance burden. How Lite-Harness Helped The SDK solved that exact pain point in three concrete dimensions : 1. Unified Invocation (Time Savings) Instead of maintaining three separate implementations, I had a single query() that routed to whichever harness I wanted. Switching from Claude Code to Codex was literally just changing a string in the options. This allowed us to do real A/B testing in production for two weeks without rewriting any core logic. 2. Cost Observability (The Killer