Deploying Agents: Containers, Orchestration, and Scaling the Loop
Book: Agents in Production — Building, Tracing, and Shipping Multi-Step AI You Can Trust Also by me: Observability for LLM Applications — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub The agent works on your laptop. It passes evals. Your manager asks when it ships and you say Monday, because the modeling is done. Then you try to put it behind a load balancer and it falls apart, because you deployed it like a web service. An agent is not a web service. A web service answers in milliseconds and forgets. An agent thinks for minutes, burns tokens across two or three providers, streams partial output to a browser, and sometimes decides to call delete_invoice on the eighth turn. Every deployment decision you make flows from one question: what does this thing do to your infrastructure while it is running? Here is how to package it, where to hold state, and how to scale a workload whose bottleneck is a model call you do not control. The shape is decided by the longest step The single rule that saves you the most pain: an agent's deployment shape is decided by its longest step, not its average step. A support chatbot answers in two seconds. A code-review agent thinks for six minutes. A research agent runs for forty. You cannot put all three behind the same HTTP endpoint and expect any of them to survive. Pick the pattern that matches the longest step, then cap the rest with timeouts. Under 30s → stateless HTTP endpoint (Cloud Run, Fly.io). 30s to 5m with a user watching → streaming over WebSocket or SSE. 5m to an hour, async → queue plus worker (Temporal, Inngest, or Redis). Longer than an hour → still queue plus worker, whether you like it or not. Do not hold an HTTP request open for forty minutes. Something you did not know existed will kill it at the worst moment: a proxy, a CDN, a load-balancer idle timeout. Package it: p