标签：#llmops

Everyone's talking about agentic AI shipping production code. Nobody's talking about what happens when you actually sit down and grade thousands of lines of it against a rubric, line by line, for months. I have. And the failure pattern that shows up over and over isn't the one Twitter/X is arguing about. The job title that didn't exist two years ago "AI evaluator." "AI trainer." "Expert contributor to frontier model training data." None of these existed as job titles when I started my career. Now they're where a chunk of the most interesting engineering signal in the industry is actually happening — quietly, behind NDAs, far from the demo videos. Here's what the job actually is: agentic coding outputs land on your desk, and you grade them against a structured rubric — correctness, instruction adherence, quality, edge-case handling. You design adversarial prompts to find where the model's reasoning breaks. You decide which checks can be programmatic and deterministic, and which genuinely need a human who's shipped production systems to make the call. This is RL environment design and LLMOps in its rawest form, and it's a completely different skill from "prompt engineer" or "ML researcher." It's closer to being a QA lead for a junior engineer who never sleeps, never gets embarrassed, and will confidently ship the wrong answer with perfect syntax. The pattern: agents are great at code, bad at consequences Here's the uncomfortable part. The failure mode people are loudest about — hallucinated APIs, made-up library functions — is the easy failure mode. It's loud, it's obvious, and any decent test suite catches it in seconds. The failure mode that actually matters, the one that slips past a surface read and even past a naive test suite, looks like this: The code is syntactically perfect and semantically wrong about failure. It handles the happy path beautifully and quietly assumes the retry, the timeout, the partial write, the duplicate message never happens. It optimises

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a