标签：#llmo

Everyone's talking about agentic AI shipping production code. Nobody's talking about what happens when you actually sit down and grade thousands of lines of it against a rubric, line by line, for months. I have. And the failure pattern that shows up over and over isn't the one Twitter/X is arguing about. The job title that didn't exist two years ago "AI evaluator." "AI trainer." "Expert contributor to frontier model training data." None of these existed as job titles when I started my career. Now they're where a chunk of the most interesting engineering signal in the industry is actually happening — quietly, behind NDAs, far from the demo videos. Here's what the job actually is: agentic coding outputs land on your desk, and you grade them against a structured rubric — correctness, instruction adherence, quality, edge-case handling. You design adversarial prompts to find where the model's reasoning breaks. You decide which checks can be programmatic and deterministic, and which genuinely need a human who's shipped production systems to make the call. This is RL environment design and LLMOps in its rawest form, and it's a completely different skill from "prompt engineer" or "ML researcher." It's closer to being a QA lead for a junior engineer who never sleeps, never gets embarrassed, and will confidently ship the wrong answer with perfect syntax. The pattern: agents are great at code, bad at consequences Here's the uncomfortable part. The failure mode people are loudest about — hallucinated APIs, made-up library functions — is the easy failure mode. It's loud, it's obvious, and any decent test suite catches it in seconds. The failure mode that actually matters, the one that slips past a surface read and even past a naive test suite, looks like this: The code is syntactically perfect and semantically wrong about failure. It handles the happy path beautifully and quietly assumes the retry, the timeout, the partial write, the duplicate message never happens. It optimises

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

When I decided to ship this blog in four languages, I had a clear mental ranking. English would win on volume. Spanish would be runner-up because of the sheer speaker count. Japanese would stay steady because it's my native language. Portuguese, I figured, was the long tail. I added it mostly out of completism. Twenty-two days later, the GA4 snapshot disagrees with every part of that ranking. PT: 748 pageviews , 709 sessions EN: 195 pageviews , 176 sessions JA: 27 pageviews , 29 sessions ES: 7 pageviews , 7 sessions That is Portuguese pulling roughly 3.8× English, 28× Japanese, and 107× Spanish on the same blog, same publishing cadence, same author. One Portuguese article on its own (a post about a 24-hour security agent: 375 PV) got more pageviews than my entire English blog combined. I wrote that article hoping Spanish would surprise me. Instead Portuguese surprised me, and Spanish quietly continued to not exist. The setup, so you can discount my numbers properly This is not a clean comparative experiment. It's a single blog, kenimoto.dev , running four language directories ( /en/ , /ja/ , /pt/ , /es/ ). Articles get translated through a cross-language LLM pipeline, then hand-edited for register and locale (BR Portuguese vs PT Portuguese, LatAm-neutral Spanish vs Spain Spanish). The window: 2026-04-30 to 2026-05-21, 22 daily snapshots. EN has 26 articles. JA has 25. PT has 17. ES has 10. So PT has fewer articles than EN and still beats it almost 4 to 1. If you stop reading here, take this one thing: language asymmetry can swallow article-count asymmetry whole . Adding articles in a saturated language is slower than adding articles in an underserved one. Why Portuguese pulled ahead I don't think the answer is "Portuguese readers like me more." I think three asymmetries are stacking on top of each other. 1. TabNews is a community door English doesn't have TabNews is a Brazilian developer community where you can post a technical article and have it actually read by h