标签：#Telemetry

OpenTelemetry has quietly become table stakes. That's a good thing, but if you've instrumented a real codebase, you know the tax. A method that does one obvious thing slowly fills up with StartActivity , SetTag , AddEvent , SetStatus . The bookkeeping of telemetry starts to drown out the intent of the code, and in review you spend half your time mentally separating "what this code does" from "what we report about what it does." It's easy to think "oh, but the framework takes care of this with auto-instrumentation", but if you talk to the experts in OTel, they'll go to great lengths to explain that auto-instrumentation is a floor, not a ceiling. Most of the value in telemetry comes from the custom instrumentation you add to your code that adds business context to your traces. And that custom instrumentation is the stuff that clutters up your code. Here's the kind of thing I mean: // before - ugly, obtuse, who put that there var orderId = order . Id ; Activity . Current ?. SetTag ( "orderId" , orderId ); Sidemark is my answer to that code-obfuscation problem: non-invasive instrumentation via what I'm calling Active Comments . // after - glorious, beautiful, basking in the light of the sun, closer to god, happy, satiated var orderId = order . Id ; //? The idea A small set of comment syntaxes - //? , //! , //?! - become ride-along annotations . They travel next to the code, get read at build time, and turn into the equivalent Activity calls in the compiled output. The code you read stays the code that does the work. The telemetry rides along instead of competing with your logic for attention. The framing is loosely inspired by Wallaby.js's Live Annotations , which project runtime values inline next to the code that produced them. Sidemark takes the same instinct in the other direction: comments as a write surface for instrumentation, rather than a read surface for debug values. Comments are an under-used channel for information about code that isn't itself code - and su

TLDR Monitoring AI agents in production requires distributed tracing: a single user request fans out into 10 or more internal operations, and logs alone cannot show you which step is slow, failing, or burning your token budget. OpenTelemetry's gen_ai.* semantic conventions give you standardized span attributes for LLM calls, tool invocations, and agent steps. Some are stable today; others are still experimental. Auto-instrumentation libraries (OpenLLMetry, OpenInference, OpenLIT) cover most agent frameworks with two to three lines of initialization code. You do not change your agent code. Traces ship to OpenObserve over OTLP. From there you get SQL-queryable trace data, token usage dashboards, cost attribution by agent and model, and alerting on latency and cost anomalies. OpenObserve also exposes an MCP server. You can query your live agent traces from a Claude or GPT session without opening a dashboard. Why Agents Are Harder to Monitor Than a Single LLM Call A single LLM call is straightforward to observe. One HTTP request, one response, one latency number. You can log the input and output and call it done. An agent is different. When a user sends a message, the agent calls an LLM to decide what to do, invokes a tool, processes the result, calls the LLM again, possibly calls another tool, and eventually returns a response. That one user message becomes ten or more internal operations. Some of those operations call external APIs. Some retry. Some spawn sub-agents. Without distributed tracing, you see none of this structure. You know the response took 8 seconds. You do not know whether the LLM took 7 of those seconds or whether a tool made three retries before timing out. Four categories of problems appear in production agents that you cannot debug without traces: Latency. Which step is slow? The LLM call? The tool execution? A retry loop the agent entered because the tool returned ambiguous output? Cost. Which agent, which task, which model is consuming tokens? A s