标签：#OpenTelemetry

Where Copilot's telemetry stream lands, decided centrally GitHub added a control on July 8 that lets an enterprise mandate where the Copilot Chat extension in VS Code and Copilot CLI send OpenTelemetry data, removing the need for individual developers to set OTEL_* environment variables. Per the GitHub changelog, the setting is delivered through a telemetry block in the enterprise-managed settings, and a managed value takes precedence over environment variables and user settings. Four things are configurable in the block: the OTLP export endpoint and transport ( otlp-http or otlp-grpc ), the OTel service name and resource attributes, exporter headers such as an authentication token for the collector, and whether prompt, response and tool content is captured, with a separate flag for whether developers can change that. Delivery uses the channels documented on the same page: native MDM (Windows Registry or macOS managed preferences), server-managed settings from a signed-in GitHub account, or a file-based managed-settings.json . Where this bites The precedence rule is the point. If a platform team owns the collector and needs traces routed to it, this is exactly the switch they wanted. If a developer had their own OTLP endpoint pointed at a local sink, they will see the session start emitting somewhere else. The changelog does not describe a per-user override once a managed value is set. A scoping note is worth reading twice. The changelog states that managed exporter headers apply only to the Copilot Chat extension's OTLP exporter. The endpoint and transport policy still reach the CLI agent host, but the auth-token flow the changelog calls out is bound to the Chat surface. On-call teams standing up the collector should plan for that asymmetry before it lands as a surprise during triage.

The Cloud Native Computing Foundation (CNCF) has announced the graduation of OpenTelemetry, elevating the project to the foundation's highest level of maturity and formally recognizing it as production-ready for enterprise use. By Craig Risi

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

OpenTelemetry has become the standard for observing modern systems. But when you start building AI applications, traditional traces aren't enough. You don't just want to know that a request happened. You want to know: Which model generated the output? Which provider was used? How many tokens were consumed? What embedding model processed the documents? How much did the operation cost? These questions become even more important when building Retrieval-Augmented Generation (RAG) systems. Recently while contributing to Mastra, I discovered an observability gap involving RAG embedding operations. This led me to open a pull request that introduced proper OpenTelemetry GenAI semantic mappings for RAG_EMBEDDING spans. The Problem Mastra already exported rich metadata for several AI operations. However, RAG embedding spans were missing standardized GenAI semantic attributes. As a result, observability tools could see that an embedding operation occurred, but they couldn't easily understand: Model information Provider information Token usage Embedding-specific metadata Without standardized semantic conventions, dashboards and tracing systems lose valuable context. This becomes a bigger issue in production environments where teams need visibility into AI workloads. Understanding RAG Embedding Spans A typical RAG pipeline looks like this: Documents ↓ Chunking ↓ Embedding Model ↓ Vector Database ↓ Similarity Search ↓ LLM Generation The embedding stage is critical. Every document chunk gets transformed into a vector representation. If observability data from this stage is incomplete, debugging performance issues becomes significantly harder. Why OpenTelemetry Semantic Conventions Matter OpenTelemetry doesn't just define traces. It also defines semantic conventions. These conventions create a common language for telemetry data. Instead of every framework inventing custom field names, everyone follows the same standard. For GenAI workloads this means tools can automatically underst

You've built a checkout reactor that reserves inventory, charges a card, generates a shipping label, and sends a confirmation email. It runs through Sidekiq. When something fails, compensation logic rolls it back. It works. Then your team asks: "How many checkouts failed this week? Which step? How long does the charge step take at p99? Can we see a trace through the entire system?" Before v0.5.0, you'd need to add logging calls to every step, build a custom Sidekiq middleware, and figure out how to correlate traces across async job boundaries. Now it's one line of config. Enter Middlewares Ruby Reactor 0.5.0 introduces a middleware pipeline — the same pattern that powers Rack, but designed for saga execution. A middleware is a plain Ruby object that hooks into the reactor lifecycle: class TimingMiddleware < RubyReactor :: Middleware def initialize ( ** options ) super @started = {} end def on_start_step ( step_name , _arguments , _context ) @started [ step_name ] = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) end def on_complete_step ( step_name , _result , _context ) started = @started . delete ( step_name ) return unless started elapsed = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) - started logger . info ( "step #{ step_name } took #{ elapsed . round ( 4 ) } s" ) end end This middleware times every step. Register it globally: RubyReactor . configure do | config | config . middlewares = [ TimingMiddleware ] end Now every reactor — every checkout, every refund, every data import — gets step-level timing, for free. The full lifecycle (20+ events) Middlewares can observe the complete execution lifecycle: Phase Events Reactor on_start_reactor , on_complete_reactor , on_failed_reactor Step on_start_step , on_complete_step , on_failed_step , on_retry_attempt Compensation on_start_compensation , on_complete_compensation , on_failed_compensation Undo on_start_undo , on_complete_undo , on_failed_undo Coordination on_lock_acquired , on_lock_failed , on_

Here's a five-line function. It calls an LLM, logs the answer, returns it. async function ask ( question : string ) { const res = await openai . responses . create ({ model : " o4-mini " , input : question }); console . log ( " answer: " , res . output_text ); return res . output_text ; } This compiles. It passes tests. It ships. And it will quietly cost you four figures a month before anyone notices, because nothing in that log tells you the model burned 8,000 hidden reasoning tokens to produce a 40-token reply. That's the gap this article is about. AI calls are not regular HTTP calls. The interesting state isn't the response body - it's the messages you sent, the tools the model picked, the tokens it consumed (visible and otherwise), and the dollars that drained out of the budget. If your observability story is "we log the answer," you're flying a plane with one gauge and that gauge is the altimeter. Let's talk about what to actually capture. The four signals that matter Every AI system has the same four dimensions worth instrumenting, and most teams only track one or two of them: Logs - the request/response pair, the error, the latency. The boring stuff that traditional APM already covers. Prompts - the actual text that went in and the actual text that came out. Including system prompts, tool definitions, and history. Tool calls - which tool the model picked, with what arguments, what came back, in what order, with what retries. Cost - input tokens, output tokens, cached tokens, reasoning tokens, model, and the per-million-token price for each. Multiplied per user, per feature, per request. Lose any one of these and you're working blind on a different axis of the problem. Lose the cost signal and you wake up to a Slack message from finance. Lose the tool-call signal and you can't tell why your agent kept booking the wrong flight. Lose the prompt signal and a prod regression becomes a guessing game. Lose plain logs and you don't even know the call happened. The go

OpenTelemetry has introduced a new "Blueprints" initiative aimed at reducing the growing complexity of deploying and operating observability systems at scale. By Craig Risi

OpenTelemetry has quietly become table stakes. That's a good thing, but if you've instrumented a real codebase, you know the tax. A method that does one obvious thing slowly fills up with StartActivity , SetTag , AddEvent , SetStatus . The bookkeeping of telemetry starts to drown out the intent of the code, and in review you spend half your time mentally separating "what this code does" from "what we report about what it does." It's easy to think "oh, but the framework takes care of this with auto-instrumentation", but if you talk to the experts in OTel, they'll go to great lengths to explain that auto-instrumentation is a floor, not a ceiling. Most of the value in telemetry comes from the custom instrumentation you add to your code that adds business context to your traces. And that custom instrumentation is the stuff that clutters up your code. Here's the kind of thing I mean: // before - ugly, obtuse, who put that there var orderId = order . Id ; Activity . Current ?. SetTag ( "orderId" , orderId ); Sidemark is my answer to that code-obfuscation problem: non-invasive instrumentation via what I'm calling Active Comments . // after - glorious, beautiful, basking in the light of the sun, closer to god, happy, satiated var orderId = order . Id ; //? The idea A small set of comment syntaxes - //? , //! , //?! - become ride-along annotations . They travel next to the code, get read at build time, and turn into the equivalent Activity calls in the compiled output. The code you read stays the code that does the work. The telemetry rides along instead of competing with your logic for attention. The framing is loosely inspired by Wallaby.js's Live Annotations , which project runtime values inline next to the code that produced them. Sidemark takes the same instinct in the other direction: comments as a write surface for instrumentation, rather than a read surface for debug values. Comments are an under-used channel for information about code that isn't itself code - and su

TLDR Monitoring AI agents in production requires distributed tracing: a single user request fans out into 10 or more internal operations, and logs alone cannot show you which step is slow, failing, or burning your token budget. OpenTelemetry's gen_ai.* semantic conventions give you standardized span attributes for LLM calls, tool invocations, and agent steps. Some are stable today; others are still experimental. Auto-instrumentation libraries (OpenLLMetry, OpenInference, OpenLIT) cover most agent frameworks with two to three lines of initialization code. You do not change your agent code. Traces ship to OpenObserve over OTLP. From there you get SQL-queryable trace data, token usage dashboards, cost attribution by agent and model, and alerting on latency and cost anomalies. OpenObserve also exposes an MCP server. You can query your live agent traces from a Claude or GPT session without opening a dashboard. Why Agents Are Harder to Monitor Than a Single LLM Call A single LLM call is straightforward to observe. One HTTP request, one response, one latency number. You can log the input and output and call it done. An agent is different. When a user sends a message, the agent calls an LLM to decide what to do, invokes a tool, processes the result, calls the LLM again, possibly calls another tool, and eventually returns a response. That one user message becomes ten or more internal operations. Some of those operations call external APIs. Some retry. Some spawn sub-agents. Without distributed tracing, you see none of this structure. You know the response took 8 seconds. You do not know whether the LLM took 7 of those seconds or whether a tool made three retries before timing out. Four categories of problems appear in production agents that you cannot debug without traces: Latency. Which step is slow? The LLM call? The tool execution? A retry loop the agent entered because the tool returned ambiguous output? Cost. Which agent, which task, which model is consuming tokens? A s