标签：#telemetry

TL;DR In generative video pipelines, running cheap low-step sketches to pick parameters sounds like free optimization. But when prompts go out-of-distribution, surrogate scorers return noise, turning a \$0.002 check into a bad decision that triggers a \$15 compounding failure. Here's why skipping the cheap step is sometimes the cheapest option. Three numbers run Scenematic's generation loop. A think-frame costs \$0.002. A full render costs \$0.50. A bad scene that slips through and gets built on costs about \$15.50, because the scene chain compounds it before anyone looks. The constant in lib/generation-loop.ts carries the arithmetic in a comment: 15.502, // CALIBRATION_TARGET: 0.002 + 0.50 + 15.00 . Most of the pipeline exists to keep spend at the cheap end of that ladder. One module decides when the cheap step should be skipped entirely. A hundred-contract baseline then put numbers on how often that decision was wrong. 1. The rehearsal lib/think-frames.ts generates quick, low-inference-step sketches before committing to a full-quality keyframe. The file header credits DeepGen's think tokens as the inspiration. Each sketch tries a different preservation focus, character, environment, mood, composition, or atmosphere, with its own image-to-image strength and seed. The reward mixer scores the batch and the winner's parameters go to the full render. The economics only work if those scores mean something. That assumption fails quietly, and it fails hardest on the prompts where a rehearsal looks most useful. 2. Where the scores stop meaning anything Scoring a sketch of A detective leans forward across a metal table, interrogating a nervous suspect under fluorescent lights works fine. The scorer has seen a thousand shots like it. Scoring A sentient equation writes itself across a blackboard that extends infinitely in all dimensions does not fail loudly. It returns a number, and the number is noise. Both prompts are verbatim from the baseline harness. lib/ood-detector.ts

I ran a penetration test on my own infrastructure last week. No Burp Suite, no exploit fired at production, no CVE popped. The whole engagement came down to one habit: refusing to believe a control was working until I had watched it work. The target is a small observability hub I built for my own AI-assisted coding. Six services in one compose file: a tunnel, an OpenTelemetry Collector taking metrics and logs from Claude Code, Prometheus, Grafana, Loki, and a status API. The public surface is three aggregate numbers. Everything else stays private. That boundary, three numbers out and nothing else, was the whole thing I was testing. The word "pentest" carries a picture that does not match, so: no attack traffic at the live system. The platform bills by usage and there is a WAF in front, so a flood of probes would have cost money and poisoned its own results. What I did was a read-only audit of the code and config, plus a dynamic run against the whole stack brought up locally in Docker. I expected the findings to cluster around the parts nobody had looked at. They did the opposite. Nearly every serious defect sat inside a control written days or hours earlier, usually by me, usually with a comment beside it naming what it protected against. Old code has been observed: it has run against real traffic and somebody has been surprised by it. A defence written yesterday has only been reasoned about, which feels like the same thing and is not. "Independent" is a measurement, not a comment The privacy boundary is an allow-list rather than a deny-list, and that part was right. Claude Code was measured sending five identity attributes, user.email among them carrying a real address, and no flag turns them off. A delete_key for each works until the client adds a sixth, and this telemetry is beta: its attribute set is not a contract. - context : resource statements : - keep_keys(resource.attributes, ["service.name"]) - set(resource.attributes["service.name"], "claude-code") The s

Cloudflare launched agent tracing, adding spans for agent invocations, model calls, tool runs, and approvals to existing Workers traces. Sessions replay turn by turn, though the docs warn traces are not lossless and payloads may be truncated. Payload recording defaults differ by framework, and from October 1, 2026 every span counts as a billable event. By Steef-Jan Wiggers

For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps. That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how. The seductive wrong model The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right? So my working theory became: observability is logging plus some fancy logic to analyze the logs. Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other. Descent one: metrics aren't events at all A metric is not a record you write. It's a number sitting in your app's memory . requests_total . increment () // 1, 2, 3... request_duration . record ( 0.23 ) // adds to a histogram Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out. That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time. And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app. Descent two: logs are the familiar part Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched. The only upgrade

A follow-up to How Old Is My Data? The failure mode that gets worse when a machine is reading the data In a classic dashboard, stale data is a human problem: someone looks at a number that's six hours old and makes a slightly worse decision. Annoying, rarely catastrophic. Now hand that same data to a retrieval-augmented-generation (RAG) pipeline, or to an autonomous agent. The stakes change. The system doesn't pause to sanity-check the timestamp — it acts. And when the data it acts on is stale, three things are true at once: The answer is confidently wrong. There is no error to fire on — the query succeeded, the model responded, latency was normal. Every other signal on your dashboard is green. That's the worst combination in observability: a real failure that is completely invisible to the signals we currently emit. Where staleness hides in AI systems RAG: index vs. corpus. Your vector index was built from a corpus at some point in time. The corpus keeps changing — documents get added, edited, retracted. If the re-embedding job stalls or falls behind, the index quietly drifts out of date. The retriever still returns plausible chunks; the model still writes a fluent answer. It's just answering from a version of reality that no longer exists. The quantity you care about is the age of the index relative to its source — not the age of either one alone. Feature stores: online–offline skew. The features your model trained on and the features it serves on are supposed to match. When the online store lags the offline pipeline, predictions degrade in a way that looks like model drift but is actually data staleness wearing a costume. Agents: stale shared state. Multi-agent systems coordinate through shared memory, scratchpads, and context. An agent reasoning over state that another agent updated ten steps ago — but which never propagated — makes locally reasonable, globally wrong decisions. This isn't a new or exotic problem: it's exactly the regime that Age of Information t

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up. The problem OpAMP solves A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans. Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed. OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane. What OpAMP actually is OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel

Modern applications don't just need features—they need observability . When an API becomes slow, a database query takes too long, or an unexpected error occurs, developers need answers quickly. That's where SigNoz and OpenTelemetry come in. For the Agents of SigNoz Hackathon 2026 , I integrated SigNoz into my existing MERN Stack e-commerce application called Ram Store and transformed it into a fully observable application. In this blog, I'll walk through what I built, how I instrumented it, the challenges I faced, and what I learned. About the Project Ram Store is a full-stack e-commerce platform built using the MERN Stack. Features User Authentication Product Management Categories & Subcategories Shopping Cart Address Management Order Management MongoDB Database Tech Stack React (Vite) Node.js Express.js MongoDB Docker OpenTelemetry SigNoz Winston Logger Why Observability? Before integrating SigNoz, I could only rely on: console.log() Manual debugging Browser Network tab When something failed, I had questions like: Which API is slow? Why is the response delayed? Which MongoDB query is taking time? How many requests is my backend serving? Where exactly did the error happen? Without observability, finding answers takes time. I wanted a single place where I could monitor everything. That's why I chose SigNoz . Setting Up SigNoz I self-hosted SigNoz locally using Docker. Once all the containers were running successfully, the dashboard became available. After the initial setup, the next step was instrumenting my backend. Integrating OpenTelemetry I added OpenTelemetry to my Node.js backend. Using the Node SDK together with automatic instrumentation, I configured: Express instrumentation HTTP instrumentation MongoDB instrumentation OTLP gRPC exporter Now every request automatically generates telemetry without modifying every route. Distributed Traces One of my favorite features is Distributed Tracing . Whenever I perform an action in Ram Store—like opening products or ad

Expedia Group has introduced STAR, an internal AI-assisted observability platform that helps engineers investigate production incidents using service telemetry and LLMs. Built with FastAPI, Datadog, Celery, Redis, and Langfuse, STAR follows structured workflows to analyze telemetry, generate root cause assessments, and support incident response while keeping engineers in the loop. By Leela Kumili

Ben O'Mahony discusses building custom AI-powered Language Server Protocols (LSPs) that go beyond standard rule-based checkers. He explains how to instrument AI agents natively with OpenTelemetry to track concrete user actions (accepting, dismissing, or regenerating code fixes) as implicit labels, creating a continuous data flywheel to distill frontier capabilities into cheaper, local SLMs. By Ben O'Mahony

A new technical analysis published by the Cloud Native Computing Foundation (CNCF) argues that the future of agentic AI will be built not on entirely new infrastructure, but on the mature cloud-native ecosystem that already powers modern distributed applications By Craig Risi

GitHub announced enterprise-managed OpenTelemetry export for Copilot activity from VS Code and Copilot CLI on July 8, 2026. Primary source: GitHub Changelog, July 8, 2026 . Export availability is only the start. The receiving collector becomes an enterprise ingress point. This is an unexecuted operating plan; signal types, attributes, endpoint requirements, and controls must be checked against current GitHub documentation. Isolate the path managed clients -> private telemetry ingress -> dedicated OTel Collector pool -> field policy + bounded queue -> dedicated backend dataset Do not point every developer client directly at the primary observability backend. Give the collector write-only destination credentials, separate its dataset from production application telemetry, and define retention before rollout. Isolation is not anonymity. Stable user, device, organization, or repository identifiers may still be sensitive. Start with a field budget Category Initial policy Product and version Keep bounded values Operation and status Keep documented enums Timing and counts Keep numeric measures Raw prompts or generated code Drop by default File paths and repository URLs Drop or transform after review User identity Prefer scoped pseudonymous identity Free-form errors Drop raw text; keep reviewed classes These categories are recommendations, not a description of GitHub's payload. Inspect a restricted canary before naming actual keys. processors : memory_limiter : check_interval : 1s limit_mib : 512 spike_limit_mib : 128 attributes/field_budget : actions : # Illustrative keys only; replace after payload review. - key : user.email action : delete - key : file.path action : delete - key : command.arguments action : delete batch : send_batch_size : 512 timeout : 5s Verify processors against the chosen Collector distribution. A valid startup does not prove that records satisfy policy. Drill three failures Backend outage: block the exporter. Retries must be bounded, queue growth vi

This article is my submission for the Agents of SigNoz Hackathon: Blog Track, where participants...

Where Copilot's telemetry stream lands, decided centrally GitHub added a control on July 8 that lets an enterprise mandate where the Copilot Chat extension in VS Code and Copilot CLI send OpenTelemetry data, removing the need for individual developers to set OTEL_* environment variables. Per the GitHub changelog, the setting is delivered through a telemetry block in the enterprise-managed settings, and a managed value takes precedence over environment variables and user settings. Four things are configurable in the block: the OTLP export endpoint and transport ( otlp-http or otlp-grpc ), the OTel service name and resource attributes, exporter headers such as an authentication token for the collector, and whether prompt, response and tool content is captured, with a separate flag for whether developers can change that. Delivery uses the channels documented on the same page: native MDM (Windows Registry or macOS managed preferences), server-managed settings from a signed-in GitHub account, or a file-based managed-settings.json . Where this bites The precedence rule is the point. If a platform team owns the collector and needs traces routed to it, this is exactly the switch they wanted. If a developer had their own OTLP endpoint pointed at a local sink, they will see the session start emitting somewhere else. The changelog does not describe a per-user override once a managed value is set. A scoping note is worth reading twice. The changelog states that managed exporter headers apply only to the Copilot Chat extension's OTLP exporter. The endpoint and transport policy still reach the CLI agent host, but the auth-token flow the changelog calls out is bound to the Chat surface. On-call teams standing up the collector should plan for that asymmetry before it lands as a surprise during triage.

The Cloud Native Computing Foundation (CNCF) has announced the graduation of OpenTelemetry, elevating the project to the foundation's highest level of maturity and formally recognizing it as production-ready for enterprise use. By Craig Risi

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

OpenTelemetry has become the standard for observing modern systems. But when you start building AI applications, traditional traces aren't enough. You don't just want to know that a request happened. You want to know: Which model generated the output? Which provider was used? How many tokens were consumed? What embedding model processed the documents? How much did the operation cost? These questions become even more important when building Retrieval-Augmented Generation (RAG) systems. Recently while contributing to Mastra, I discovered an observability gap involving RAG embedding operations. This led me to open a pull request that introduced proper OpenTelemetry GenAI semantic mappings for RAG_EMBEDDING spans. The Problem Mastra already exported rich metadata for several AI operations. However, RAG embedding spans were missing standardized GenAI semantic attributes. As a result, observability tools could see that an embedding operation occurred, but they couldn't easily understand: Model information Provider information Token usage Embedding-specific metadata Without standardized semantic conventions, dashboards and tracing systems lose valuable context. This becomes a bigger issue in production environments where teams need visibility into AI workloads. Understanding RAG Embedding Spans A typical RAG pipeline looks like this: Documents ↓ Chunking ↓ Embedding Model ↓ Vector Database ↓ Similarity Search ↓ LLM Generation The embedding stage is critical. Every document chunk gets transformed into a vector representation. If observability data from this stage is incomplete, debugging performance issues becomes significantly harder. Why OpenTelemetry Semantic Conventions Matter OpenTelemetry doesn't just define traces. It also defines semantic conventions. These conventions create a common language for telemetry data. Instead of every framework inventing custom field names, everyone follows the same standard. For GenAI workloads this means tools can automatically underst

You've built a checkout reactor that reserves inventory, charges a card, generates a shipping label, and sends a confirmation email. It runs through Sidekiq. When something fails, compensation logic rolls it back. It works. Then your team asks: "How many checkouts failed this week? Which step? How long does the charge step take at p99? Can we see a trace through the entire system?" Before v0.5.0, you'd need to add logging calls to every step, build a custom Sidekiq middleware, and figure out how to correlate traces across async job boundaries. Now it's one line of config. Enter Middlewares Ruby Reactor 0.5.0 introduces a middleware pipeline — the same pattern that powers Rack, but designed for saga execution. A middleware is a plain Ruby object that hooks into the reactor lifecycle: class TimingMiddleware < RubyReactor :: Middleware def initialize ( ** options ) super @started = {} end def on_start_step ( step_name , _arguments , _context ) @started [ step_name ] = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) end def on_complete_step ( step_name , _result , _context ) started = @started . delete ( step_name ) return unless started elapsed = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) - started logger . info ( "step #{ step_name } took #{ elapsed . round ( 4 ) } s" ) end end This middleware times every step. Register it globally: RubyReactor . configure do | config | config . middlewares = [ TimingMiddleware ] end Now every reactor — every checkout, every refund, every data import — gets step-level timing, for free. The full lifecycle (20+ events) Middlewares can observe the complete execution lifecycle: Phase Events Reactor on_start_reactor , on_complete_reactor , on_failed_reactor Step on_start_step , on_complete_step , on_failed_step , on_retry_attempt Compensation on_start_compensation , on_complete_compensation , on_failed_compensation Undo on_start_undo , on_complete_undo , on_failed_undo Coordination on_lock_acquired , on_lock_failed , on_

Here's a five-line function. It calls an LLM, logs the answer, returns it. async function ask ( question : string ) { const res = await openai . responses . create ({ model : " o4-mini " , input : question }); console . log ( " answer: " , res . output_text ); return res . output_text ; } This compiles. It passes tests. It ships. And it will quietly cost you four figures a month before anyone notices, because nothing in that log tells you the model burned 8,000 hidden reasoning tokens to produce a 40-token reply. That's the gap this article is about. AI calls are not regular HTTP calls. The interesting state isn't the response body - it's the messages you sent, the tools the model picked, the tokens it consumed (visible and otherwise), and the dollars that drained out of the budget. If your observability story is "we log the answer," you're flying a plane with one gauge and that gauge is the altimeter. Let's talk about what to actually capture. The four signals that matter Every AI system has the same four dimensions worth instrumenting, and most teams only track one or two of them: Logs - the request/response pair, the error, the latency. The boring stuff that traditional APM already covers. Prompts - the actual text that went in and the actual text that came out. Including system prompts, tool definitions, and history. Tool calls - which tool the model picked, with what arguments, what came back, in what order, with what retries. Cost - input tokens, output tokens, cached tokens, reasoning tokens, model, and the per-million-token price for each. Multiplied per user, per feature, per request. Lose any one of these and you're working blind on a different axis of the problem. Lose the cost signal and you wake up to a Slack message from finance. Lose the tool-call signal and you can't tell why your agent kept booking the wrong flight. Lose the prompt signal and a prod regression becomes a guessing game. Lose plain logs and you don't even know the call happened. The go

OpenTelemetry has introduced a new "Blueprints" initiative aimed at reducing the growing complexity of deploying and operating observability systems at scale. By Craig Risi

OpenTelemetry has quietly become table stakes. That's a good thing, but if you've instrumented a real codebase, you know the tax. A method that does one obvious thing slowly fills up with StartActivity , SetTag , AddEvent , SetStatus . The bookkeeping of telemetry starts to drown out the intent of the code, and in review you spend half your time mentally separating "what this code does" from "what we report about what it does." It's easy to think "oh, but the framework takes care of this with auto-instrumentation", but if you talk to the experts in OTel, they'll go to great lengths to explain that auto-instrumentation is a floor, not a ceiling. Most of the value in telemetry comes from the custom instrumentation you add to your code that adds business context to your traces. And that custom instrumentation is the stuff that clutters up your code. Here's the kind of thing I mean: // before - ugly, obtuse, who put that there var orderId = order . Id ; Activity . Current ?. SetTag ( "orderId" , orderId ); Sidemark is my answer to that code-obfuscation problem: non-invasive instrumentation via what I'm calling Active Comments . // after - glorious, beautiful, basking in the light of the sun, closer to god, happy, satiated var orderId = order . Id ; //? The idea A small set of comment syntaxes - //? , //! , //?! - become ride-along annotations . They travel next to the code, get read at build time, and turn into the equivalent Activity calls in the compiled output. The code you read stays the code that does the work. The telemetry rides along instead of competing with your logic for attention. The framing is loosely inspired by Wallaby.js's Live Annotations , which project runtime values inline next to the code that produced them. Sidemark takes the same instinct in the other direction: comments as a write surface for instrumentation, rather than a read surface for debug values. Comments are an under-used channel for information about code that isn't itself code - and su