标签：#observability

The laptop was never the interesting part of coding agents. It was just the first convenient runtime. Your laptop has the repository, the shell, the secrets, the package cache, the local database, the half-working dev server, and whatever credentials you forgot were still loaded in the background. So the early version of agentic coding naturally ran there. It was close to the work. It had all the messy context. It was also a very strange place to run something that might edit code for an hour while calling tools, installing dependencies, and touching private systems. AWS published a Bedrock AgentCore post this month with a very good hook: you should be able to close your laptop while the coding agent keeps working. That is the demo-friendly version. The more important version is this: once the agent leaves the laptop, "what happened?" becomes a production question. And that is where observability stops being a nice enterprise add-on and becomes part of the product. remote is not the same as trustworthy Moving a coding agent to a hosted runtime solves some obvious problems. The agent can keep running after your machine sleeps. Multiple agents can run in parallel without fighting over the same local Postgres port. The filesystem can persist between sessions. The environment can be isolated in a microVM or container instead of sharing your real shell with everything else you do all day. Good. But remote execution also removes a useful kind of accidental visibility. When the agent is on your laptop, you can at least see the terminal, notice the fan spin up, watch the test output, and feel the blast radius because it is your machine. In a hosted runtime, that little bit of intuition disappears. The agent is now somewhere else, with its own filesystem, network path, credentials, tools, retry behavior, and bill. So the platform has to replace intuition with evidence. Not a transcript pasted into a PR. Actual operational evidence: traces, logs, metrics, command history, too

It's 2:47am. The pager has just gone off for the third time in twenty minutes. Checkout latency is spiking. The error rate on /api/orders is climbing. Slack is filling with screenshots of half-finished trace views. Somewhere in your logs, the answer is sitting there in plain text, buried under a few million other lines that all look just as urgent. This is the moment people are talking about when they say "AI is going to change how we debug production." Not the demo where someone asks ChatGPT to write a regex. The 2:47am moment. The one where a tired human has to hold five tabs open in their head and form a hypothesis before the executive team starts asking for an ETA. It turns out that's where the technology has the most to offer, and also where it embarrasses itself most often. Let's break down what's actually working in 2026, where the seams still show, and how to wire an LLM into your incident-response loop so it earns its keep instead of just adding another window to glance at. What AI is genuinely good at during an incident The two boring superpowers first: reading fast and correlating across heterogeneous signals . Those are the things humans get worst at when they're tired and time-pressured, and they're the things a good LLM does at the same speed at 2am as at 2pm. Datadog's Bits AI SRE, which the company benchmarked against real incidents from hundreds of internal Datadog teams, is built around exactly this insight: an agent that can fan out across metrics, logs, traces, recent deploys, and incident history simultaneously, then collapse the findings into a single readable narrative. Datadog runs the agent against tens of thousands of evaluation scenarios and claims time-to-resolution wins of up to 95% in its published material. That headline number is marketing (you should always read it as "in the cases where the agent worked, this is what it shaved"), but the underlying capability is real, and it isn't unique to Datadog. Honeycomb's Query Assistant has b

Observability isn't just a buzzword; it's a necessity, especially when diving into distributed systems. If you're using NestJS, you might want to take a look at RedisX. It's a modular toolkit that can boost the observability of your applications. A standout feature? The Metrics Plugin. It meshes well with Prometheus, delivering insights into Redis operations in your NestJS setup. Getting RedisX Metrics Rolling in NestJS So, first things first. To harness the power of RedisX Metrics, you need to set up your NestJS app with RedisX. This means installing some packages and configuring the RedisModule with the MetricsPlugin. Hit your terminal and run: npm install @nestjs-redisx/core @nestjs-redisx/metrics Now, let's tweak your AppModule . You want it to use RedisModule with MetricsPlugin: import { Module } from ' @nestjs/common ' ; import { ConfigModule , ConfigService } from ' @nestjs/config ' ; import { RedisModule } from ' @nestjs-redisx/core ' ; import { MetricsPlugin } from ' @nestjs-redisx/metrics ' ; @ Module ({ imports : [ ConfigModule . forRoot ({ isGlobal : true }), RedisModule . forRootAsync ({ imports : [ ConfigModule ], inject : [ ConfigService ], plugins : [ new MetricsPlugin ({ prefix : ' redisx_ ' , endpoint : ' /metrics ' , defaultLabels : { service : ' my-service ' } }) ], useFactory : ( config : ConfigService ) => ({ clients : { host : config . get ( ' REDIS_HOST ' , ' localhost ' ), port : config . get ( ' REDIS_PORT ' , 6379 ), }, }), }), ], }) export class AppModule {} Prometheus Metrics: What You Get With MetricsPlugin set up, your app now exposes a /metrics endpoint. Prometheus can scrape this endpoint, dishing out detailed metrics about your Redis operations. Here's a snapshot of what you get: redisx_cache_hits_total : Tracks total cache hits. redisx_lock_acquired_total : Total locks acquired. redisx_redis_commands_total : Total Redis commands run. Making Prometheus Work for You To get those insights, set up Prometheus to scrape your /metrics end

My detector passed every synthetic test with zero false positives. Then I pointed it at one real trace and found a crack. This is the honest version of where I am. I'm building Clew — a tool that finds the redundant loops, re-queries, and handoffs that silently burn tokens when multiple AI agents work together. No crash, no error, just two agents quietly re-doing each other's work while the token bill climbs. I build in public, and I publish the negatives. So here's the whole arc, including the part that isn't working yet. First, I killed my own hypothesis The original idea wasn't waste detection at all. It was failure prediction: watch the behavior between agents and forecast multi-agent failures before they happen. The differentiator was a single metric built on two signals — structural cycles in the inter-agent message graph, and the decay of novelty in embeddings. Before I ran anything, I pre-registered the success bar: AUC ≥ 0.80. I numbered every change and kept the signal code physically separated from the labels so I couldn't leak my way to a good number. Then I ran it on MAST-Data — UC Berkeley's dataset of 1,600+ real multi-agent traces across 7 frameworks[( Cemri et al., arXiv:2503.13657 )](url) Result: AUC ≈ 0.455. A coin flip. It got worse. The signal correlated with trace length at r ≈ 0.86 — it was mostly measuring how long a trace was, not whether it failed. Correcting for that dropped AUC to 0.42 and reversed the direction: successful traces actually showed more decay (p ≈ 0.013). The honest read: not disproven, but unvalidated. On this implementation, on this data — negative. So I shut it down. And I counted it as a win, because I got a fast, honest answer in weeks instead of building a dashboard on a metric that secretly measures string length. That experiment became the DNA of everything since: design the experiment that's allowed to kill the idea. The pivot: from predicting failure to cutting waste The intuition behind v1 — that you need structu

Here's a five-line function. It calls an LLM, logs the answer, returns it. async function ask ( question : string ) { const res = await openai . responses . create ({ model : " o4-mini " , input : question }); console . log ( " answer: " , res . output_text ); return res . output_text ; } This compiles. It passes tests. It ships. And it will quietly cost you four figures a month before anyone notices, because nothing in that log tells you the model burned 8,000 hidden reasoning tokens to produce a 40-token reply. That's the gap this article is about. AI calls are not regular HTTP calls. The interesting state isn't the response body - it's the messages you sent, the tools the model picked, the tokens it consumed (visible and otherwise), and the dollars that drained out of the budget. If your observability story is "we log the answer," you're flying a plane with one gauge and that gauge is the altimeter. Let's talk about what to actually capture. The four signals that matter Every AI system has the same four dimensions worth instrumenting, and most teams only track one or two of them: Logs - the request/response pair, the error, the latency. The boring stuff that traditional APM already covers. Prompts - the actual text that went in and the actual text that came out. Including system prompts, tool definitions, and history. Tool calls - which tool the model picked, with what arguments, what came back, in what order, with what retries. Cost - input tokens, output tokens, cached tokens, reasoning tokens, model, and the per-million-token price for each. Multiplied per user, per feature, per request. Lose any one of these and you're working blind on a different axis of the problem. Lose the cost signal and you wake up to a Slack message from finance. Lose the tool-call signal and you can't tell why your agent kept booking the wrong flight. Lose the prompt signal and a prod regression becomes a guessing game. Lose plain logs and you don't even know the call happened. The go

** How I instrument ASR, LLM, TTS, and the client with OpenTelemetry, and which number in each layer I actually look at ** TL;DR. A voice agent is four moving parts stuck together: speech to text, the model that writes the reply, text to speech, and the client that plays the audio back. End to end latency hides which of those four is slow on any given turn, so I stopped tracking it as one number and started tracing each stage as its own OTel span with a shared session id. The number I watch hardest is barge-in: when the user starts talking over the agent, how many milliseconds until the agent actually stops sending audio. In our setup we want that under 200ms, and when p95 barge-in creeps past that, the agent feels like it is talking at you instead of with you. Everything below is how I wire the spans, what attributes go on each one, and the p95 I page on per layer. The thing I keep saying, and the thing that keeps being true: voice agents fail in production not because of raw latency but because nobody simulated the audio and LLM pipeline together. You can have a fast ASR, a fast model, a fast TTS, and a voice agent that still feels broken, because the failure lives in the seams between them and in the parts (barge-in, jitter) that no single-stage benchmark touches. Tracing is how I get the seams to show up. A note before the layers. This is just the setup we run, the spans we emit, and the mistakes that made us add each attribute. Some of it is probably specific to our stack and will not transfer. I will flag that where I can. The shape of a turn, and why one span is not enough One turn is: user says a thing, agent says a thing back. Underneath that is roughly: audio frames come in, ASR turns them into text (streaming partials as it goes); the text plus history goes to the LLM, which streams tokens back; as text comes out, TTS turns it into audio, also streaming; the client receives audio frames and plays them, with some buffering to smooth out jitter. If you wrap

Netflix has shared details about Service Topology. This internal system creates and updates a live dependency graph for thousands of microservices. It helps engineers see how services connect and resolve issues more quickly. The system merges three separate data sources into a single, queryable graph. It updates almost in real-time as traffic patterns shift. By Claudio Masolo

Harness Base Definition: The Control System Outside the Model Previously, we split Agent into several minimal parts: Model: judge the next step Loop: keep the process moving Tools: interact with the real world State: keep the task connected At this point, a natural question appears: If Agent already has model, loop, tools, and state, why talk about Harness? An even easier confusion is: Is Harness a higher-level, smarter Agent that manages other Agents? That sounds plausible, but it bends the architecture in the wrong direction. Harness is not another Agent. It is not a larger prompt, and it is not a framework name. It is the control system outside the model. Continue with the same small CLI Agent: User says: help me figure out why this project's tests are failing, and fix it. If this CLI Agent is only a demo, it can be simple: send user input to model model says read file program reads file put result back into prompt model says edit file program edits file model says run tests program runs tests This chain can work once and already look like an Agent. But as soon as someone else really uses it, questions appear. What if the model wants to execute rm -rf ? What if it wants to read private files under the user's home directory? If it runs for ten minutes and the user interrupts, how is the working state saved? After a tool error, should the next model turn see the full log or only a summary? If the same task continues tomorrow, where does the session resume from? If a modification looks successful but no test verified it, how does the system know it is done? If a user says the Agent damaged a file, how do we reconstruct what happened? These questions do not belong to the model itself. They should not be left for the model to decide. The model only generates the next-step judgment from the current context. Permission, execution environment, session lifecycle, observability logs, verification criteria, and governance policy are engineering responsibilities outside the

OpenTelemetry has introduced a new "Blueprints" initiative aimed at reducing the growing complexity of deploying and operating observability systems at scale. By Craig Risi

How RAGScope Knows Which Chunks Your LLM Actually Used Your retriever fetched 10 chunks. Your LLM only used 3. RAGScope shows a precision score of 30 out of 100. The question every new user asks: how does it know? There is no OpenTelemetry attribute that says "this chunk was in the context window." RAGScope infers it — and the way it does this is the most consequential piece of engineering in the whole tool. There Is No "In Context" Attribute in OTel The OpenTelemetry semantic conventions for generative AI ( gen_ai.* ) define attributes for model, input/output tokens, and retrieved documents. They do not define anything like gen_ai.chunk.reached_llm or gen_ai.retrieval.used_document_ids . When your RETRIEVER span fires, you get a list of documents. When your LLM span fires, you get a prompt and a completion. The two spans are connected by a parent-child trace relationship — but there is no attribute that maps which retrieved documents appear in which prompt. This gap matters. A reranker might drop 7 of your 10 chunks. Your application code might apply a token budget and truncate 4 more. From the trace alone, you cannot tell. RAGScope needs this information to compute the precision sub-score — the highest-weighted metric at 40% of the overall score. Getting it wrong would make precision meaningless. The Substring Match — How assembleContext Works RAGScope's answer is in src/enrichment/pipeline.ts , in a function called assembleContext : function assembleContext ( chunks : RagChunk [], llmSpans : ParsedSpan []): RagChunk [] { const llmPrompts = llmSpans . map (( s ) => s . prompt ). filter (( p ): p is string => !! p ); if ( llmPrompts . length === 0 ) return chunks ; let position = 0 ; return chunks . map (( chunk ) => { if ( ! chunk . content ) return chunk ; const inContext = llmPrompts . some (( p ) => p . includes ( chunk . content ! )); if ( inContext ) { return { ... chunk , inContext : true , contextPosition : position ++ }; } return { ... chunk , inContext :

TL;DR We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Sumo Logic's JP (Tokyo) region deployment. For Japanese enterprises with data residency requirements under APPI (Act on the Protection of Personal Information), this means audit logs never leave Japan. FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Sumo Logic HTTP Source (JP) │ ▼ ┌───────────────────┐ │ Sumo Logic JP │ │ (Tokyo) │ │ │ │ • 500 MB/day FREE │ │ • Data stays in │ │ Japan │ │ • 7-day retention │ │ (free tier) │ └───────────────────┘ Key advantages: 500 MB/day free tier (~15 GB/month) — covers most FSx for ONTAP deployments at zero vendor cost JP region deployment — data residency in Tokyo Simplest auth model — URL-embedded token, no header management 30-minute end-to-end — HTTP Source URL is the only credential needed Verified on Sumo Logic JP region. Logs searchable via _sourceCategory=aws/fsxn/audit . This is Part 12 of the Serverless Observability for FSx for ONTAP series. Why Sumo Logic for Japanese Enterprises? For organizations operating under Japanese data protection regulations, the choice of observability platform often comes down to one question: where does the data physically reside? Requirement Sumo Logic JP Other Options Data residency in Japan ✅ Tokyo deployment Varies by vendor APPI compliance consideration ✅ Data stays in JP May require cross-border assessment Free tier for validation ✅ 500 MB/day Most offer 14-day trials only No agent installation ✅ HTTP Source (agentless) Some require collectors Sumo Logic's JP deployment ( service.jp.sumologic.com ) processes and stores all data within Japan, making it a straightforward choice for organizations that need to demonstrate data residency compliance. Compliance note : This integration provides a technical path for data residency. Evaluate your specific regulatory requirements with your compliance team — data residency alone does not constitute full regulatory compliance. Architecture ┌────

TL;DR We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Dynatrace via the Log Ingest API v2. The real value: Dynatrace's Davis AI can automatically correlate file access anomalies with application performance degradation — answering "why is the app slow?" with "because 500 users hit the same NFS share simultaneously." FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Dynatrace Log Ingest API v2 │ ▼ Davis AI ┌───────────────────┐ │ Correlates: │ │ • File access │ │ anomalies │ │ • APM metrics │ │ • Infrastructure │ │ health │ │ │ │ → Root cause │ │ in seconds │ └───────────────────┘ Verified on Dynatrace SaaS Trial (Tokyo-equivalent region). Logs visible in Logs Viewer within 1-2 minutes. This is Part 11 of the Serverless Observability for FSx for ONTAP series. Why Dynatrace for FSx for ONTAP? Most observability tools treat storage logs as isolated data. Dynatrace is different — it builds a topology map of your entire stack and uses Davis AI to find causal relationships through time-window correlation and entity connectivity: Scenario Without Dynatrace With Dynatrace App latency spike "Check the logs" Davis AI detects temporal correlation: file access to /vol/data/ increased 10x within the same 5-minute window as app response time degradation, connected via topology (app → NFS mount → SVM) Storage I/O anomaly Manual investigation Automatic correlation via shared topology entities — Davis identifies which services are affected based on entity relationships User reports slow file access Grep through audit logs DQL query + topology view showing the full dependency path from user request to storage operation The key differentiator: Davis AI correlates events across entities that share topology connections within overlapping time windows — not just keyword matching or manual dashboard correlation. Architecture ┌─────────────────────────────────────────────────────────┐ │ Event Sources │ ├─────────────────────────────────────────

TL;DR We built a serverless pipeline that ships FSx for ONTAP audit logs to Honeycomb, where its high-cardinality query engine turns file access data into actionable insights. Two delivery paths verified: [Path A: Direct] FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Honeycomb Events Batch API [Path B: OTel Collector] FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → OTel Collector → OTLP → Honeycomb Why Honeycomb for file access logs? Because file access data is inherently high-cardinality : thousands of users × millions of file paths × dozens of operations × multiple SVMs. Traditional log tools force you to pre-aggregate or sample. Honeycomb lets you query the raw events at full resolution. ┌──────────────────────────────────────────────────────┐ │ Honeycomb Query Engine │ │ │ │ "Show me which users accessed /vol/finance/* │ │ between 2am-4am last Tuesday" │ │ │ │ → BubbleUp: auto-detect anomalous dimensions │ │ → Heatmap: visualize access density over time │ │ → GROUP BY user, path, operation — no pre-indexing │ │ │ │ 20M events/month FREE │ └──────────────────────────────────────────────────────┘ This is Part 10 of the Serverless Observability for FSx for ONTAP series. Why Honeycomb for File Access Logs? Most observability tools index a fixed set of fields. When you have high-cardinality dimensions — like file paths ( /vol/data/project-alpha/2026/Q1/report-final-v3.docx ) or Active Directory usernames — you hit index bloat, slow queries, or forced sampling. Honeycomb's columnar storage handles this natively: Capability Traditional Logs Honeycomb Query by arbitrary field Pre-index or full scan Instant (columnar) GROUP BY high-cardinality field Expensive / limited Native BubbleUp (anomaly detection) Manual investigation Semi-automatic (select time range, BubbleUp identifies differing dimensions) Heatmap visualization Requires pre-aggregation Raw events For FSx for ONTAP audit logs, this means you can ask questions like: "Which

OpenTelemetry has quietly become table stakes. That's a good thing, but if you've instrumented a real codebase, you know the tax. A method that does one obvious thing slowly fills up with StartActivity , SetTag , AddEvent , SetStatus . The bookkeeping of telemetry starts to drown out the intent of the code, and in review you spend half your time mentally separating "what this code does" from "what we report about what it does." It's easy to think "oh, but the framework takes care of this with auto-instrumentation", but if you talk to the experts in OTel, they'll go to great lengths to explain that auto-instrumentation is a floor, not a ceiling. Most of the value in telemetry comes from the custom instrumentation you add to your code that adds business context to your traces. And that custom instrumentation is the stuff that clutters up your code. Here's the kind of thing I mean: // before - ugly, obtuse, who put that there var orderId = order . Id ; Activity . Current ?. SetTag ( "orderId" , orderId ); Sidemark is my answer to that code-obfuscation problem: non-invasive instrumentation via what I'm calling Active Comments . // after - glorious, beautiful, basking in the light of the sun, closer to god, happy, satiated var orderId = order . Id ; //? The idea A small set of comment syntaxes - //? , //! , //?! - become ride-along annotations . They travel next to the code, get read at build time, and turn into the equivalent Activity calls in the compiled output. The code you read stays the code that does the work. The telemetry rides along instead of competing with your logic for attention. The framing is loosely inspired by Wallaby.js's Live Annotations , which project runtime values inline next to the code that produced them. Sidemark takes the same instinct in the other direction: comments as a write surface for instrumentation, rather than a read surface for debug values. Comments are an under-used channel for information about code that isn't itself code - and su

TLDR Monitoring AI agents in production requires distributed tracing: a single user request fans out into 10 or more internal operations, and logs alone cannot show you which step is slow, failing, or burning your token budget. OpenTelemetry's gen_ai.* semantic conventions give you standardized span attributes for LLM calls, tool invocations, and agent steps. Some are stable today; others are still experimental. Auto-instrumentation libraries (OpenLLMetry, OpenInference, OpenLIT) cover most agent frameworks with two to three lines of initialization code. You do not change your agent code. Traces ship to OpenObserve over OTLP. From there you get SQL-queryable trace data, token usage dashboards, cost attribution by agent and model, and alerting on latency and cost anomalies. OpenObserve also exposes an MCP server. You can query your live agent traces from a Claude or GPT session without opening a dashboard. Why Agents Are Harder to Monitor Than a Single LLM Call A single LLM call is straightforward to observe. One HTTP request, one response, one latency number. You can log the input and output and call it done. An agent is different. When a user sends a message, the agent calls an LLM to decide what to do, invokes a tool, processes the result, calls the LLM again, possibly calls another tool, and eventually returns a response. That one user message becomes ten or more internal operations. Some of those operations call external APIs. Some retry. Some spawn sub-agents. Without distributed tracing, you see none of this structure. You know the response took 8 seconds. You do not know whether the LLM took 7 of those seconds or whether a tool made three retries before timing out. Four categories of problems appear in production agents that you cannot debug without traces: Latency. Which step is slow? The LLM call? The tool execution? A retry loop the agent entered because the tool returned ambiguous output? Cost. Which agent, which task, which model is consuming tokens? A s