标签：#Observability

Artificial intelligence is rapidly changing how engineering teams respond to production incidents, offering the ability to summarize incident channels, analyze unfamiliar code, suggest remediation steps, generate pull requests, and increasingly assist with diagnosis. By Craig Risi

Point a model at your repositories, ask why something broke, and you get an answer. Coherent, names a mechanism, shows you how it got there. The working is the problem rather than the reassurance. It looks the same whether it ends at a value in a record or at what usually happens, and only one of those is evidence. I ran the same question at four levels of evidence, on two systems, across three models. The question was never find me a bug. It was always this already happened, work out how. The pass that worries me is the one just short of enough evidence. There is enough there to build a real hypothesis, and not enough to tell a real one from a plausible one. Two of its guesses sent me to look at things I had not checked. One was a dead end. The other was a real problem I had not known about, and nothing in either answer told me which was which. A good guess costs the same to chase as a real one. You find out which it was at the end. The four rungs, and what each one removes I stopped thinking about this as adding context. Each rung takes away something the model would otherwise have to guess at, and that is the more useful way to look at it. One. Repository access, broad. Point it at everything and ask. It removes nothing. The answer was structurally sensible, named components that were not involved, and arrived with no more hesitation than the correct one did three rungs later. If your team tried this once, got something confident and wrong, and decided the tooling is not there yet, this is probably where you stopped. Two. Three bounded repositories, plus a written map. Which service talks to which, over what protocol, with what delivery and ordering guarantees. This removes rediscovery. What made it work was not the map. It was telling the model to treat the map as true and not go and check. That bought focus and gave up verification. Maps go stale quietly, and I have removed the step where it might have noticed. Three. Plus traces and logs covering real executio

The owner had already asked for the alert emails to stop. A fix shipped. Then another email landed. Then another. "ong it just ssent me abother email," he said, voice-dictated, unedited. Fifteen minutes later: "go another one." The system was reporting an outage that did not exist. The Transport That Only Ever Failed A 14-PR merge train had just moved every cron producer's alerting off shared email and onto Buzz, a Nostr-relay team chat. One producer per PR, each with its own liveness contract and a bead receipt. It shipped cleanly. But the library backing those producers carried a default that had only one job: fail. AF_BUZZ_CMD = " ${ AF_BUZZ_CMD :- af_default_buzz_post } " af_default_buzz_post returned 1 with "no Buzz transport injected". Every caller that sourced the library (which is every cron producer) exhausted its Buzz retries and fell through to the email floor. The system reported a false Buzz outage while the relay was healthy. It did this 2 to 5 times per hour. Evidence arrived in the logs: 581 dedup markers, a steady stream of "[INTENT ALERT FLOOR: Buzz unreachable]" emails, and sweep.log showing buzz=ok only for the handful of callers invoked through the CLI entrypoint rather than by sourcing the library. That asymmetry was the bug. The CLI had a one-line fixup swapping in the real transport, annotated in a comment as "the library path is unchanged". The library path did not, and the cron producers all take the library path. The fix promoted the real transport to the default for both seams. af_buzz_transport already discovers the installed buzz-notify.sh and already fails closed when it is genuinely missing. The dead CLI fixup was deleted. Fail-closed behavior survives, but now it is conditional on genuine absence rather than on every caller remembering to opt in. Why not migrate callers one at a time? Because the per-caller route leaves the next new producer to rediscover this the same way. Flipping the default fixes the class, not the instance. The

TL;DR If you're a five-person startup shipping a Node.js API and you want a metrics dashboard by Friday, send your telemetry to a managed backend and keep only the instrumentation layer inside your own repo. The alternative — standing up a time-series database, an object store for long-term blocks, and a dashboard service — puts three more components on an on-call rotation that hasn't earned its first SLO yet. Settle the wire format now and treat the backend as a config line you can change later. I own the platform team's roadmap, which in practice means I'm the person who defends the monitoring bill in a budget review and also the person who gets paged when a disk fills at 03:00. Those two jobs pull in opposite directions, and most of the advice online is written by people who only hold one of them. Usually the pager wins the argument. Should a startup run its own metrics stack, or pay for a managed dashboard? Start with capacity, because that's the step everyone skips before signing anything. A moderately instrumented Node.js API — say 40 HTTP routes, two queue workers, default runtime and event-loop metrics, one latency histogram with ten buckets — sits somewhere around 3,000 to 8,000 active series per process. Multiply by replicas. Multiply again by every environment you keep alive, including the staging cluster nobody admits to. You are at 50k active series before a single engineer has written a custom counter, and a self-hosted scraper will chew through that on a 2 GB VM without noticing. It will still be fine at 500k. Past a few million active series you're into sharding, remote storage, and a retention argument with whoever pays for object storage — that's the point where the self-hosted route stops being free and turns into a project with a headcount attached. None of that work is hard. It's just never zero. Dimension Self-hosted stack Managed metrics backend Time to first dashboard 1–3 days under an hour Who owns retention you, plus the storage bill vendor

Wide Events is a Rails gem that puts the production context a coding agent needs onto one OpenTelemetry root span per request or job. In one production search request, the root event showed 30.0 seconds total duration, 446 ms of Postgres time, and 29.4 seconds of outbound HTTP time. That was enough to focus the investigation on an external dependency. The trace then identified a POST that took 28.9 seconds. The trace contained 82 spans and 20,261 bytes of attribute JSON. The root event contained 40 attributes and 1,420 bytes. This is not a token benchmark, but it shows why the root event is a more compact starting point for an agent. Agents can read the code, but not the running system A coding agent starts with an unusual advantage: it can search every model, controller, job, migration, and test in a few seconds. It also starts with a serious blind spot. The repository cannot tell it: which account experienced the problem which build was running which feature-flag variant was active how many queries the request issued whether a semantic-search leg degraded how much an LLM call cost whether the same symptom appears in one tenant or every tenant Those answers often exist somewhere, but “somewhere” might mean a trace waterfall, application logs, a feature-flag service, product analytics, and a database console. Pulling all of that into a context window is expensive and usually requires several joins that were never designed in advance. A wide event changes the starting point. The app accumulates the context it learns while processing one unit of work, then attaches the completed flat map to the OpenTelemetry root span. The span is marked main=true , so every request or job can be queried as one row. request or job -> Rails and domain context accumulate -> child spans contribute dependency counts and timings -> one flat map is flushed onto the root span -> ClickHouse stores one queryable row The trace still exists. Wide Events gives it an application-shaped index. If y

Use a durable idempotency receipt when feature flag retries can reach a rollout toggle endpoint, otherwise reach for a read-only flag evaluation that cannot create duplicate writes. Short answer: the backend must bind one caller-generated key to one operation and commit the receipt beside the state change; a retry should recover that recorded result, not perform the write again. The flag is not the transaction. Record the invariant at the write boundary My architecture decision is to enforce idempotency inside the backend that owns the mutable state. The caller creates an operation key before its first attempt, sends the same key and operation on every retry, and never manufactures a fresh key inside the retry loop. The backend binds that key to a stable digest of the requested change. If the key and digest have already been committed, it returns the stored result. If the key exists with a different digest, it rejects the integration error as a conflict. The state mutation and receipt belong in one transaction, because two separate commits create an interval in which the state says “done” while the receipt still says nothing. I write the invariant this way: one idempotency key identifies one logical operation within a documented scope; one committed operation has one durable result. The defensible claim is effectively-once mutation within that scope, not exactly-once delivery. Clients, queues, proxies, and deployment controllers can all repeat an attempt, so delivery count isn't a useful correctness boundary. There are three failure boundaries I test. A response can disappear after commit, two workers can race on the same key, and the flag decision can change between attempts. The first requires replaying the stored result. The second requires a uniqueness constraint rather than a check-then-insert sequence. The third requires persisting the evaluated decision with the operation; reevaluating a flag during recovery can turn one logical request into two different his

Watching an autonomous agent run through a loop of tasks is like watching a black box try to solve a puzzle in another room. You can see the final result, but the middle part—the reasoning, the failures, and that pivotal moment where it realizes its plan was garbage—is buried in thousands of lines of unstructured logs. If you've ever deployed an agentic workflow only to check back an hour later and find it has been stuck in a high-latency loop of 'I made a mistake... let me try again' for forty minutes, you know the pain. You didn't have failure; you had expensive, silent repetition. The problem with current LLM observability is that we focus too much on the input and output (the traces) and not enough on the internal state transitions of the agent itself. We need to quantify how often an agent is actually self-correcting versus just spinning its wheels. I recently started working with a specific tool designed for this exact visibility gap: the Agent Self-Reflection & Sentiment Scanner . The Observability Gap in Agentic Loops When we talk about 'agents,' we're usually talking about a loop: Observe, Think, Act, Repeat. In a perfect world, the 'Think' step includes self-correction. If an action fails (e.g., a 403 error from an API), the agent should reflect on that failure and adjust its next move. But how do you measure if your agent is actually getting better during a session? How do you distinguish between an agent that is 'Proceeding' with confidence and one that is in a state of constant 'Correction'? You can't just look at the final success/fail status. You need to parse the execution logs for deterministic markers. Why Deterministic Matching Wins Over LLM-Based Analysis The temptation here would be to pipe your agent logs into another, even larger LLM and ask, 'Is this agent struggling?' Don't do that. It’s redundant, it’s slow, and if you're running high-volume loops, the cost will kill your margin. You've already paid for the primary reasoning engine; don't p

The Observability Black Box As autonomous AI agents evolve from isolated chat assistants into multi-agent systems executing multi-step business logic across databases, APIs, and microservices, enterprise platform teams face an acute operational challenge: black-box opacity. When an autonomous agent fails, hallucinates, or executes an out-of-bounds API call, traditional Application Performance Monitoring (APM) tools fall short. Standard HTTP request logging and basic prompt-response captures cannot reconstruct the non-deterministic reasoning loops, tool selection branches, or sub-agent delegations that led to an incident. Furthermore, enterprise auditors, security teams, and regulatory bodies (governed by SOC 2, FedRAMP, and the EU AI Act) now require non-repudiable proof of agent execution. Organizations must be able to answer five fundamental questions for every production run: Which human or non-human identity authorized the agent run? What planner reasoning path or tool routing logic was chosen? Which exact data assets or vector embeddings were retrieved into context? What was the precise execution latency, token cost, and error tax of each intermediate step? Can the complete execution graph be cryptographically reconstructed for compliance review? To resolve this challenge, platform engineering teams must deploy Audit, Observability & Lineage —an architecture anchored in OpenTelemetry (OTel), OWASP Agent Observability Standards, and immutable lineage graphs. Deep-Dive Architecture: OpenTelemetry & Lineage Integration A production-grade Agent Observability stack avoids proprietary vendor lock-in by standardizing on OpenTelemetry (OTel) OTLP trace ingestion and open metadata stores. 1. The Unified OpenTelemetry Span Tree Every agent execution unit — from user intent trigger to final task completion — is encapsulated within a single root trace context ( agent.run ). Sub-tasks, tool calls, and model invocations are recorded as hierarchical child spans: [ Root Trace:

Autonomous agents don't just fail loudly—they fail expensively. A single misconfigured retry loop between an agent and an LLM can generate thousands of redundant tool calls and API requests before anyone notices, turning a minor logic bug into a five-figure cloud bill. PolicyAware is built to be the operational safety net that catches this class of failure before it reaches your finance team's dashboard. 1. The Recursive Agent Crisis Every SRE and platform engineer who has run agentic workloads in production has a version of this story. An agent is wired to call an LLM, interpret the response, and take an action—often invoking another tool, which produces output that gets fed straight back into the same LLM. Under normal conditions this loop terminates in a few steps. Under a bad prompt, a malformed tool response, or a subtle logic error, it doesn't. The agent gets stuck reasoning in circles: it calls a tool, receives an ambiguous or malformed result, decides the task is incomplete, and calls the LLM again to "retry." Each retry consumes tokens, each tool call hits a downstream API, and there is no natural circuit breaker unless one has been explicitly engineered. Within minutes, a single stuck session can produce: Thousands of duplicate or contradictory API calls to internal and third-party services. Sustained LLM token consumption that dwarfs normal daily usage. Cascading load on downstream systems that were never designed for machine-speed request volume. By the time monitoring dashboards catch the anomaly—if they catch it at all—the damage is already done: a runaway bill, a rate-limited API partner, or a compromised production database from thousands of unchecked write attempts. Traditional APM tools tell you a service is under load; they don't tell you an autonomous agent is the one generating that load, or why. This is why the recursive agent crisis is fundamentally a governance problem, not just a monitoring problem. Rate limits and cost alerts fire after the

A follow-up to How Old Is My Data? The failure mode that gets worse when a machine is reading the data In a classic dashboard, stale data is a human problem: someone looks at a number that's six hours old and makes a slightly worse decision. Annoying, rarely catastrophic. Now hand that same data to a retrieval-augmented-generation (RAG) pipeline, or to an autonomous agent. The stakes change. The system doesn't pause to sanity-check the timestamp — it acts. And when the data it acts on is stale, three things are true at once: The answer is confidently wrong. There is no error to fire on — the query succeeded, the model responded, latency was normal. Every other signal on your dashboard is green. That's the worst combination in observability: a real failure that is completely invisible to the signals we currently emit. Where staleness hides in AI systems RAG: index vs. corpus. Your vector index was built from a corpus at some point in time. The corpus keeps changing — documents get added, edited, retracted. If the re-embedding job stalls or falls behind, the index quietly drifts out of date. The retriever still returns plausible chunks; the model still writes a fluent answer. It's just answering from a version of reality that no longer exists. The quantity you care about is the age of the index relative to its source — not the age of either one alone. Feature stores: online–offline skew. The features your model trained on and the features it serves on are supposed to match. When the online store lags the offline pipeline, predictions degrade in a way that looks like model drift but is actually data staleness wearing a costume. Agents: stale shared state. Multi-agent systems coordinate through shared memory, scratchpads, and context. An agent reasoning over state that another agent updated ten steps ago — but which never propagated — makes locally reasonable, globally wrong decisions. This isn't a new or exotic problem: it's exactly the regime that Age of Information t

A useful agent trace is not a list of timestamps. It is a causal tree. When a TypeScript agent retrieves documents in parallel, calls a model, retries a tool, and falls back to cached data, each operation needs a trace ID, its own span ID, and the correct parent span. Without those relationships, completion order is easily mistaken for execution structure. This article builds a small Node.js tracer to demonstrate the core mechanics: immutable async context, parent-child spans, reliable finalization, and a pluggable sink. It is intentionally smaller than a production observability library, but the design avoids several common mistakes found in minimal examples. Completion Order Is Not Causality Imagine three tools running in parallel: 80 ms search_tickets completes 100 ms load_account completes 120 ms search_docs completes Those timestamps describe completion order. The execution tree describes why the operations existed: research_agent └─ parallel_retrieval ├─ search_docs ├─ search_tickets └─ load_account Both views are useful, but only the tree preserves the relationship between the agent decision and its child tools. The normal JavaScript call stack cannot serve as that tree. Async work may resume later, execute concurrently, or outlive the function that scheduled it. Tracing therefore needs an explicit logical context. The Context We Need Each asynchronous branch needs two values: type TraceContext = { traceId : string ; parentSpanId : string | null ; }; When a new span starts, it reads the current context, records parentSpanId , creates its own spanId , and runs child work inside a new context whose parent is that span. In Node.js, AsyncLocalStorage provides the propagation primitive. It carries a value through normal asynchronous resources without adding trace parameters to every application function. Do not mutate one shared context object. Parallel siblings would race to replace the current span. Create a new context value for every nested span instead. Defin

Ten agent sessions ("minds," in this codebase) run continuously on one box, each with its own responsibility — one writes code, one talks to me on Telegram, one watches sensors, one just measures the fleet itself. They coordinate the way a lot of multi-agent systems eventually do: a shared log file, one line per event, [task] / [taking] / [done] . That log is fine for "what happened." It is useless for "what do we owe, and how much did it cost" — the two questions I actually needed answered before I was willing to let the fleet run unattended overnight. The board is not a ledger, but it can feed one The fix wasn't a new coordination protocol. It was noticing that every line on that board is already a transaction if you're willing to look at it that way: board event ledger meaning [task] fix-the-thing a liability opens [taking] pub: fix-the-thing the liability moves to a specific debtor [done] pub: fix-the-thing the liability settles a provider round-trip (one agent turn) a unit of labour is spent So the board gets replayed into three separate double-entry hledger journals, each tracking a different commodity: money — imputed USD (token counts priced through one rate table). promises — commodity PROMISE : an open [task] with no matching [done] is a standing liability, not a line that scrolled off screen. labour — commodity TURN : one provider round-trip, the fungible unit every mind actually spends, regardless of whether it's writing code or answering a sensor. Each journal gets checked two independent ways — hledger check for internal parity, plus a second, independently-written replay of the same board that has to agree with the balance query. A booking bug fails loud, not silently, because two things that should compute the same number just disagreed. Querying "who owes what" stops being a grep and starts being a query: $ mesh-promises --balance standing open obligations (bal liabilities:promises · 1 PROMISE = open, netted): 1 PROMISE liabilities:promises:pub:chat

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up. The problem OpAMP solves A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans. Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed. OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane. What OpAMP actually is OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel

AI assistance disclosure: This article was drafted with the help of Claude. All technical content, design decisions, code references, and screenshots reflect production systems I designed and operate at airCloset; the prose was revised by me prior to publication. Hi, I'm Ryan , CTO at airCloset (a fashion-rental subscription service based in Japan). "Everything changes with AI" is the prevailing mood. My experience building and then running an internal AI platform (cortex) points the other way. The principles don't change at all. Only the machinery does. This post is about what I've come to treat as principle, what I've concluded should be broken, and the thinking behind that split. Disclaimer : "cortex" in this article is the internal codename for the AI platform built in-house at airCloset. It is unrelated to existing commercial services like Snowflake Cortex or Palo Alto Networks Cortex. I've written about the individual pieces before: code-graph , product-graph , db-graph , biz-graph , AI-Observability , the auto-review harness , and Self-Healing . This post isn't about any of them. It's about the design principle sitting behind all of them, one abstraction level up, more essay than build log. The principle, in one sentence: how do we make accurate information accessible? It's an old question. Libraries, legal case books, encyclopedias, search engines — every era has had its own answer using whatever tools that era gave it. Even the technology revolutions people call "paradigm shifts" mostly just changed the means . The underlying question didn't move. Now AI has arrived, and my read (probably not a controversial one) is that its shift is at least on the scale of the internet, possibly larger. As with every previous paradigm shift, the means of answering "how do we make accurate information accessible?" will get redesigned from the ground up. That's what this post is about: AI-Native Redesign — a view where you rebuild the whole design with AI treated as a given

I've built dozens of dashboards. Most have been ignored. A few have been used constantly. The difference isn't the graphs. It's the design. The 3-second test A useful dashboard answers 'is everything OK?' in 3 seconds. Not 'let me scroll through 40 graphs to find out.' Big colored header at the top: green = healthy, yellow = watching, red = broken. That's the 3-second answer. Everything else is drill-down. The hierarchy rule Three layers, no more: Overview — one line per service, status color, key SLI Service detail — one dashboard per service, 6-12 graphs max Deep dive — triggered from service detail, domain-specific Anything beyond 3 layers is 'please get lost in my dashboard tree.' The on-call test Imagine you're on-call at 3 AM. You get paged for 'service X is slow.' Can you, in 30 seconds, use this dashboard to tell if the problem is the service itself, its database, its upstream dependency, or its downstream consumers? If yes, the dashboard works. If no, redesign. What to cut Graphs with no baseline (flat line or spiky forever — how do you know if it's bad?) Metrics you've never used in an actual incident Vanity metrics (total requests ever) Graphs where the y-axis is in units nobody understands The hidden metric The real measure of a dashboard's value: does the on-call engineer open it before or after the paging tool? If they open it first — it's their compass. If they open it only after being paged — it's a reference, not a dashboard. Aim for the first. Written by Dr. Samson Tanimawo BSc · MSc · MBA · PhD Founder & CEO, Nova AI Ops. https://novaaiops.com

Modern applications don't just need features—they need observability . When an API becomes slow, a database query takes too long, or an unexpected error occurs, developers need answers quickly. That's where SigNoz and OpenTelemetry come in. For the Agents of SigNoz Hackathon 2026 , I integrated SigNoz into my existing MERN Stack e-commerce application called Ram Store and transformed it into a fully observable application. In this blog, I'll walk through what I built, how I instrumented it, the challenges I faced, and what I learned. About the Project Ram Store is a full-stack e-commerce platform built using the MERN Stack. Features User Authentication Product Management Categories & Subcategories Shopping Cart Address Management Order Management MongoDB Database Tech Stack React (Vite) Node.js Express.js MongoDB Docker OpenTelemetry SigNoz Winston Logger Why Observability? Before integrating SigNoz, I could only rely on: console.log() Manual debugging Browser Network tab When something failed, I had questions like: Which API is slow? Why is the response delayed? Which MongoDB query is taking time? How many requests is my backend serving? Where exactly did the error happen? Without observability, finding answers takes time. I wanted a single place where I could monitor everything. That's why I chose SigNoz . Setting Up SigNoz I self-hosted SigNoz locally using Docker. Once all the containers were running successfully, the dashboard became available. After the initial setup, the next step was instrumenting my backend. Integrating OpenTelemetry I added OpenTelemetry to my Node.js backend. Using the Node SDK together with automatic instrumentation, I configured: Express instrumentation HTTP instrumentation MongoDB instrumentation OTLP gRPC exporter Now every request automatically generates telemetry without modifying every route. Distributed Traces One of my favorite features is Distributed Tracing . Whenever I perform an action in Ram Store—like opening products or ad

Engineers are increasingly arguing that modern LLMs can already reason through root cause analysis once given correctly prepared context, shifting the hard problem to the pipelines that correlate telemetry. A Coroot experiment across eleven models offers early evidence for the claim. By Mark Silvester

I have a ritual. Whenever a pipeline latency alert hits my phone, my first instinct isn't to open a heavy dashboard or spin up a full Grafana instance. I grab my terminal and start firing curl commands at port 9600. curl -s localhost:9600/_node/stats?pretty ... curl -s localhost:9600/_cat/pipelines ... curl -s localhost:9600/_plugins . It's a repetitive, mindless sequence of commands. It works, but it's reactive and solo. You are the one parsing the JSON, you are the one looking for the pattern in the JVM heap usage, and you are the one manually correlating a spike in event flow with a specific thread lock. With the Model Context Protocol (MCP), that ritual is becoming obsolete. I've been experimenting with connecting MCP-compatible agents—specifically through Cursor and Claude—directly to Logstash via a specialized API server. The difference isn't just 'convenience.' It's an architectural shift from manual inspection to agentic triage. Moving beyond the Chatbot Most people treat AI like a documentation search engine. They ask, "How do I configure a JDBC input in Logstash?" That’s fine, but it doesn't help when your production cluster is turning 'yellow' at 3 AM. The real value of MCP isn't the ability to talk to an AI; it's the ability to give that AI a set of hands—specifically, a set of tools that can interact with live infrastructure. I recently integrated the Logstash Server-side Log Pipeline API into my workflow. This isn't some experimental script I wrote over a weekend; it’s a production-grade implementation built on MCPFusion. It gives an AI agent direct access to several critical Logstash endpoints through a controlled, sandboxed environment. The Triage Workflow: A Real Scenario Let's walk through how this actually changes the debugging loop. Imagine you have a spike in ingestion lag. In the old way, you’d be digging through terminal history. In the new way, your agent acts as an extension of your SRE toolkit. 1. Initial Health Check Instead of parsing raw

GitHub announced enterprise-managed OpenTelemetry export for Copilot activity from VS Code and Copilot CLI on July 8, 2026. Primary source: GitHub Changelog, July 8, 2026 . Export availability is only the start. The receiving collector becomes an enterprise ingress point. This is an unexecuted operating plan; signal types, attributes, endpoint requirements, and controls must be checked against current GitHub documentation. Isolate the path managed clients -> private telemetry ingress -> dedicated OTel Collector pool -> field policy + bounded queue -> dedicated backend dataset Do not point every developer client directly at the primary observability backend. Give the collector write-only destination credentials, separate its dataset from production application telemetry, and define retention before rollout. Isolation is not anonymity. Stable user, device, organization, or repository identifiers may still be sensitive. Start with a field budget Category Initial policy Product and version Keep bounded values Operation and status Keep documented enums Timing and counts Keep numeric measures Raw prompts or generated code Drop by default File paths and repository URLs Drop or transform after review User identity Prefer scoped pseudonymous identity Free-form errors Drop raw text; keep reviewed classes These categories are recommendations, not a description of GitHub's payload. Inspect a restricted canary before naming actual keys. processors : memory_limiter : check_interval : 1s limit_mib : 512 spike_limit_mib : 128 attributes/field_budget : actions : # Illustrative keys only; replace after payload review. - key : user.email action : delete - key : file.path action : delete - key : command.arguments action : delete batch : send_batch_size : 512 timeout : 5s Verify processors against the chosen Collector distribution. A valid startup does not prove that records satisfy policy. Drill three failures Backend outage: block the exporter. Retries must be bounded, queue growth vi

This article is my submission for the Agents of SigNoz Hackathon: Blog Track, where participants...