标签：#Observability

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly what one line of well-intentioned code told it to do: announce a condition once . The line looked like this, and I suspect you have written it: if [ ! -f " $OFFLINEFILE " ] ; then echo "body context n/a — phone unreachable" > &2 touch " $OFFLINEFILE " fi exit 2 Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a rate. It changes the tense of the sentence. Every number, code listing, and command output below was re-measured on the machine while writing this, not quoted from the commit that fixed it. Two of the things I expected to find turned out to be false; both are in section 6, and one of them is the most interesting part. 1. Present tense, past tense phone unreachable is a claim in the present tense . It is a statement about the world right now, and it is what a reader of this tool wants: is the body sensor readable at this moment? Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense : the phone became unreachable, at some earlier point, at least once. That is a different proposition. It is true exactly once per transition and false forever after, which is why the guard can never fire twice, and why the sentinel's own mtime is the only surviving record of when the sentence was last true. The two propositions coincide on the first run. That is the whole trap. A first-time-only notice is indistinguishable from a live one for the length of one invocation, which is exactly the length of the test you will write for it. 2. What the reader got instead Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I pulled the pre-fix version straight out of git into a scratch path and ra

Most gateway tutorials stop at "here's how you route a request." That's the easy 20%. The hard part is what happens when a client hammers you with requests, a downstream service falls over mid-traffic, or you're staring at a 500 trying to figure out which of your four services actually caused it. I wanted to build something that hits those problems on purpose, so I put together spring-gateway-sample : a public gateway , an api-server that fans out to two downstream services, and a full observability stack sitting behind all of it. It's not a real product and never will be. But I tried to make it behave like one — including the annoying bits, like config tradeoffs and races that most demos just quietly ignore. Stack, for context: Spring Boot 4.1, Spring Cloud Gateway on WebFlux, Resilience4j, Redis, Postgres, Keycloak, Prometheus/Grafana/Tempo/Loki, and a small Vue 3 app for throwing traffic at it from a browser. The system, in one request Browser (Vue traffic simulator) │ Keycloak PKCE login + API key ▼ Gateway ── JWT + API-key auth, Redis rate limiting ──▶ routes to │ ▼ api-server ── WebClient delegation, circuit breakers, Caffeine cache ──▶ │ │ ▼ ▼ product-service pricing-service (JPA / Postgres) (JPA / Postgres) Every hop re-validates the JWT on its own — defense in depth, so the gateway isn't the single thing standing between the internet and the data. The gateway also checks an API key on top, because a JWT tells you who the user is, not which client application is calling on their behalf. You need that second identity if you want per-client rate limits or the ability to revoke one app's access without touching anyone else's. Two checks, one specific order Every request needs a Keycloak JWT and an API key, and the order they're checked in isn't an accident: Missing or expired JWT → 401 , before the API key is even looked at. Valid JWT, bad API key → 401 , but a different error code. Both valid, wrong role → 403 . Why bother with the ordering? Because "you're no

Free Tokens Are Not an SLO: An Ops Cost Drill for AI Batch Queues This week, two numbers trended: a harness at 100%, a model at 30%. For platform teams, a better pair is queue age and deadline slack. This article is a cost drill for the simplest AI batch path: free tokens, free server, non-negotiable deadline. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. That capacity is real. It is not an SLO. The tokens cost nothing. The queue is patient. Your deadline is not. The missing variable Token cost is easy to measure. Operations cost is easy to ignore. A free endpoint converts a per-token bill into a per-hour bill. The bill becomes your time, your retries, and your queue age. This drill keeps the ledger honest. It answers one question: what does a completed request cost when the token price is zero? Topology # worker.py (minimal, single-threaded) import queue import time import csv work = queue . Queue () for i in range ( 1000 ): work . put ({ " id " : i , " prompt_tokens " : 512 , " max_tokens " : 256 }) def call_model ( payload ): # replace with your free model endpoint return { " ok " : True , " in_tokens " : 512 , " out_tokens " : 180 } completed = 0 retries = 0 started_at = time . time () while not work . empty (): item = work . get () attempt = 0 while attempt < 4 : try : call_model ( item ) completed += 1 break except Exception : retries += 1 attempt += 1 time . sleep ( 2 ** attempt ) The worker is deliberately single-threaded. Free capacity often serializes. Serialization turns a token problem into a time problem. Declared test conditions 1,000 requests. One worker process. One free model endpoint. No client-side rate limiting. Deadline: 30 minutes. Ledger: one CSV row per request. Ledger and report # cost_ledger.py import csv import time HOURLY_OPS_COST = 50.0 # loaded engineering rate, adjust def record ( item , elapsed , retries ): with open ( " ledger.csv " , " a

Short answer: record one structured completion event at the request boundary, emit separate events for every asynchronous notification attempt, and join them with a stable notification ID; middleware latency and status code alone cannot reconstruct a delivery failure. For a gaming notification service, the deciding constraint is time. An API response may say that a guild invite was accepted while the actual push attempt occurs seconds later, perhaps on another process. Treating those two facts as one log event produces a comforting dashboard and a weak incident record. The architecture decision is to preserve both boundaries, give each event a precise meaning, and ship them outside the request's success path. This is deliberately an evidence design, not a logging-library choice. Express and Pino can implement the request-side contract in Node.js, but changing a serializer does not repair a missing correlation key or an ambiguous definition of completion. Decision, invariants, and failure boundaries The request completion event answers a narrow question: what did this process observe at its HTTP boundary? It should carry a timestamp, severity, service and environment, request ID, normalized route, method, response status code, and elapsed duration. If the request creates or addresses a notification, add a notification ID that remains stable across the queue and delivery worker. Do not make raw request or response bodies part of the default schema; tokens, chat text, player identifiers, and device data have different retention and access requirements from operational metadata. The delivery attempt event answers a different question: what happened when a worker tried to deliver that notification? Its useful fields include the same notification ID, an attempt number, channel, destination class rather than raw destination, outcome, and a bounded error category. A retry is another attempt event, not an edit to an old record. That append-only shape matters because the inte

Choice Setup burden Incident evidence Best fit Hosted metrics API Low Good if event context is preserved Small teams with an on-call rotation Postgres plus a custom dashboard Medium Excellent for joining metrics to business records Low-volume systems with strong SQL skills Self-hosted metrics stack High Configurable, but operationally demanding Teams that already run observability infrastructure Short answer: start with a hosted metrics dashboard API, send a small set of custom application metrics from Node.js, and retain reconstruction fields in Postgres. Choose the custom Postgres path when joins are the investigation, or self-hosting when data control outweighs maintenance. That recommendation has a catch. A chart can show when enrollment failures rose, but it cannot explain which course, release, region, or feature state produced them unless those dimensions were recorded at write time. For an edtech SaaS, the real deliverable isn't a pretty dashboard. It is enough evidence to replay the story of a customer incident without guessing. How can Node.js send custom app metrics to a hosted dashboard API? Capture the dimensions an investigator can act on: metric name, timestamp, deployment identifier, region, tenant or school identifier, operation, outcome, and a bounded error class. Keep direct student data out of labels. A useful event might say that lesson_publish failed validation in the EU region on deployment 7f3c2a1 ; it should not contain a learner's name, email, answer, or free-form support message. Small is good. Stop there. Start with service-level signals tied to customer work: request count, failure count, latency distribution, queue depth, and the age of the oldest queued job. Add business-flow counters such as course publication attempts only when they answer a concrete incident question. Don't export every database column as a label. High-cardinality dimensions make charts harder to read, alerts harder to tune, and the ingestion boundary harder to reas

Short answer: give each browser fetch a request ID, carry it to the backend in a standard HTTP header, and emit that same ID in structured logs on both sides. Keep the pricing decision itself behind a flag with an explicit evaluation ID, so a rollback can be verified instead of guessed. The browser is the first audit surface Rolling out a new pricing rule in an edtech app sounds like a feature-flag task. Operationally, it is a tracing problem with money attached. A student sees a price in the browser, the frontend calls the checkout backend, and the backend evaluates a flag before writing an order. When those events cannot be joined, a rollback turns into a debate about which request produced which price. I've been paged for missed jobs and duplicate deliveries. The same failure pattern appears here: a dashboard says the system is healthy, but the individual request that matters is hard to reconstruct. A request ID doesn't prove that a price was correct. It makes the evidence joinable. The smallest useful contract is straightforward: The browser creates a non-secret request ID for each outbound fetch. The ID travels in X-Request-ID (or the equivalent header chosen by the team). The server validates or replaces malformed values, then logs the accepted value. Every log record for the request includes the ID, route, outcome, and duration. A separate flag-evaluation ID identifies the pricing decision and its rule version. Don't put a user email, token, or price in the request ID. It's a correlation key, not an authorization mechanism or a business record. How should frontend and backend logs correlate a browser fetch request ID? The browser and server need a shared boundary, not a shared logging library. For a JavaScript or Node.js application, the fetch wrapper should generate an ID before sending the request and attach it to the headers. The Node.js service should read that header at the HTTP edge, bind it to request context, and include it in every subsequent log eve

A solid observability setup usually comes down to three pieces working together: something that collects metrics, something that exposes system-level metrics, and something that visualizes it all. Here's what each one does and how to install them. The Theory: How This All Fits Together Before installing anything, it helps to understand the model, because it's a bit different from how logging or alerting tools usually work. Pull, not push. Most people's first instinct is "the app should send its metrics somewhere." Prometheus flips that around — it pulls metrics on a timer instead. Every target (a machine, a service, an app) exposes a simple HTTP endpoint, usually /metrics , that just returns plain text numbers. Prometheus visits that endpoint every N seconds (the "scrape interval") and saves whatever it finds, with a timestamp attached. Nothing gets pushed to Prometheus — Prometheus goes and asks. This means for anything to show up in Prometheus, it has to satisfy one requirement: something has to expose a /metrics endpoint Prometheus can reach. That's the whole game. Everything else in this stack exists to satisfy that one requirement or to make the data useful afterward. Why Node Exporter exists. Your operating system doesn't naturally speak Prometheus's language — it doesn't expose CPU/memory/disk stats as a /metrics endpoint by default. Node Exporter's only job is to read stats the OS already tracks (via /proc and /sys on Linux) and republish them in the text format Prometheus expects, on port 9100. It's a translator, not a monitoring tool by itself — it collects nothing, decides nothing, alerts on nothing. It just answers "what does this machine look like right now?" whenever asked. Why Prometheus itself is separate. Prometheus doesn't know anything about CPUs or memory — it has no idea what it's scraping. It just knows: "go hit this list of URLs on a schedule, and remember what comes back." The intelligence is in the config (which targets to scrape, how often)

Short answer: compare a hosted app log search service, self-hosted Loki, and Elastic Cloud by the operational boundary each one creates. Low effort, data control, and search depth are different decision axes; the cheapest choice is the one that produces a trustworthy signal without making a small team operate a second product. That last sentence is the decision rule. A low invoice is not a useful bargain if the first incident reveals missing logs, duplicate alerts, or an index that nobody knows how to restore. The incident lesson: a log is not a health signal I've been paged for two different failures: a scheduled import that stopped producing results, and a job that delivered the same result twice. Both incidents had logs. Neither incident was solved by collecting more text. The invariant is simple: observability has to describe both activity and the absence of expected activity. An app log search system can help investigate an import after an alert fires. It cannot, by itself, prove that an import that should have run did not run. That missing event needs a heartbeat, a durable job record, or a metric with an explicit freshness deadline. For an edtech application importing course data, I would record the import name, run identifier, start and finish timestamps, outcome, item count, and an idempotency key. The alert should fire when the expected completion window passes, not whenever somebody happens to search a log stream. Duplicate deliveries should be visible as a repeated idempotency key, not mistaken for two successful business operations. Keep the signal narrow. The log search layer then answers the next question: what happened around the missed or duplicated run? That division keeps noisy search data from becoming the only source of truth for scheduled work. How should a small business compare self-hosted and hosted app log search? Compare the complete operating boundary, not the storage line item. A self-hosted Loki deployment gives the team direct control

Short answer: for a Node.js app in Docker or Kubernetes, give startup, readiness, and liveness probes separate meanings, keep routine health traffic out of application logging, and measure state transitions instead of counting every successful check. For a property-management API rolling out a new pricing rule, this preserves useful metrics: whether an instance can calculate rent correctly and accept traffic, without turning each kubelet poll into noise. Which health signal should control each container decision? Start with the decision, not the endpoint name. Signal Question it answers Include Exclude Action Startup Has initialization completed? Configuration parsing, pricing-rule compilation, required local warm-up Long-term dependency health Allow the process more time before other probes apply Readiness Can this instance safely receive a new pricing request now? Ability to serve the active rule version and any required dependency state Optional analytics and background exports Remove the pod from Service endpoints Liveness Is the process stuck beyond local recovery? Event-loop progress or another narrow process invariant Database, cache, and third-party availability Restart the container This split is the main noise filter. A downstream dependency becoming unavailable can make a pod unready, but restarting the same healthy process usually doesn't repair that dependency. If the dependency is placed in liveness anyway, every pod can restart together. The health response has then amplified one problem into two: lost capacity plus a restart storm. The pricing rollout makes readiness more demanding than “the port is open.” Imagine rule version rent-2026-08 is enabled for one building cohort. A newly started instance has loaded configuration but hasn't compiled that version yet. It is alive. It isn't ready. Its startup check should hold back liveness and readiness until initialization finishes; afterward, readiness should stay false until the active rule can be evalua

A failure message can be technically correct and still be frustratingly incomplete. Consider a timeout. It tells us something important about the failure mechanism, but not which operation encountered it. Adding the complete request target might answer that question, yet it can also expose identifiers, query parameters, access material, or other data that never belonged in a broadly visible diagnostic record. A safer middle ground is to give failures two separate coordinates: a reason code that explains how the operation failed, and a bounded operation label that explains where it failed. That distinction makes diagnostics more useful without turning failure handling into an accidental data-exposure channel. A reason code is not a location Reason codes describe failure mechanics. Generic examples might include deadline , cancelled , unauthorised , or invalid_response . These codes are valuable because they let systems group similar outcomes. A dashboard can count deadline failures across operations, while application logic can decide whether a particular reason is retryable. What a reason code cannot reliably explain is the operation being attempted. A deadline during a summary read may require a different investigation from a deadline while assembling a detailed response. Combining both meanings into one free-form message makes failures harder to query and encourages presentation text to become an informal data model. Model the two coordinates separately A deliberately generic, invented C# model might look like this: public enum OperationArea { Summary , Detail , Archive } public sealed record FailureDetail ( string ReasonCode , OperationArea ? Area = null ); The reason remains suitable for classification. The operation label adds location without carrying an unrestricted request value. An enum is not the only option. A validated value object or centrally managed set of constants can work too. The important constraint is that labels come from a small, reviewed voca

TL;DR For marketplace sales calls, use an async job when several documents must become one reviewed set of CRM actions; use an inline request only when one short document can finish inside the caller's latency budget. Preserve one result per input, expose partial progress, and export only records that carry their source ID, outcome, and schema version. Start with this decision table: Pick Use it when Quality and latency consequence Operational burden Inline request One short transcript produces one independent summary Fast feedback, but the request deadline limits retries and review stages Low until traffic spikes or callers retry Bounded parallel calls A small set of independent transcripts can finish separately Lower wall time, with variable completion order The caller owns concurrency, backoff, and reconciliation Durable async job Multiple documents feed one CRM export or need validation More queue latency, but enough room for retries and quality checks Requires job state, idempotency, metrics, and retention rules The important boundary is not "batch or no batch." It is ownership. If the API accepts a collection, the service should own that collection through terminal results and a verifiable export. Don't make a client reconstruct truth from whichever promises happened to resolve. What should a Node.js batch summarization API do with multiple documents? It should turn an admission request into a stable job record, process every document under a declared concurrency limit, and publish an item-level outcome before it declares the job complete. The result model needs at least four identities: job, input document, processing attempt, and export. Without them, a duplicate submission can look like new work, a retry can overwrite useful evidence, and an export can silently omit a failed call. For the marketplace example, imagine that a seller has three calls about the same account: discovery, pricing, and legal review. The desired CRM update is not merely three paragra

Grafana Labs has announced general availability for two tools that let AI coding agents query live observability data during development: the gcx CLI and the Grafana MCP server. Both allow agents to pull metrics, logs, traces, SLOs, and Synthetic Monitoring results from Grafana Cloud or a self-hosted stack By Claudio Masolo

The retry trap A free model CI job fails on a timeout. You click retry. The whole pipeline starts over: checkout, build, dependencies, model call. That is the trap. Why re-run the world for one timeout? Retrying the pipeline does not isolate the flaky step. It makes a small problem expensive. I wanted a workflow that replays just the model call, not the whole pipeline. So I made every free model call leave behind a tiny reproducible record. A record has two halves: the input envelope and the output hash. If the job fails, I can replay the input against the same model and compare the output hash. No full pipeline re-run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I use MonkeyCode's free model access for the model step and its free server option as a small replay store. I do not assume exact quotas, model names, or availability windows here. The pattern works with any free HTTP model endpoint and any tiny key-value store or CI artifact. Why a hash and not the full prompt Full prompt logs are useful until they are not. A free model job may receive a snippet of a merge request, an error message, or an environment variable. Store the raw text in CI logs and you can accidentally leak source or secrets. Store a hash and the replay input in a locked artifact, and the risk drops. A hash also gives me one cheap comparison target. I do not need to reason about the entire response to see that an endpoint changed. I only need byte-level equality. The record shape For every model call, I save the fields below. request_id: a hash derived from model, prompt hash, and a timestamp. prompt_hash: the hash of the normalized prompt. response_hash: the hash of the raw response. status: the HTTP status of the original call. bytes: the length of the response. The exact hash algorithm matters less than using the same one on both sides. I use SHA-256 because it is available everywhere. GitLab CI wiring I run two jobs. The first job calls the model and post

For a long time, my mental model of observability was this: you import an SDK, sprinkle some calls through your code, each call fires off data to a server somewhere, and a dashboard reads it back. A logging system with extra steps. That model is wrong in a specific, interesting way. And I couldn't see how it was wrong until I stopped looking at the dashboards and started looking at what actually gets emitted, and how. The seductive wrong model The wrong model is seductive because the plumbing really does look identical. Logging: emit, store, search. Observability: emit, store, query. Same loop, right? So my working theory became: observability is logging plus some fancy logic to analyze the logs. Close. But no. The difference isn't in the analysis. It's in the emission — and it splits into three mechanisms that have almost nothing in common with each other. Descent one: metrics aren't events at all A metric is not a record you write. It's a number sitting in your app's memory . requests_total . increment () // 1, 2, 3... request_duration . record ( 0.23 ) // adds to a histogram Nothing is sent when this line runs. The number just changes in RAM. Periodically — every 15 seconds, say — either a backend scrapes an endpoint your app exposes, or a collector ships the current values out. That's why metrics are absurdly cheap: a million requests is one counter reading "1,000,000", not a million records. You could never reconstruct a clean p99 latency graph by parsing log text. The histogram was built for it at write time. And the stateless-container objection answers itself: the in-memory counter is disposable. Each instance flushes to the backend on a schedule (on serverless, a sidecar collector even does a final flush at shutdown), and the backend sums across instances. The durable truth never lived in your app. Descent two: logs are the familiar part Logs work exactly the way I always assumed everything worked: an event, written out, shipped, searched. The only upgrade

One of the most common assumptions in engineering leadership is that a rising number of reported incidents signals declining system reliability. However, a recent article from Great Circle argues that the opposite is often true: an increase in incident counts may actually indicate that an organization's incident management culture is improving. By Craig Risi

Per-tenant cost visibility changes the design: don't begin with parallel API calls; begin with a durable ledger that ties every classification result and usage record to a tenant, policy version, and source item. For a customer-support catalog backfill, the practical choice is a bounded Node.js worker that reads existing posts and comments, classifies them through a replaceable adapter, checkpoints each result, and exports tenant-scoped JSONL. Short answer: make the ledger the product of the job and the LLM call one restartable step inside it. That ordering matters when support conversations contain messy product descriptions such as “the small blue charger for the old tablet.” The moderation label decides whether the text is safe to reuse; the enrichment labels connect it to a catalog candidate. Operations still need to answer a less glamorous question: which tenant consumed the tokens? Make tenant cost visible before optimizing it Token totals belong beside decisions, not in an unrelated monthly dashboard. Record normalized input and output token counts on every completed row, then aggregate by tenantId , policyVersion , and time window. If the API reports different usage units, preserve the raw usage payload in restricted telemetry and map it explicitly; don't pretend unlike units are interchangeable. Start there. Three signals are enough for the first useful view: Signal Group by Operational question Completed items tenant, policy version Is the backfill moving? Input and output tokens tenant, model Where is consumption occurring? Review and block counts tenant, content kind Did the decision mix shift? Cost in currency should be derived from a versioned rate configuration, not baked into historical rows. Store usage and the model identifier, then apply the applicable rate when producing a report. This keeps a rate change from rewriting what the runtime actually observed. It also lets finance reproduce an invoice-period view while engineering inspects tokens per

An adapter layer becomes strategically useful when several teams need one observability contract but cannot, or should not, standardize on one agent framework. AI SDK, LangChain.js, OpenAI Agents SDK, and direct model clients organize execution differently. One emphasizes generation and streaming, another exposes hierarchical callbacks, another has agent runs and handoffs, and a direct client exposes only provider requests unless the application adds its own spans. Unified tracing should preserve those differences while translating the common lifecycle into one model. Done well, teams can share execution-tree tooling, CI quality gates, privacy policy, and telemetry export without coupling every consumer to every framework. Unify Semantics, Not APIs The frameworks do not need a shared callback interface. They need a shared answer to a smaller set of questions: What is the root operation? Which model, tool, retrieval, decision, and handoff spans occurred? What was each span’s parent? How did it end? Which usage and timing metrics are available? Which facts are unavailable from this integration? The framework adapter converts its native lifecycle into those semantics. Consumers never call framework hooks directly. AI SDK lifecycle ---------┐ LangChain callbacks ------+--> framework adapters --> trace core OpenAI Agents tracing ----+ direct client wrappers ---┘ | +------------+------------+ | | | execution UI CI rules telemetry sinks A Practical Mapping Matrix The exact public APIs change over time, so keep the mapping conceptual and verify it against the supported framework version. Normalized concept AI SDK-style integration LangChain-style integration OpenAI Agents-style integration Direct client Root run Application request or generation Chain, graph, or agent run Agent trace or runner invocation Manual application span Model span Generation or stream lifecycle LLM/chat-model callback Model generation item/span Provider request wrapper Tool span Tool execution lifec

Part 6 of a series building a support-ticket agent with no framework. Previous: Part 5 (guardrails). Repo: github.com/akash-pal/agent-from-scratch "Run the eval set" and "is this agent healthy right now" are different questions, and it's easy to only build infrastructure for the first one. Eval sets run offline, on cases you already thought of. Production traffic doesn't ask permission to send you a ticket type you didn't anticipate. Observability is what tells you when that's happening — and it's also, unglamorously, what makes offline evaluation possible in the first place: you can't debug a failing eval case without knowing what the agent actually did, step by step. The minimum trace payload Every tool call in this build logs a structured record — src/trace.ts : export interface TraceStep { trace_id : string ; step_id : number ; tool_name : string ; args_hash : string ; // hashed, never raw args duration_ms : number ; result_summary : string ; model : string ; token_usage : { input : number ; output : number }; } Two details here that look small and aren't: args_hash , not raw args . This trace log is meant to be safe to keep around, ship to a monitoring system, or paste into a bug report — none of which should require thinking about what secrets might be embedded in a tool call's arguments. Hashing means you can still confirm two calls used identical arguments (for debugging idempotency, for instance) without ever persisting the actual values: export function hashArgs ( args : Record < string , unknown > ): string { return " sha256: " + createHash ( " sha256 " ). update ( JSON . stringify ( args )). digest ( " hex " ). slice ( 0 , 8 ); } result_summary , truncated. Full tool results can be large (a kb_search returning full article bodies, for instance) — logging the whole thing on every step makes trace output unreadable and bloats whatever's storing it. summarizeResult takes the first few fields and truncates long values: const MAX_FIELD_LEN = 70 ; export funct

You attach the OpenTelemetry Java agent, point it at a collector, and within minutes Grafana is drawing a service map you never drew. A box for each service, arrows between them, latency on every edge. It feels like magic, and — more dangerously — it feels complete . "The agent traces everything" is the sentence repeated in every onboarding doc. This is the story of the moment that sentence stopped being true on my platform, why I'm glad it did, and the difference between a system that is working and a system you can actually see . The flow everyone trusts The platform is an event-driven set of Spring Boot services: an API gateway in front, a user-service backed by MySQL, a notification-service backed by PostgreSQL, and Kafka carrying events between them. A user is created, an event is published, a notification is sent. I didn't want to draw that topology. A hand-drawn architecture diagram is documentation that drifts — true the day you commit it, slightly wrong a month later, actively misleading after a quarter. I wanted the dependency graph generated from live traffic , so it would always reflect what the system actually does. Grafana Tempo does exactly this. Its service-graphs processor reads matched client/server span pairs out of trace data and emits a metric — traces_service_graph_request_total — that Grafana renders as a node graph. No edge is ever wired by hand. The topology is derived, continuously, from real spans. The edge that wasn't there I generated the graph and the synchronous edges lit up immediately: api-gateway → user-service user-service → MySQL notification-service → PostgreSQL Then I looked for the one edge I actually cared about — user-service → notification-service , the asynchronous hop over Kafka. It wasn't there. The naive conclusion (and why it's wrong) The tempting read is immediate and obvious: the async hop is broken. The event isn't getting across. Go debug the consumer. So I checked. And the consumer was completely fine. notification

It’s 2:15 AM. Your phone buzzes aggressively. 🚨 You jump out of bed, open your laptop with half-closed eyes, and join an emergency incident response call. Your team’s Slack channel is exploding: ⚠️ [ALERT] Payment API 500 Error Rate > 15% ⚠️ [ALERT] Redis Latency Timeout (>5000ms) ⚠️ [ALERT] Node-04 CPU Saturation (98%) You spend the next 2 hours manually connecting the dots: querying Prometheus metrics, scrolling through endless Loki logs, cross-referencing Tempo traces, and checking recent ArgoCD deployments. Eventually, you uncover the truth: Deployment #218 , pushed right before midnight, introduced a subtle memory leak that triggered GC pressure, spiked CPU, starved the Redis connection pool, and knocked down the Payment API. Sounds familiar? 😅 💥 The Problem: Observability Shows Symptoms , Not Causes Modern observability tools like Grafana, Prometheus, Loki, and Jaeger are fantastic at collecting metrics, logs, and traces. But they suffer from one fundamental design limitation: They tell you WHAT is breaking, but leave you to figure out WHY it broke. When a microservice fails in Kubernetes, it triggers a domino effect ( cascading failure ): Deployment #218 (Memory Leak) │ ▼ Garbage Collection Pressure │ ▼ CPU Saturation (98%) │ ▼ Redis Connection Timeout │ ▼ API Gateway Retry Storm │ ▼ Payment Service Down (HTTP 500) Traditional alerting floods you with alerts for the bottom 4 nodes (the symptoms), leaving SREs and DevOps engineers stuck sifting through noise during high-stakes outages. 💡 Introducing IRCAE: Autonomous Root Cause Engine To solve this, we are building IRCAE (Intelligent Root Cause Analysis Engine) —an open-source, enterprise-grade platform designed to turn raw telemetry into autonomous causal reasoning . Instead of asking SREs to correlate telemetry manually, IRCAE automatically answers: "Why did the system fail?" in less than 10 seconds. 🌟 Key Highlights 🚀 Written in Rust (Axum + Tokio) : Built for high-throughput, near-bare-metal performance wi