标签：#Observability

TL;DR: If you only collect metrics, Prometheus Agent mode is lightweight, familiar, and difficult to beat. If you collect metrics, logs, or traces together, or expect to in the future, Grafana Alloy's unified pipeline is usually worth the additional complexity. Once you've decided to move from pull-based scraping to a push architecture , the next question is which agent should actually run on each host. In 2026, the two strongest choices are Prometheus Agent mode and Grafana Alloy. I run Alloy across my production fleet, but that doesn't automatically make it the right answer for everyone. The Shift in the Monitoring Landscape Over the last couple of years, Grafana has consolidated both metrics and log collection into Grafana Alloy. Grafana Agent reached end of life on November 1, 2025, and Promtail followed on March 2, 2026. Neither receives security fixes anymore. The practical choice moving forward: Feature Prometheus Agent Grafana Alloy Metrics ✅ ✅ Logs ❌ ✅ Traces ❌ ✅ Config Prometheus YAML Alloy components Footprint Smaller Larger Learning curve Low Moderate Future direction Metrics agent Unified telemetry The table gives the short answer. The rest of this article explains where those differences actually matter in practice. Prometheus Agent mode. Run the Prometheus binary with the --agent flag and it stops acting as a full Prometheus server. It no longer stores local TSDB blocks, evaluates alerting rules, or serves queries. Instead, it scrapes targets, buffers samples in a write-ahead log, and forwards them upstream via remote_write . It is Prometheus with the storage and query layers removed. Grafana Alloy. A single agent that collects metrics, logs, and traces, processes them in a component pipeline, and pushes each signal to its backend. It embeds many exporters directly, so a line like prometheus.exporter.unix "node_exporter" {} gives you full node_exporter functionality without installing a separate binary. The Case for Prometheus Agent If you only need m

How I shipped structured JSON logging + Prometheus metrics with zero new dependencies I almost added structlog and prometheus_client to my pyproject.toml . Then I read what they actually do. Both libraries are excellent. structlog is the right call when you have a 30-engineer team shipping 50 services. prometheus_client is the right call when you have five teams of consumers scraping different metrics. For a single-author Python project with one process and one user, both are over-engineered. The 80 lines of code I would have pulled in, I can write in 200. The result: zero new runtime dependencies, full control over the output, and a smaller pip install footprint for every user. Here is what I did instead. The minimum useful observability surface A small Python service needs four things, in order of importance: Every log line is one JSON object. (No parsing for downstream tools.) Every request has a trace id. Every log line in that request carries the same trace id. (So you can grep by id and see the whole story.) Every log line goes to stderr. (So journald , Docker, and kubectl logs all see it without any extra configuration.) Every metric is exposed in Prometheus text format at a stable URL. structlog gives you #1, #2, #3 with a lot of flexibility. prometheus_client gives you #4 with a lot of flexibility. Both are about 16 MB of transitive dependencies combined. For a service that runs in a single process and exports maybe 20 metric names, the libraries are doing more work than the project. The 80-line JsonFormatter The custom logging formatter is the simplest part. The whole thing is here: import json import logging from contextvars import ContextVar from datetime import datetime , timezone _trace_id_var : ContextVar [ str | None ] = ContextVar ( " trace_id " , default = None ) class JsonFormatter ( logging . Formatter ): def format ( self , record : logging . LogRecord ) -> str : payload = { " ts " : datetime . now ( tz = timezone . utc ). isoformat (), " level

Where Copilot's telemetry stream lands, decided centrally GitHub added a control on July 8 that lets an enterprise mandate where the Copilot Chat extension in VS Code and Copilot CLI send OpenTelemetry data, removing the need for individual developers to set OTEL_* environment variables. Per the GitHub changelog, the setting is delivered through a telemetry block in the enterprise-managed settings, and a managed value takes precedence over environment variables and user settings. Four things are configurable in the block: the OTLP export endpoint and transport ( otlp-http or otlp-grpc ), the OTel service name and resource attributes, exporter headers such as an authentication token for the collector, and whether prompt, response and tool content is captured, with a separate flag for whether developers can change that. Delivery uses the channels documented on the same page: native MDM (Windows Registry or macOS managed preferences), server-managed settings from a signed-in GitHub account, or a file-based managed-settings.json . Where this bites The precedence rule is the point. If a platform team owns the collector and needs traces routed to it, this is exactly the switch they wanted. If a developer had their own OTLP endpoint pointed at a local sink, they will see the session start emitting somewhere else. The changelog does not describe a per-user override once a managed value is set. A scoping note is worth reading twice. The changelog states that managed exporter headers apply only to the Copilot Chat extension's OTLP exporter. The endpoint and transport policy still reach the CLI agent host, but the auth-token flow the changelog calls out is bound to the Chat surface. On-call teams standing up the collector should plan for that asymmetry before it lands as a surprise during triage.

On March 3, 2026, Helicone announced it was joining Mintlify. If you run Helicone in production, the practical question is not whether the acquisition is good or bad. It is what changes for you, and whether you need to do anything about it. Here is the honest version, and a checklist if you decide to move. What actually changed Helicone's founders joined Mintlify, and active feature development on the standalone product has wound down. The team has said security patches, bug fixes, and new model support will continue. New features and roadmap work are the part that stopped. For a lot of teams that is fine for a while. A logging proxy that already works does not stop working the day the roadmap freezes. But two situations make people start looking. You are on Helicone Cloud and you want to know the plan is still moving forward, not just being kept alive. Or you self-host and you were counting on features that are now unlikely to ship. Helicone was one of three observability tools acquired in a few months. ClickHouse bought Langfuse and Cisco bought Galileo in the same window. If you are picking a replacement, that pattern is worth keeping in mind. More on that at the end. Do you even need to move right now Worth saying plainly. If you self-host Helicone, you are happy with it, and you do not need anything new from it, there is no fire. The code keeps running. You can migrate on your own schedule instead of someone else's. The case for moving sooner is stronger if you are on the hosted product, if you depend on the gateway staying current with new providers and models, or if you would rather switch once now than watch and decide later. If that is you, the rest of this is for you. The migration checklist Helicone and Spanlens are both drop in proxies, so the mechanical part is short. The work is mostly finding every place your code sets a base URL and updating headers. 1. Swap the base URL This is the one required change. // Before, Helicone const openai = new OpenAI (

Request logging: vì sao mỗi log line phải có request id, và structured log cứu debug thế nào khi có sự cố production Request logging không phải là in console.log('got a request') cho vui. Nó là dấu vết duy nhất còn lại khi một request đã đi qua service, đi qua vài downstream, gặp lỗi, và bị đóng socket. Nếu log không có gì để nối lại các dòng thuộc cùng một request — không có requestId , không có traceId — thì log của một service RPS trung bình biến thành một mớ text xen kẽ giữa hàng chục request đồng thời, và câu hỏi "request X đã đi tới đâu, fail ở service nào" trở thành không trả lời được. Structured log (mỗi dòng là một JSON object với field cố định) + một correlation id truyền qua toàn bộ pipeline chính là cái làm log có thể query được, thay vì grep mù. Hai framework Express và Fastify tiếp cận khác nhau: Fastify tích hợp pino sẵn và tự sinh req.id , Express phải tự dán qua pino-http hoặc middleware tay. Nhưng cả hai đều vỡ theo cùng một kiểu khi correlation id bị đứt giữa các service. Cơ chế hoạt động Ba miếng ghép: (1) một structured logger — thường là pino trong ecosystem Node vì nó ghi NDJSON và có async destination, (2) một cơ chế sinh/nhận requestId , (3) một cách propagate id đó qua async work và qua HTTP call sang service khác. pino là JSON logger tối giản: mỗi lời gọi logger.info({...}, 'message') xuất một dòng NDJSON ra một WritableStream (mặc định là stdout). Log level là số ( trace=10, debug=20, info=30, warn=40, error=50, fatal=60 , theo pino docs), và có child logger — logger.child({ reqId }) tạo một logger mới bind sẵn các field, mọi log line từ child đều có reqId mà không phải truyền tay. Với Fastify , chỉ cần logger: true là có pino và request logging tự động — Fastify sinh request.id (mặc định là monotonic counter, đổi được qua option genReqId ) và log một cặp dòng "incoming request" / "request completed" cho mỗi request. Trong handler, request.log là child logger đã bind reqId : import Fastify from ' fastify ' import crypto from ' node:crypto

The previous code-graph series was about reshaping a static analysis graph so AI could query it. The same kind of reshaping is needed on the observability side. This post walks through four axes — application / infrastructure / CI / LLM — and the deliberately different shapes each one ends up in. The design judgments worth calling out: computing Gemini cost client-side instead of from billing API, sending Claude Code OTel straight to BigQuery instead of Loki, and shipping CI logs via post-hoc pull instead of webhook push.

Book: Observability for LLM Applications — Tracing, Evals, and Shipping AI You Can Trust Also by me: Agents in Production — the companion book in The AI Engineer's Library (2-book series) My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub Your triage agent has been in production for three months. The traces look clean. Every span is green, every run terminates, the p95 latency is flat, the token bill is boring. Then support forwards you a screenshot: the agent routed a billing refund to the security queue. You pull the trajectory. Nothing is broken. The agent called a reasonable tool with reasonable arguments, got a reasonable result, and picked the wrong queue with total confidence. That is silent drift. The trace shows you what the agent did. It does not tell you whether what it did was any good. Between a model provider's minor version bump, a prompt tweak someone shipped on Tuesday, and the slow shift in your incoming traffic, the quality of your agent's decisions moves. It rarely announces itself with an error. It shows up as a support ticket, then a second one, then a churned account. You catch it the same way you catch a memory leak: with a baseline and an alarm, not by staring at dashboards. Decision quality is a distribution, not a number The failing traces in Chapter 12 of Agents in Production are the easy case. Twenty retries of the same empty search is visibly wrong. You see the loop count in the invoke_agent parent and you know. Most quality regressions are not visible in a single trace. They show up only when you look at the distribution of decisions across thousands of runs. So the first thing to instrument is the decision itself. If you followed the tracing chapter you already emit a small fixed vocabulary per chat span: span . set_attribute ( " gen_ai.agent.step " , 3 ) span . set_attribute ( " gen_ai.agent.decision " , " call_tool " ) span . set_attribute ( " gen_ai.

The Cloud Native Computing Foundation (CNCF) has announced the graduation of OpenTelemetry, elevating the project to the foundation's highest level of maturity and formally recognizing it as production-ready for enterprise use. By Craig Risi

Book: Decoupled PHP — Clean and Hexagonal Architecture for Applications That Outlive the Framework Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools Me: xgabriel.com | GitHub You already run three tools that half-cover this job. Pulse gives you a live wall on a local route. Datadog runs an agent and prices on host and usage volume, so the bill scales with your infrastructure. Sentry catches the exceptions after they already hurt someone. And none of them can tell you the one thing you actually asked: the checkout request that took 900ms at 14:03 dispatched a job, that job ran a query, and the query is what timed out. Laravel Nightwatch reached general availability in 2025 as the framework's own APM, aimed straight at that gap. It is worth knowing exactly what it captures, what it charges, and where its knowledge of your app stops and yours begins. What Nightwatch actually is Two moving parts. A Composer package inside your app, and a separate agent process that ships the data. composer require laravel/nightwatch The package writes events to a local socket. The agent listens on 127.0.0.1:2407 , batches what it receives, and sends it to Nightwatch's cloud. Because the agent runs outside your request cycle, the request thread is not blocked waiting on a network call to a telemetry backend. Laravel puts the added cost at under 3ms per request ; take that as a starting figure and measure your own before you trust it. # environment token per app + environment NIGHTWATCH_TOKEN = your-env-token # start the collector (keep it running under a # process monitor: Forge daemon, Vapor, supervisor) php artisan nightwatch:agent # confirm it is alive and receiving php artisan nightwatch:status One detail that bites people: the agent has to be running for anything to arrive. In local dev you start it by hand. In product

Instacart redesigned its personalized marketing system using a configuration-driven multi-tenant architecture on Storefront Pro. The system replaces retailer-specific implementations with a shared execution engine, enabling scalable personalization, faster configuration propagation in under a minute, and 99.9% delivery success across hundreds of retail banners through a unified campaign platform. By Leela Kumili

TL;DR — A Kafka + Flink + OTel ingestion pipeline cost us ~$700–800/month at 10 MB/s. We rebuilt it as a single binary where the data, the write-ahead log, and the Iceberg catalog all live in S3 alone — no Kafka, no local disks, no coordination service — for ~$100/month . Here's the design. Self-hosted observability sooner or later runs into the problem of storing state. Query load, CPU, and data volume can all be handled by scaling out, but the stateful layer is something you have to operate by hand. At first it's almost unnoticeable: a disk degrades here, replication falls behind there, a recovery hangs somewhere else. As the data grows, incidents stop being one-offs and start to recur. At some point your observability stack - whether it's Grafana Loki, Elastic, or ClickHouse - starts demanding the same attention as a full-blown database that you're on the hook for. Kubernetes operators cover some of these cases, but operating the state is still on you. Managed solutions take that burden away and bring their own: rising costs, ingestion-pipeline constraints, and limits on retention and cardinality. But if you'd rather not sign up for the constant operational grind - or live with the constraints of managed solutions - it's worth asking: can we take the stateful part out of operations entirely? Storage and format The first candidate for offloading storage responsibility is Amazon S3. S3 gives you what local disks can't: fault tolerance and practically unlimited scale, with no storage to manage yourself. It isn't free, though: data-access latency goes up, and you pick up separate costs for API operations. For OLTP workloads that's a dealbreaker. For observability workloads - which are dominated by sequential writes and analytical reads - these trade-offs are often acceptable. At first glance, this problem is already solved. Loki , for example, uses S3 as its primary storage. But according to Loki's public documentation (v3.6.x) at the time of writing, Loki doesn't re

Asymm Systems has released Eliya 25.0.3, an OpenJDK 25 LTS distribution aimed at improving production diagnostics in Java environments. It consolidates several HotSpot features into an opt-in Production profile. Eliya is designed for teams needing reliable diagnostic data, especially in regulated settings. Future enhancements are planned for Phase 2. By A N M Bazlur Rahman

When Logs and Metrics Aren't Enough You have great dashboards. Your log aggregation is solid. But when a user reports "the checkout page is slow," you still spend 30 minutes jumping between services trying to find the bottleneck. That's the gap distributed tracing fills. What Tracing Actually Shows You A trace is a complete picture of a single request as it flows through your system: User Request → API Gateway → Auth Service → Product Service → DB → Cache → Response 5ms 12ms 45ms 120ms 3ms ^ This is your bottleneck Without tracing, you'd see: API Gateway: latency looks fine Auth Service: latency looks fine Product Service: latency is HIGH but why? With tracing, you see the exact DB query inside Product Service that's taking 120ms. Getting Started with OpenTelemetry OpenTelemetry is the standard. Here's a minimal setup: # Python example with Flask from opentelemetry import trace from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter # Setup provider = TracerProvider () provider . add_span_processor ( BatchSpanProcessor ( OTLPSpanExporter ( endpoint = " http://otel-collector:4317 " )) ) trace . set_tracer_provider ( provider ) # Auto-instrument everything FlaskInstrumentor (). instrument_app ( app ) RequestsInstrumentor (). instrument () SQLAlchemyInstrumentor (). instrument ( engine = db . engine ) That's it. Three auto-instrumentations cover 80% of what you need. Custom Spans for the Other 20% Auto-instrumentation gives you HTTP calls and DB queries. Add custom spans for business logic: tracer = trace . get_tracer ( __name__ ) def process_order ( order ): with tracer . start_as_current_span ( " process_order " ) as sp

We moved off our first tracer in month eight. The migration took one engineer the better part of a sprint, because the trace data lived in a schema we did not own. Nobody costed that line item on day one. I am writing this so you can. I run reliability for a small team shipping LLM features. When the pager goes off at 2am, I do not care which dashboard is prettiest. I care about two numbers: what this tool costs me per month, and what it costs me to leave. Those two numbers are the whole story, and they are almost never on the comparison page. So here are six Langfuse alternatives. For each I tracked both numbers: the monthly bill on the invoice, and the exit bill that only shows up the day you migrate. I compared Helicone, Arize Phoenix, LangSmith, Braintrust, Laminar, and Future AGI traceAI. They all trace LLM calls (prompts, tokens, retrieval spans, latency). The axis that decides your exit cost is whether the trace format is OpenTelemetry-native or a vendor schema. Get that wrong and the migration bill lands later, with interest. The cost nobody puts on the pricing page Your monthly invoice is the visible cost. The exit cost is the invisible one: re-instrumenting the app, rebuilding integrations, and losing historical traces when the schema does not travel. If your spans are OTel, the exit cost trends toward zero because the data is portable by construction. If they are proprietary, you are paying a deferred bill every month you stay. Sort on that first. Helicone. The gateway-first option. You proxy model calls through it and get logging, cost tracking, and analytics with almost no code change. Apache-2.0, self-hostable, roughly 5,800 GitHub stars as of June 2026. On pure observability ergonomics this is one of the strongest picks, and the proxy model means low setup cost. The thing to watch at scale: a gateway in the request path is one more hop to reason about when latency spikes. Arize Phoenix. The open-source OTel option. Tracing plus evals, self-hostable, a

Zabbix is an open-source monitoring platform that tracks the health and performance of servers, networks, applications, and services, with built-in alerting and visualisation. This guide deploys the Zabbix server, web UI, agent, and MySQL database using Docker Compose, with Traefik handling automatic HTTPS for the dashboard. By the end, you'll have Zabbix monitoring its own host with a secured dashboard at your domain. Set Up the Project Directory 1. Create the project directory: $ mkdir ~/zabbix-docker $ cd ~/zabbix-docker 2. Create the environment file: $ nano .env DOMAIN = zabbix.example.com LETSENCRYPT_EMAIL = admin@example.com MYSQL_PASSWORD = YOUR_DB_PASSWORD MYSQL_ROOT_PASSWORD = YOUR_ROOT_PASSWORD Deploy with Docker Compose 1. Add your user to the Docker group: $ sudo usermod -aG docker $USER $ newgrp docker 2. Create the Compose manifest: $ nano docker-compose.yaml services : traefik : image : traefik:v3.6 container_name : traefik restart : unless-stopped command : - " --providers.docker=true" - " --providers.docker.exposedbydefault=false" - " --entrypoints.web.address=:80" - " --entrypoints.websecure.address=:443" - " --entrypoints.web.http.redirections.entrypoint.to=websecure" - " --entrypoints.web.http.redirections.entrypoint.scheme=https" - " --certificatesresolvers.le.acme.httpchallenge=true" - " --certificatesresolvers.le.acme.httpchallenge.entrypoint=web" - " --certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}" - " --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json" ports : - " 80:80" - " 443:443" volumes : - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt networks : - zabbix-net mysql-server : image : mysql:8.4.8 container_name : zabbix-mysql environment : MYSQL_DATABASE : zabbix MYSQL_USER : zabbix MYSQL_PASSWORD : ${MYSQL_PASSWORD} MYSQL_ROOT_PASSWORD : ${MYSQL_ROOT_PASSWORD} volumes : - ./mysql-data:/var/lib/mysql networks : - zabbix-net restart : unless-stopped zabbix-server : image : zabbix/zabbix-se

Good architecture is not only about how a system is built. It is also about how well the team can understand that system once it is running. That is where observability belongs in the architecture conversation. It is common for observability to be treated as something that comes after the main engineering work. The service gets built. The API works. The deployment succeeds. Then, somewhere near the end, the team starts thinking about logs, dashboards, alerts, and operational visibility. That approach creates a gap. The architecture may look clean on paper, but once the system is in production, the team has to understand how it behaves under real conditions. Real users do not follow the happy path perfectly. Dependencies slow down. Queues back up. Data arrives in unexpected shapes. Deployments change behavior in ways that are not always obvious. If the system does not give the team a way to see those things clearly, the architecture is incomplete. Observability is not decoration around the system. It is part of the system design. Architecture Describes the System. Observability Shows the Truth. Architecture is built on assumptions. During design, teams make reasonable guesses about usage patterns, service boundaries, dependency behavior, data flow, scale, latency, and failure modes. Some of those assumptions are based on experience. Some are based on current requirements. Some are simply the best call the team can make with the information available at the time. That is normal. The problem is not that architecture contains assumptions. Every architecture does. The problem is when those assumptions cannot be tested once the system is real. A design might assume that an external dependency will be reliable enough. Production may show that it is the slowest part of the request path. A queue might look like a clean decoupling point during design. Production may reveal retry behavior, duplication, or ordering concerns that were not obvious upfront. A serverless function m

Dan Fineran explores how eBPF has evolved far beyond its roots in packet filtering into a robust, safe way to extend the Linux kernel. He explains how the eBPF "verifier", the security guardrail, enables implementation of deep observability and networking without the risks of traditional kernel modules or the slow upstreaming process. By Dan Fineran

OpenTelemetry has become the standard for observing modern systems. But when you start building AI applications, traditional traces aren't enough. You don't just want to know that a request happened. You want to know: Which model generated the output? Which provider was used? How many tokens were consumed? What embedding model processed the documents? How much did the operation cost? These questions become even more important when building Retrieval-Augmented Generation (RAG) systems. Recently while contributing to Mastra, I discovered an observability gap involving RAG embedding operations. This led me to open a pull request that introduced proper OpenTelemetry GenAI semantic mappings for RAG_EMBEDDING spans. The Problem Mastra already exported rich metadata for several AI operations. However, RAG embedding spans were missing standardized GenAI semantic attributes. As a result, observability tools could see that an embedding operation occurred, but they couldn't easily understand: Model information Provider information Token usage Embedding-specific metadata Without standardized semantic conventions, dashboards and tracing systems lose valuable context. This becomes a bigger issue in production environments where teams need visibility into AI workloads. Understanding RAG Embedding Spans A typical RAG pipeline looks like this: Documents ↓ Chunking ↓ Embedding Model ↓ Vector Database ↓ Similarity Search ↓ LLM Generation The embedding stage is critical. Every document chunk gets transformed into a vector representation. If observability data from this stage is incomplete, debugging performance issues becomes significantly harder. Why OpenTelemetry Semantic Conventions Matter OpenTelemetry doesn't just define traces. It also defines semantic conventions. These conventions create a common language for telemetry data. Instead of every framework inventing custom field names, everyone follows the same standard. For GenAI workloads this means tools can automatically underst

You've built a checkout reactor that reserves inventory, charges a card, generates a shipping label, and sends a confirmation email. It runs through Sidekiq. When something fails, compensation logic rolls it back. It works. Then your team asks: "How many checkouts failed this week? Which step? How long does the charge step take at p99? Can we see a trace through the entire system?" Before v0.5.0, you'd need to add logging calls to every step, build a custom Sidekiq middleware, and figure out how to correlate traces across async job boundaries. Now it's one line of config. Enter Middlewares Ruby Reactor 0.5.0 introduces a middleware pipeline — the same pattern that powers Rack, but designed for saga execution. A middleware is a plain Ruby object that hooks into the reactor lifecycle: class TimingMiddleware < RubyReactor :: Middleware def initialize ( ** options ) super @started = {} end def on_start_step ( step_name , _arguments , _context ) @started [ step_name ] = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) end def on_complete_step ( step_name , _result , _context ) started = @started . delete ( step_name ) return unless started elapsed = Process . clock_gettime ( Process :: CLOCK_MONOTONIC ) - started logger . info ( "step #{ step_name } took #{ elapsed . round ( 4 ) } s" ) end end This middleware times every step. Register it globally: RubyReactor . configure do | config | config . middlewares = [ TimingMiddleware ] end Now every reactor — every checkout, every refund, every data import — gets step-level timing, for free. The full lifecycle (20+ events) Middlewares can observe the complete execution lifecycle: Phase Events Reactor on_start_reactor , on_complete_reactor , on_failed_reactor Step on_start_step , on_complete_step , on_failed_step , on_retry_attempt Compensation on_start_compensation , on_complete_compensation , on_failed_compensation Undo on_start_undo , on_complete_undo , on_failed_undo Coordination on_lock_acquired , on_lock_failed , on_

TL;DR OOMKill is a reporting artifact, not a root cause. By the time the kernel logs the kill event and your alerting pipeline fires, the service already degraded for every user who hit The Alert You See Is Not the Problem You Have OOMKill is a reporting artifact, not a root cause. By the time the kernel logs the kill event and your alerting pipeline fires, the service already degraded for every user who hit it in the preceding minutes. Operators page on the kill. The latency damage is already done. Aspect What Operators Observe What Actually Happens OOMKill event Alert fires; pod is restarted Kill is the kernel's final action after degradation is already complete Silent pressure window No alert fires; no dashboard turns red p99 latency climbs as allocator contention serializes parallel work Incident attribution Logged as "OOM, increased limit"; latency spike blamed on network or dependency Root cause (limit headroom erosion) goes unaddressed; pattern repeats Limit headroom over time No automated signal warns of erosion Gap between working set and limit shrinks as traffic grows or data shapes shift Recommended alert threshold Triggered at kill event Trigger at 80% headroom consumption before kernel involvement The mechanism works like this. Kubernetes memory limits define a hard ceiling enforced by the Linux kernel's cgroup subsystem. When a container's resident set size approaches that ceiling, the kernel does not wait. It begins refusing new memory allocations. Silent pressure window The application's allocator blocks, retries, or falls back to slower paths. Garbage collectors in JVM and Go runtimes trigger earlier and more aggressively because the heap has no room to grow. Each of these responses adds latency to in-flight requests before a single OOMKill event appears in your logs. The kill is the kernel's final action after the application has already been running degraded. Silent pressure window. The interval between first memory pressure and pod termination is