标签：#monitoring

A sensor on my machine returned nothing at all — empty stdout, empty stderr, exit code 2 — on every invocation for 36 days. It was not crashed. It was not misconfigured. It was doing exactly what one line of well-intentioned code told it to do: announce a condition once . The line looked like this, and I suspect you have written it: if [ ! -f " $OFFLINEFILE " ] ; then echo "body context n/a — phone unreachable" > &2 touch " $OFFLINEFILE " fi exit 2 Read it as a rate limiter and it is obviously fine: don't spam the log with the same message every five minutes. Read it as what it actually is and it is a bug, because the guard does not limit a rate. It changes the tense of the sentence. Every number, code listing, and command output below was re-measured on the machine while writing this, not quoted from the commit that fixed it. Two of the things I expected to find turned out to be false; both are in section 6, and one of them is the most interesting part. 1. Present tense, past tense phone unreachable is a claim in the present tense . It is a statement about the world right now, and it is what a reader of this tool wants: is the body sensor readable at this moment? Wrapping it in [ ! -f "$SENTINEL" ] silently rewrites it into the past tense : the phone became unreachable, at some earlier point, at least once. That is a different proposition. It is true exactly once per transition and false forever after, which is why the guard can never fire twice, and why the sentinel's own mtime is the only surviving record of when the sentence was last true. The two propositions coincide on the first run. That is the whole trap. A first-time-only notice is indistinguishable from a live one for the length of one invocation, which is exactly the length of the test you will write for it. 2. What the reader got instead Here is the tool, before the fix, run twice in a row against a phone that is genuinely away. I pulled the pre-fix version straight out of git into a scratch path and ra

A Windows 11 VM in Proxmox VE was showing nearly 100% memory usage in monitoring. Inside Windows Task Manager, however, actual memory usage was only around 30–50% . At first glance, that looks like a monitoring problem. It wasn't. The issue was in the VM configuration: the PVE Ballooning Device had been disabled , which meant Proxmox VE was not receiving the guest memory statistics needed to reflect the actual Windows memory state. I encountered this while monitoring a Proxmox VE environment with OpsHome NOC. This post documents how I traced the discrepancy and fixed it. The symptom On the same Proxmox VE host, the memory usage of Ubuntu VMs looked normal. One Windows 11 VM was different. The VM had 24 GB of RAM configured, but the monitoring result remained close to: Memory: 100% Used: about 24.2 GB Total: 24 GB Inside Windows 11 Task Manager, however, the VM was clearly not using all of its memory. The difference looked roughly like this: Monitoring: 90%–100% Windows 11: 30%–50% That is too large a difference to treat as a normal sampling variation. If you encounter something similar, especially when Linux VMs on the same Proxmox host look normal, do not immediately assume: Windows has a memory leak The monitoring threshold is wrong The monitoring application is calculating memory incorrectly The more important question is: Is Proxmox VE actually receiving the correct memory statistics from the Windows guest? Checking BalloonService inside Windows 11 For Proxmox VE to obtain useful guest memory statistics from a Windows VM, the VirtIO Balloon driver and its related Windows service need to be available. Inside Windows 11, I opened PowerShell and checked BalloonService: Get-Service * balloon * The result showed: Running BalloonService So the Windows-side BalloonService was already installed and running. At this point, the guest-side service did not appear to be the problem. The next step was to check the VM configuration on the Proxmox side. Checking the Proxmox VE

Linux server running out of memory? Learn how to diagnose and fix high memory usage with real commands — before it takes down your app. Your app starts slowing down, the OOM killer fires, or your monitoring page turns red — and the culprit is memory. High memory usage on a Linux server is one of the most common production crises for small teams, and it's easy to misread. Linux intentionally uses most of your RAM for caching, so a server showing 95% memory used isn't necessarily in trouble. But one that's exhausting real working memory and swapping is. Here's how to tell the difference and actually fix it. Step 1: Get a Clear Picture of What's Using Memory Start with the basics. Run 'free -h' to see total, used, free, and available memory. Focus on the 'available' column — that's the real number. It accounts for reclaimable cache and is far more useful than 'free'. free -h — quick overview of RAM and swap usage vmstat 1 5 — five one-second snapshots; watch the 'si' and 'so' columns for swap-in and swap-out activity cat /proc/meminfo — full breakdown including Slab, PageTables, and AnonPages If swap is actively being used (si/so values above zero consistently), your server is genuinely memory-constrained. That's different from swap space existing but sitting idle. Step 2: Find the Processes Eating Your RAM Once you know memory is tight, you need to know what's consuming it. Run 'ps aux --sort=-%mem | head -20' to list the top 20 processes by memory percentage. For more detail on actual RSS (resident set size) in human-readable form: ps -eo pid,ppid,cmd,%mem,rss --sort=-%mem | head -20 RSS is the memory a process actually holds in RAM — not virtual memory, which is often misleadingly large. Another useful tool is 'smem', which calculates PSS (proportional set size) and gives a fairer view when processes share memory libraries. Install it with 'apt install smem' or 'yum install smem', then run 'smem -r -k | head -20'. Look for processes with unexpectedly high RSS. A Nod

Seri: Proxmox VE Cluster ve Corosync | Hafta 5 Serinin adı "Cluster ve Corosync"; ama dört modüldür ağırlık HA Manager, resource affinity ve CRS'teydi, Corosync'in kendisine (redundant link'ler, izleme araçları) hiç dönmemiştim. Bu modülde iki konuyu birleştirip derinlemesine işledim: birden fazla corosync link'i tanımlayıp gerçekten birini kesip diğerinin devralmasını kanıtlamak, ve günlük operasyonda kullanılacak izleme araçlarını tek tek denemek. İkisi de planladığımdan çok daha fazla soru açtı; biri yanlış bir config anahtarı yüzünden saatler süren bir araştırmaya dönüştü, diğeri ise hiç beklemediğim bir kilitlenme keşfiyle bitti. Bölüm 1: Redundant Corosync Links Kurulum: İkinci Link'i Eklemek Şu ana kadar cluster'ımızda tek bir corosync link'i vardı ( link1 , izole corosync-net ağı). Management ağını ( 192.168.122.x ) link0 olarak ekleyip gerçek bir yedeklilik kurdum; /etc/pve/corosync.conf 'u kopyalayıp düzenleyip atomik olarak yerine taşıdım: cp /etc/pve/corosync.conf /etc/pve/corosync.conf.new # nodelist'teki her node'a ring0_addr ekledim, totem'e ikinci bir interface bloğu ekledim mv /etc/pve/corosync.conf.new /etc/pve/corosync.conf Doğrulama: corosync-cfgtool -s LINK ID 0 udp addr = 192.168.122.11 status: ... connected ... connected LINK ID 1 udp addr = 10.10.10.11 status: ... connected ... connected Teknik olarak başarılı; iki link de bağlı. Ama log'a dikkatlice bakınca, mimarimizin niyetini tersine çeviren bir şey oldu: [KNET ] rx: host: 3 link: 0 is up [KNET ] host: host: 3 (passive) best link: 0 (pri: 1) link_mode: passive modunda, öncelik eşitken düşük numaralı link kazanıyor . link0 'ı sonradan eklediğim için, o Corosync'in asıl trafiğini üstlenmiş; Modül 0'da özellikle izole ettiğimiz corosync-net ( link1 ) sessizce yedek konuma düşmüştü. Yanlış Anahtar, Saatler Süren Bir Araştırma Bunu düzeltmek için link1 'e daha yüksek öncelik vermeye çalıştım: interface { linknumber : 0 priority : 5 } interface { linknumber : 1 priority : 10 } İşe yaramadı. cor

Most monitoring watches for something bad to appear: a 500, a timeout, an expired certificate, a slow response. A heartbeat monitor does the opposite. It watches for something good to stop appearing . Your cron runs, your backup completes, your embedded device phones home, your queue worker drains — and each of those pings a URL to say "I'm still alive." The monitor's job is to notice when the pings go quiet. That inversion is the entire value. A cron that fails throws an error you can catch. A cron that stops being scheduled — the box got reimaged, the systemd timer got disabled, the container never came back after a deploy, the account got suspended for an unrelated billing issue — throws nothing at all. There is no log line, no exception, no non-zero exit. There is only the absence of the thing that used to happen. You cannot alert on an event that does not fire. You can only alert on the silence. So heartbeat monitoring looks trivial: store a timestamp on every ping, and if now - last_seen > expected_interval , fire an alert. It is about ten lines. And it is exactly those ten lines that will let 43 days of downtime pass without a second word — because the hard part of a dead-man's switch is not detecting the death. It is staying loud after it. I know because it happened to our own. Three states, and why the third one must stay silent Start with the check itself. A naive heartbeat has two states — alive or dead — and both are wrong at the edges. The real answer set has three: alive — a beat arrived within period + grace . Everything is fine. dead — the last beat is older than period + grace . The thing stopped. Page someone. unknown — the monitor exists but has never received a single beat. That third state is where two-state heartbeat monitors self-immolate. A brand-new heartbeat you just created has no last_seen timestamp. If your rule is "alert when last_seen is too old," a null last_seen is infinitely old, so the monitor pages you the instant you create it —

Last week I turned my data audit into a build step : a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself. It did nothing about the site agreeing with itself on something false. That's not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage . No diff between my own sources could ever have caught it, because every internal source was equally behind the world. A build gate proves agreement. Agreement is not truth. Something has to look outside. You can't diff against the world, but you can sample it The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction. I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on. Laws don't change on my audit schedule. So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row. Fourteen, not all 271, because the selection is doing the real work: A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most. Everything else sits on a rotating cursor : eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week. Two rules were non-negotiable, both inherited from

When every unauthorized request becomes the same dashboard line, diagnosis turns into guessing. ASP.NET Core 10 authentication metrics give me a better split: did the handler have nothing to authenticate, reject supplied credentials, or accept them? That distinction matters because a client deployment that drops credentials needs a different response from a surge of malformed or expired credentials. ASP.NET Core 10 added built-in authentication and authorization instruments to System.Diagnostics.Metrics . I can collect them without rewriting each handler, and I can lock their behavior into an offline test before wiring up a production exporter. Why one 401 hides two different problems A protected endpoint normally challenges an unauthenticated caller. The final status is 401 whether the caller sent nothing or the handler rejected what it received. The authentication duration histogram exposes the missing context through aspnetcore.authentication.result : Result What the handler reported A common interpretation none No authentication result No applicable credentials were available failure Authentication failed Supplied credentials were rejected or processing failed success A principal was created Authentication completed successfully _OTHER Another framework result Preserve it as an explicit catch-all none is a handler result, not a universal synonym for “missing Authorization header.” A policy scheme or custom handler can make a different choice. I verify the behavior of the schemes I actually deploy instead of building an alert from the label alone. Likewise, success means the handler produced an authentication ticket. Authorization can still deny that principal, so it does not promise a 2xx response. The separate aspnetcore.authentication.challenges counter answers another question: how often was a scheme challenged? Both a none result and a failure result can be followed by a challenge, so challenge count cannot replace the result split. A challenge is an authent

Sarvar's post about discovering a hidden retry in a 5-agent pipeline (one agent taking 22.6s while others took 5s) is a perfect case study in why observability infrastructure matters for agentic systems. Here's what jumped out: Agent-as-black-box is dangerous. When you string together multiple agents, you lose visibility into retry logic, backoff strategies, and cascade failures unless you instrument at the span level. The latency wasn't in the agent logic itself; it was in the retry envelope. Span hierarchy exposes the invisible. Sentry's approach of grouping spans hierarchically made the problem visible at a glance. Without it, you'd see "agent took 22.6s" and assume it was compute-bound. With hierarchy, the retry pattern was obvious. This scales badly across agents. In a 5-agent system, one bad retry strategy can block or cascade. Add error handling, timeout logic, and fallback chains, and you're building a retry forest no one fully understands. The observability debt compounds. The fix is cheap, the insight is priceless. Once Sarvar knew what was happening, tuning retry counts or backoff curves took minutes. The time cost was finding it. Takeaway: If you're building multi-agent systems, instrument early. Span-level observability isn't optional; it's the difference between "it's slow" and "here's why, and here's the fix."

Originally published on hexisteme notes . A status column in one of my agent fleet's ledgers said 30 items were queued to publish. A working session that day stated a backlog close to a month at the fleet's normal rate and deferred the work that keeps posts flowing into the queue. At that moment the ledger showed the same backlog. That exact numeric match suggests — but does not prove — that the ledger informed the decision. The real number of items actually waiting was 6. At one post published per day, that is six days of runway, against a low-water alarm configured to fire at 3. The gap came from a status value that was never advanced after publication, not from the queue-file count itself. A column just quietly stopped meaning what everyone assumed it meant, and by the time it mattered, it had been wrong for a while. The pipeline, briefly The fleet runs a small publishing pipeline: a draft gets written, a promotion step validates it and drops a file into a queue directory, and a scheduled job runs once a day, picks the oldest file in that directory, publishes it, moves the file into a published folder, and appends one line to a log. Alongside the queue directory sits a separate ledger: a flat TSV file, one row per item, with a status column meant to track where each item sits in its life — staged, queued, published. Two different things track the same concept: the files actually sitting in the queue directory, and a column in a table that is supposed to describe them. Where it broke Exactly one piece of code writes status=queued : the promotion step, at the moment an item enters the queue. Nothing else ever changes that value afterward. The daily publish job moves the file and writes to the log; it never opens the ledger. Nobody had assigned any code the job of setting the status forward to published . So queued stopped meaning "currently waiting." It came to mean "was queued at some point," which, once true, is true forever. Every item that had ever passed throu

Use a kill switch inside the checker when your uptime probes start amplifying an incident — one feature flag, read on every tick, that can disable the noisy checks and stop the retries at the source. Reach for tuned backoff and jitter instead when the retry storm stays inside a single process and never fans out onto a dependency somebody else is paging for. Both are cheap to build. Only one of them lets you quiet a polling client while its target is already on fire. I run cron and queue infrastructure, so most of my pages arrive as either "the job didn't run" or "the job ran four times." Health checking sits in the same family of problems: a small, frequent, automated request that multiplies badly when something upstream changes shape. What follows is the runbook I settled on after a fleet of pollers turned a non-incident into a real one — the failure mode, where the switch belongs, the implementation, and how to verify the flip before you walk away from the terminal. What actually turns a polling client's uptime checks into a retry storm? Amplification. A single check is one request every 15 or 30 seconds, which nobody notices; a fleet of checks with retries layered on top is a synchronized load generator pointed at whatever you decided was important enough to monitor. The math is unkind. Take 40 instances, a 5s interval, and 3 retries per failed attempt, and a dependency that normally handles a trickle of health traffic suddenly sees a couple thousand requests a minute — all of them arriving at the exact moment it's least able to absorb them. Retries stack on top of the polling interval rather than replacing it, and because every poller sees the same failure at the same time, they all back off together and return together. The Google SRE book calls out this shape under cascading failures, and the load pattern that comes out of it looks nothing like organic traffic: sawtooth spikes, perfectly aligned, growing until something sheds load. The worst one I've dealt wit

TL;DR: Grafana Agent reached End-of-Life on November 1, 2025 and has been replaced by Grafana Alloy. Alloy consolidates Agent's Static mode, Flow mode, and Kubernetes Operator into a single collector built on the OpenTelemetry Collector while maintaining native support for Prometheus and Loki. If you're using Flow mode, migration is relatively straightforward. If you're using Static mode, the migration process will involve reviewing and testing the converted configuration. Before switching over, verify relabeling rules, recheck resource usage, and confirm that Prometheus and Loki are receiving the same data and labels as before. If you're still running Promtail, it's worth migrating both to Alloy at the same time since Promtail is also End-of-Life. If you deployed Grafana Agent a couple of years ago, there's a good chance you haven't thought about it since. It quietly collects metrics, ships logs, and generally stays out of the way. What you may not realize is that Grafana Agent reached End-of-Life on November 1, 2025. That includes Static mode, Flow mode, and the Kubernetes Operator. Grafana Labs has stopped creating bug fixes, security patches, and official support. If you're still running it, your collection layer is probably still performing normally, but is now unsupported. That doesn't necessarily mean it will stop working tomorrow, plenty of unsupported software continues running for years. It does mean you're taking on the risk yourself, especially as the rest of your monitoring stack continues to evolve. This article covers why Grafana Labs replaced Agent with Alloy, what actually changes during the migration, and where people tend to run into problems. Why Grafana Agent was deprecated One of the biggest issues with Grafana Agent is that it was essentially three agents, not one product: Static mode, which used YAML and looked similar to Prometheus. Flow mode, which introduced a component-based configuration using River. The Kubernetes Operator, which manage

TL;DR If you're a five-person startup shipping a Node.js API and you want a metrics dashboard by Friday, send your telemetry to a managed backend and keep only the instrumentation layer inside your own repo. The alternative — standing up a time-series database, an object store for long-term blocks, and a dashboard service — puts three more components on an on-call rotation that hasn't earned its first SLO yet. Settle the wire format now and treat the backend as a config line you can change later. I own the platform team's roadmap, which in practice means I'm the person who defends the monitoring bill in a budget review and also the person who gets paged when a disk fills at 03:00. Those two jobs pull in opposite directions, and most of the advice online is written by people who only hold one of them. Usually the pager wins the argument. Should a startup run its own metrics stack, or pay for a managed dashboard? Start with capacity, because that's the step everyone skips before signing anything. A moderately instrumented Node.js API — say 40 HTTP routes, two queue workers, default runtime and event-loop metrics, one latency histogram with ten buckets — sits somewhere around 3,000 to 8,000 active series per process. Multiply by replicas. Multiply again by every environment you keep alive, including the staging cluster nobody admits to. You are at 50k active series before a single engineer has written a custom counter, and a self-hosted scraper will chew through that on a 2 GB VM without noticing. It will still be fine at 500k. Past a few million active series you're into sharding, remote storage, and a retention argument with whoever pays for object storage — that's the point where the self-hosted route stops being free and turns into a project with a headcount attached. None of that work is hard. It's just never zero. Dimension Self-hosted stack Managed metrics backend Time to first dashboard 1–3 days under an hour Who owns retention you, plus the storage bill vendor

A status page has a blind spot. It can tell you that your API is returning 502s. It can tell you that a TCP port stopped accepting connections. It can tell you when the incident started. It usually cannot tell you why . Was the application host out of memory? Was disk I/O saturated? Did load climb for 40 minutes before users noticed? Was the server completely healthy and the real problem somewhere else? Those answers often live in a separate monitoring product, disconnected from the incident timeline and disconnected from the status page. That is why I built Servers for StatusPage.me. It is a small, customer-installed host metrics agent and dashboard. You install it on a machine you operate, and it reports CPU, memory, swap, load, disk, and network metrics back to your account. The important part is not “now there are more graphs.” The important part is seeing an outage and the host evidence around it on the same timeline. External checks answer one question. Host metrics answer another. Regular uptime monitoring is still the right tool for the outside-in view: Can users reach the website? Is the API returning the expected response? Does DNS resolve correctly? Is the database port open? Did a scheduled job run? But those checks do not run inside your infrastructure. A healthy HTTP response does not prove that a background worker is about to run out of memory. A timeout does not prove that the app server is overloaded. And an incident can start with a slow disk or growing swap usage long before an endpoint is fully unavailable. The distinction is simple: External monitoring tells you what users can see. Host metrics help explain what the machine was doing when they saw it. You need both. What Servers includes Each registered host gets a dedicated dashboard page with: CPU user, system, and I/O wait utilization Memory use Swap use Load averages Disk use and read/write throughput Network inbound and outbound throughput A human-readable OS description for account owners

If you just want the recommendation: build the uptime poller yourself, put exponential backoff with jitter in front of every health check, and treat a 429 as a scheduling signal instead of an error you swallow. Query-style observability APIs hand you metrics and logs, not threshold rules or notification channels, so the polling worker is the thing that has to decide what "down" means and who gets woken up. That decision is the whole job. I got burned by exactly this. What follows is the pattern that survived the postmortem, the alternatives I weighed before writing a line of it, and the conditions where you should not do any of this yourself. The 429 my retry loop ate for six hours Last spring I was running a homegrown health checker for 40 internal services. One goroutine per service, all driven off the same 15-second ticker, which meant every check landed inside the same 200ms window. The status API we polled had a per-minute quota I'd never bothered to read, and for months it didn't matter, because 40 checks a minute sat comfortably under the ceiling. Then a colleague onboarded 12 more services, we crossed the quota, and the API started answering with HTTP 429. My retry wrapper caught it, retried three times in a tight loop, and on the last attempt returned the previous cached result — which said healthy . It logged the rate limit at debug level. Nobody reads debug. Six hours. Green dashboard. Dead queue consumer. We found out when a customer asked where their export was. The consumer had died on an unrelated deploy, the checker never noticed, and when I finally restarted it the backlog got re-processed on top of a manual replay I'd already run — two customers got the same notification twice. Duplicate deliveries are the specific thing I lose sleep over, and I had caused a batch of them with a retry loop that was trying to be helpful. The postmortem produced one line I now paste into every runbook: a check that can't reach the API reports unknown, never healthy.

Your monitoring catches problems on everything except itself. Here is how an always-firing Watchdog alert plus an external heartbeat check turns silence into a signal, so you find out when your own alerting dies. TL;DR A monitoring system can't reliably monitor its own failure, so use a dead man's switch. Create an always-firing Prometheus Watchdog alert and route it to an independent external heartbeat service. As long as the monitoring pipeline is working, the Watchdog continuously refreshes the heartbeat. If Prometheus, Alertmanager, or the delivery path fails, the heartbeat stops and the external service alerts you through a separate channel. The key is independence: the system responsible for detecting that your monitoring is down must not depend on the monitoring stack itself. One of the traps of creating alerts on a monitoring stack is the hidden assumption that the mechanism evaluating the alert is running properly and has the ability to evaluate it. Prometheus watches your hosts and Alertmanager delivers the warnings. But what watches Prometheus? If something goes wrong and the monitoring stack fails in the middle of the night, no alerts are going out but there is definitely a problem. That is the failure mode that you should be most concerned about, because it is the one your monitoring cannot report on. The fix is an old idea with a grim name: a dead man's switch. A train's dead man's switch stops the train when the operator stops holding it down. The safe state requires continuous positive action, while the absence of that action is what triggers the response. Applied to monitoring, it means building one alert whose silence is itself the alarm. Step one: an alert that always fires This feels backwards the first time you see it, and it took me a little time to get it right. Basically, you create an alert with a condition that is always true, so it fires constantly, forever, on purpose. In the Prometheus world this is conventionally called Watchdog. - aler

If you run more than a handful of OpenTelemetry Collectors, you already know the pain: a config change means SSHing into boxes, redeploying DaemonSets, or babysitting a Git pipeline per cluster, and you never quite trust that every agent is running the config you think it is. OpAMP fixes exactly that. It is a protocol that lets a central server push configuration to a fleet of Collectors, watch their health, and roll changes out in stages, without you touching each host. This post walks through how OpAMP works, the two ways a Collector can speak it, and the config you need to wire one up. The problem OpAMP solves A single Collector is easy. A hundred of them, spread across clusters, VMs, and edge nodes, is a fleet-management problem that has nothing to do with telemetry itself. Every observability team eventually builds some version of the same thing: a way to ship a new pipeline config, confirm it actually applied, and back it out when a processor starts dropping spans. Without a management protocol you end up gluing that together from ConfigMaps, Ansible runs, and dashboards that only tell you an agent is alive, not what config it is actually running. Config drift creeps in. One node keeps an old sampling rate for months because its rollout quietly failed and nobody noticed. OpAMP, the Open Agent Management Protocol, is the OpenTelemetry answer to this. Splunk donated it to the project in 2022, and it has since become the standard control channel for the Collector. It is worth pairing with a clear-eyed view of what a Collector actually is versus lighter agents; the OpenTelemetry Collector vs Grafana Alloy comparison covers that trade-off if you are still choosing a data plane. What OpAMP actually is OpAMP is a client/server network protocol for remote management of large fleets of data-collection agents. It is transport-flexible: agents connect to the server over either plain HTTP or a WebSocket, and the WebSocket path gives you a persistent bidirectional channel

I built an AI agent system that automatically switches to a backup AI model if the main one fails. I connected every step to SigNoz so I could track requests, monitor performance, and detect failures. I also built a diagnostic agent that reads the monitoring data and explains the reason for failures in simple language. During testing, it successfully detected a real AI provider outage and identified the root cause automatically. signoz

"Night gathers, and now my watch begins." The Night's Watch didn't exist to fight wars nobody saw coming — they existed because someone had to actually stand on the Wall and notice when something crossed it. That's the exact problem I kept running into with AI agents, and it's why I built Nights Watch for the "Agents of SigNoz" hackathon: a runtime resilience layer that catches an agent quietly drifting off its plan, explains why, and recovers — automatically. The problem nobody's watching for Most agent failures aren't dramatic. An agent doesn't crash, it doesn't throw an exception, it doesn't get flagged by a content filter. It just... does something slightly different from what it was asked. Told to "find and book a flight under $400," a subtly-drifted agent might reason its way into a $1,200 upgrade and report back "done" — technically true, catastrophically wrong. Nothing in a normal observability stack notices this, because nothing failed . The agent succeeded at the wrong thing. I wanted a system where SigNoz wasn't just a dashboard you check after something breaks — where it actively fed a decision-making loop while the agent was still running . Architecture, in one rule Everything else in the project falls out of one non-negotiable decision I made on day one: rollback state has to be local and durable, never dependent on an external service being reachable. If your resilience system's own safety net depends on a third-party API being up, you haven't built resilience, you've built a second point of failure. So the split looks like this: Local, critical path (SQLite): the Checkpoint Manager. Every agent step writes a durable checkpoint — plan, budget consumed, completed steps — to disk via Node's built-in node:sqlite . Rollback reads from here, always, no exceptions. SigNoz, decision-support only: the Policy Engine queries SigNoz's Query API for prior-run context before scoring severity, and the Explanation Layer calls SigNoz's MCP server to ground its natura

Evidence First, Answer Second: Building an Observable Industrial AI Agent with SigNoz Most AI systems are designed to give an answer. That is useful in a chatbot. On a factory floor, it can be dangerous. While building Industrial IoT Anomaly Control , I kept coming back to one question: What should an AI agent do when it detects a real problem but does not have enough evidence to explain the cause safely? My answer was simple: it should stop, show what it knows, and send the case to a human. This project is a real-time monitoring system for a simulated water-treatment plant. It streams live sensor data from six industrial assets, detects unusual behaviour, searches a knowledge base for similar incidents, and then chooses between two paths: recommend a safe action when the evidence is strong; escalate for human review when the evidence is weak. SigNoz is what makes this decision process visible. Instead of seeing only the final AI response, I can inspect the full path from the incoming sensor reading to anomaly detection, knowledge retrieval, policy checks, agent explanation, and recovery. The problem with traditional alarms Factories already collect large amounts of telemetry such as vibration, temperature, humidity, sequence numbers, and timestamps. The problem is not missing data. The problem is turning that data into a useful decision. A traditional threshold alarm may say: Vibration is above the configured limit. That still leaves the operator with several questions: Is the machine actually failing? Is the sensor or gateway sending bad data? Has this pattern happened before? Is there enough evidence to recommend maintenance? Why did the AI reach this conclusion? Repeated threshold alerts can also create alarm fatigue. If one fault produces dozens of alerts, operators may start treating them as noise. I wanted the system to create one investigation instead of another flood of alarms. What I built The demo represents six assets in a water-treatment plant. A TCP si

I keep running into (and hearing about) a specific kind of bug that never throws an error — an API you depend on quietly changes its response shape. A field disappears. A number becomes a string. Something that was always present is suddenly null. Nothing crashes immediately. It just produces wrong or missing data somewhere downstream, and you find out from a bug report, not a log. I'm curious how common this actually is outside my own experience, so — genuine question, not a pitch: Has this happened to you, with a third-party API or even an internal one your own team owns? How did you find out it happened — a user report, a stack trace somewhere unrelated, manual debugging? Do you currently do anything to catch this kind of thing before it bites you (contract tests, monitoring, or just... hoping)? If you don't do anything about it today, is that because it's not painful enough to bother, or because you just haven't found a lightweight way to? Not selling anything here, just trying to understand how real and how painful this actually is for people building on top of APIs day to day. Would genuinely appreciate hearing your experience, even a one-line "yeah this happened to me once, wasn't a big deal" is useful data.