A Dead-Man's Switch That Pages Once and Goes Quiet Is Worse Than None. Ours Went Silent for 43 Days.

Most monitoring watches for something bad to appear: a 500, a timeout, an expired certificate, a slow response. A heartbeat monitor does the opposite. It watches for something good to stop appearing . Your cron runs, your backup completes, your embedded device phones home, your queue worker drains — and each of those pings a URL to say "I'm still alive." The monitor's job is to notice when the pings go quiet. That inversion is the entire value. A cron that fails throws an error you can catch. A cron that stops being scheduled — the box got reimaged, the systemd timer got disabled, the container never came back after a deploy, the account got suspended for an unrelated billing issue — throws nothing at all. There is no log line, no exception, no non-zero exit. There is only the absence of the thing that used to happen. You cannot alert on an event that does not fire. You can only alert on the silence. So heartbeat monitoring looks trivial: store a timestamp on every ping, and if now - last_seen > expected_interval , fire an alert. It is about ten lines. And it is exactly those ten lines that will let 43 days of downtime pass without a second word — because the hard part of a dead-man's switch is not detecting the death. It is staying loud after it. I know because it happened to our own. Three states, and why the third one must stay silent Start with the check itself. A naive heartbeat has two states — alive or dead — and both are wrong at the edges. The real answer set has three: alive — a beat arrived within period + grace . Everything is fine. dead — the last beat is older than period + grace . The thing stopped. Page someone. unknown — the monitor exists but has never received a single beat. That third state is where two-state heartbeat monitors self-immolate. A brand-new heartbeat you just created has no last_seen timestamp. If your rule is "alert when last_seen is too old," a null last_seen is infinitely old, so the monitor pages you the instant you create it —