Observability Practices: A Hands-On Guide with Prometheus and Grafana

Introduction Modern software systems are distributed, complex, and constantly changing. When something breaks in production, you need answers fast. That's where observability comes in. Observability is the ability to understand the internal state of a system purely from its external outputs — without needing to redeploy, add debug code, or guess. It goes beyond traditional monitoring, which only tells you whether something is wrong. Observability tells you why it's wrong, where it started, and how it's spreading. In this article, we'll explore the three pillars of observability, set up a real Node.js API instrumented with Prometheus and Grafana , and walk through how to detect and diagnose a real-world issue using the data we collect. The Three Pillars of Observability 1. Logs Logs are discrete, timestamped records of events that happened in your system. They're the most familiar form of observability — every developer has done console.log debugging at some point. Example: [2026-07-02T10:34:21Z] INFO User 4821 logged in from IP 192.168.1.10 [2026-07-02T10:34:25Z] ERROR Failed to process payment for order #9932: timeout Logs are great for capturing specific events, errors, and context. But they can become expensive at scale and hard to query across millions of lines. 2. Metrics Metrics are numeric measurements collected over time. Unlike logs, they're aggregated and efficient to store and query. Common examples: HTTP request count per minute p95 response latency CPU and memory usage Error rate per endpoint Metrics are the backbone of dashboards and alerts. 3. Traces Traces follow a single request as it travels across multiple services. In a microservices architecture, a user request might touch 5–10 services. A trace shows you exactly where time was spent and where failures occurred. Tools like Jaeger , Zipkin , and OpenTelemetry handle distributed tracing. Why Prometheus and Grafana? There are many observability platforms out there: Datadog, New Relic, Dynatrace, Az