How I Built Triage: Turning SigNoz into a Blue Team SOC (And the Deployment Nightmares I Survived)

Most people use OpenTelemetry and SigNoz to watch their CPU usage, find memory leaks, or figure out why their API is taking 400ms instead of 200ms. But for the Agents of SigNoz Hackathon (Track 3: Observe Anything Weird), I wanted to do something completely different. I didn't want to watch hardware. I wanted to watch hackers. That is how Triage was born. It is an OpenTelemetry-powered Blue Team SOC (Security Operations Center) that tracks active cyber attacks instead of just generic application performance. The Original Vision vs. The Reality Building this sounded straightforward on paper: catch bad traffic, wrap it in an OpenTelemetry span, send it to SigNoz, and show it on a custom dashboard. But actually deploying this beast before the deadline was a completely different story. If you have ever tried to deploy a full-stack Next.js app, a Python honeypot, and an OTel pipeline while the clock is ticking, you know exactly what kind of panic I am talking about. Here is what actually happened behind the scenes. 1. The Azure VM & Docker Crash I started by spinning up an Azure virtual machine to self-host the SigNoz backend. I pulled the repo, ran docker compose up -d, and immediately watched my server completely freeze. Turns out, my free tier Azure instance (Standard_B2ats_v2) only had 1 GiB of RAM. You simply cannot run a massive ClickHouse database and a full OTel collector on 1GB of memory without it crashing instantly. I had to pivot fast and rely on the cloud endpoints. 2. The Vercel vs Localhost Trap Once I got the backend running, I hit my next wall. My Vercel Next.js dashboard kept throwing 500 Internal Server Error on the threat simulation API, and the SigNoz connection kept reading OFFLINE (fetch failed). I was staring at my logs losing my mind until it clicked. My Vercel environment variables had SIGNOZ_API_URL set to http://localhost:8080 and my Python script was looking for OTel on localhost:4318. Note to self (and everyone else): Vercel is a cloud serve