Beyond the Happy Path: Lessons in Resilience and Distributed State

Reflecting on two major technical challenges from my backend engineering internship, focusing on fault tolerance, infrastructure, and distributed architectures. Introduction As I wrap up my HNG internship, I’ve been reflecting on the gap between code that "works on my machine" and code that survives in production. Here is a look at two tasks from Stage 9—one solo, one team-based—that completely changed how I approach backend engineering and infrastructure. The Individual Task: Background Job Scheduler What it was For my individual Stage 9 task, I built a distributed background job scheduler backed by PostgreSQL and a FastAPI backend, featuring a vanilla HTML/CSS/JS frontend. It manages async tasks (like a mock email sending queue) using a MinHeap priority queue, Directed Acyclic Graph (DAG) dependency resolution, a Dead-Letter Queue (DLQ), and a real-time Server-Sent Events (SSE) dashboard. The problem it was solving Heavy asynchronous tasks—like email generation or batch processing—cannot block the main API thread. The system needed to successfully queue, prioritize, retry on failure, and track every job entirely independently from the standard request-response cycle. How I approached it I built the core logic from the ground up: a MinHeap and an alternative Timing Wheel algorithm for scheduling, a worker engine featuring a 3-attempt backoff sequence (1s, 5s, 25s with jitter), a DAG dependency checker, and a starvation daemon to prevent tasks from hanging. Once the CRUD API and SSE streaming were hooked up, I containerized the entire application with Docker and wrote my deploy scripts. I thought I was done. What actually broke and how I fixed it The application code took hours. The deployment took a full day of non-stop debugging across multiple cloud providers. Oracle Cloud was out of capacity on every free tier shape, and GCP demanded upfront payment. I finally got a t3.micro running on AWS, but that’s when the real DevOps nightmare began: The SSL Chicken-and-Egg