I Just Wanted to Scrape One Page. Why Did I Write 50 Lines of Puppeteer?

Last Friday at 4:30 PM, my product manager walked over: "Hey, can you grab the titles from the Hacker News homepage and send me an Excel file?" I thought: That's it? Five minutes tops. Two hours later, I was still debugging CSS selectors. How Things Spiraled Out of Control Step 1: Initialize the Project mkdir hacker-news-scraper && cd hacker-news-scraper npm init -y npm install puppeteer Hit enter, waited three minutes. Puppeteer needs to download a full Chromium browser — over 200 MB. I stared at the progress bar and started questioning my life choices. Step 2: Write the Code "It's just a document.querySelectorAll , right?" That's what I thought. Then I opened my editor: const puppeteer = require ( ' puppeteer ' ); ( async () => { const browser = await puppeteer . launch ({ headless : true , args : [ ' --no-sandbox ' , ' --disable-setuid-sandbox ' ] }); const page = await browser . newPage (); try { await page . goto ( ' https://news.ycombinator.com ' , { waitUntil : ' networkidle2 ' , timeout : 30000 }); await page . waitForSelector ( ' .titleline > a ' , { timeout : 10000 }); const titles = await page . evaluate (() => { const items = document . querySelectorAll ( ' .titleline > a ' ); return Array . from ( items ). map ( el => ({ title : el . textContent , url : el . href })); }); console . log ( JSON . stringify ( titles , null , 2 )); } catch ( err ) { console . error ( ' Scraping failed: ' , err . message ); } finally { await browser . close (); } })(); I counted: 27 lines. And this is the minimal version — no User-Agent spoofing, no retry logic, no proxy support, no concurrency control. Add all of that and you're well past 50 lines. Step 3: Run It node index.js Error: Navigation timeout of 30000 ms exceeded . Switched to domcontentloaded , got past that. But then waitForSelector timed out — because .titleline was a relatively new class name. Hacker News had silently changed it from .storylink at some point, and nobody sent me the memo. Step 4: Debug Set head