The Anti-Bot Detection Checklist I Use Before Every Scraping Project
The Anti-Bot Detection Checklist I Use Before Every Scraping Project Every scraping project I take on starts with this checklist. Not because I'm paranoid — but because I've learned the hard way that production scrapers fail silently. They return 200 OK with garbage data, or they get rate-limited so gradually you don't notice for days. This is the systematic approach I've refined over 50+ scraping projects. Pre-Scraping: Know Your Target 1. Identify the CDN and Protection Stack Before writing a single line of code, check what you're up against: # Check CDN and headers curl -I https://target-site.com # Look for these common protection headers: # X-Engine: akamai-html-protection # X-Served-By: DataDome # cf-ray: Cloudflare # X-Bot-Status: blocked Common protection platforms: Cloudflare → Look for cf-ray and __cfduid cookies DataDome → Look for datadome in headers or scripts PerimeterX → Look for _pxff cookies Akamai → Look for akamai-html-protection headers 2. Check Robots.txt Respectfully curl https://target-site.com/robots.txt | grep -v "^#" Don't take this as gospel — but it's a good signal. If they explicitly disallow your use case, that's a flag. 3. Map the Site's JavaScript Rendering Some sites are fully static (fast, easy). Others render everything with JavaScript (need Playwright/Puppeteer). Check: // Quick check - fetch raw HTML vs rendered content // If they differ significantly, you need JS rendering const https = require ( ' https ' ); const html = await fetch ( ' https://target.com ' ). then ( r => r . text ()); const hasAngularVueReact = /ng-app|vue|react|__NEXT_DATA__/i . test ( html ); console . log ( ' Needs JS rendering: ' , hasAngularVueReact ); Code-Time: Defensive Patterns 4. Rotate User Agents const USER_AGENTS = [ ' Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 Chrome/120 Safari ' , ' Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Edge/120 ' , ' Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 Chro