How to Build a Competitor Intelligence Agent with CrewAI and ZenRows

A competitor intelligence agent enables real-time pricing visibility, automated positioning tracking, and structured competitor briefs for brands without manual research. At the center of these workflows are a researcher agent responsible for gathering data and an analysis agent responsible for generating a summary and a downstream report. However, one of the things that makes the researcher agent's output trustworthy is its retrieval layer, since a poor retrieval-layer output can make the analysis agent's recommendation questionable. If the researcher agent searches the webpage and retrieves a challenge page, an empty response, or blocked content, every downstream conclusion becomes less trustworthy. With a 99.93% success rate on protected websites, ZenRows provides the reliable retrieval layer that makes these workflows practical in production. This tutorial shows why a CrewAI researcher agent can fail on protected competitor pages. It starts with a custom ZenRows-based tool setup, then later shows the MCP server as an alternative approach. Prerequisites This tutorial works best with Python 3.10 or newer. If you are on an older Python version, create a dedicated virtual environment with a current Python installation to avoid dependency conflicts. Python 3.10 to 3.13. CrewAI requires this range. ZenRows API key. Create an account at zenrows.com and copy your key from the dashboard. This key authenticates every scrape the researcher agent runs for data extraction. Anthropic API key. The crew uses Claude to drive both agents. Generate a key in the Anthropic Console. Install dependencies and the required packages using pip install "crewai[anthropic]" crewai-tools zenrows python-dotenv . Create a .env file and save your API keys there. Why the built-in ScrapeWebsiteTool fails on competitor pages The problem, as established earlier, starts before the analysis. A CrewAI workflow depends primarily on the information collected, because a competitor intelligence agent relie