How to Give AI Better Evidence: Lessons From a Security Investigation That Almost Failed
Category: My AI Experiments There's a mental model most people use when working with AI: describe your problem, get a solution. It works well enough, until it doesn't. And when it fails, the failure is invisible — because AI doesn't say "I don't have enough to go on." It gives you a confident, well-reasoned, completely wrong answer. I learned this the hard way during a website security investigation. The AI and I ran a thorough analysis, reached a clear conclusion, and were wrong. Not because the AI was weak — because I gave it the wrong kind of input. When I changed the input, the same AI found the answer in seconds. That gap — between the input that produces a wrong answer and the input that produces a right one — is what I want to talk about. The Investigation That Almost Failed My website was secretly redirecting visitors to a virus site. The attack was sophisticated: it only targeted specific browsers, fired at most once per device per day using a cookie-based cooldown, and left no trace in any file. I asked AI to help investigate. I described the symptoms. We searched through files together — .htaccess , theme functions, plugin code. Everything looked clean. The AI identified the most suspicious external element in scope: a Chinese analytics script called 51.la. I removed it. The redirect stopped. I called it solved. Three weeks later, the identical attack appeared on another site I manage. No 51.la anywhere. This time, instead of describing the symptoms, I gave the AI something different: the actual rendered HTML of an affected page, fetched using the exact browser User-Agent and IP type that triggered the attack. The AI found an 83KB malicious JavaScript payload injected into every page. Inside it: a WeChat browser detector, a link-click hijacker, a cookie-based daily cooldown. The payload was stored in the WordPress database — in plugin configuration data — where no file-level search could ever find it. Same AI. Same type of problem. Completely different ou