We’re Giving AI Agents More Tools. What Happens When the Boundaries Fail?

📌 TL;DR AI agents are becoming useful because we're giving them the ability to do more than just answer questions. They can run commands, browse the web, use APIs, read and modify files, install packages, and interact with other systems. But the more an agent can do, the more the boundaries around it matter. I started thinking about this after reading Anthropic's July 30 report about three incidents discovered during its cybersecurity evaluations. Claude models were supposed to be working inside simulated environments and were explicitly told they had no internet access. Except internet access was actually available because of a problem with how the evaluation environment was configured. While trying to complete their assigned cybersecurity exercises, the models reached real systems and initially treated them as part of the simulation. In one incident, a Claude model even published a malicious Python package to the real PyPI registry while believing it was still operating inside the exercise. This came shortly after a separate OpenAI incident involving Hugging Face. The two stories might sound similar at first, but the models reached the real internet in importantly different ways. And that brings this back to a pretty familiar software engineering idea: A prompt is not a security boundary. Telling an agent “you don't have internet access” isn't the same as actually removing internet access. Telling it “only use these files” isn't the same as restricting its permissions to those files. The model is also only one part of the system. The tools we connect, the permissions and credentials we give it, the environment it runs in, and the monitoring and safeguards around it can all affect what happens. So when something goes wrong, I don't think it's enough to stop at “the AI did it.” The model's behavior matters, but so do the systems and boundaries we build around it. As we give agents more ability to act, we also have to be thoughtful about what we're actually allowing