📌 TL;DR
AI agents are becoming useful because we're giving them the ability to do more than just answer questions. They can run commands, browse the web, use APIs, read and modify files, install packages, and interact with other systems.
But the more an agent can do, the more the boundaries around it matter.
I started thinking about this after reading Anthropic's July 30 report about three incidents discovered during its cybersecurity evaluations. Claude models were supposed to be working inside simulated environments and were explicitly told they had no internet access.
Except internet access was actually available because of a problem with how the evaluation environment was configured.
While trying to complete their assigned cybersecurity exercises, the models reached real systems and initially treated them as part of the simulation. In one incident, a Claude model even published a malicious Python package to the real PyPI registry while believing it was still operating inside the exercise.
This came shortly after a separate OpenAI incident involving Hugging Face. The two stories might sound similar at first, but the models reached the real internet in importantly different ways.
And that brings this back to a pretty familiar software engineering idea:
A prompt is not a security boundary.
Telling an agent “you don't have internet access” isn't the same as actually removing internet access. Telling it “only use these files” isn't the same as restricting its permissions to those files.
The model is also only one part of the system. The tools we connect, the permissions and credentials we give it, the environment it runs in, and the monitoring and safeguards around it can all affect what happens.
So when something goes wrong, I don't think it's enough to stop at “the AI did it.” The model's behavior matters, but so do the systems and boundaries we build around it. As we give agents more ability to act, we also have to be thoughtful about what we're actually allowing them to do.
I'm not a cybersecurity researcher. I just enjoy reading research and technical reports like these and trying to understand what we, as developers, can learn from them. That's what I wanted to explore here.
No “AI escaped and we're doomed” take 😄 Just an interesting real-world incident and some surprisingly familiar engineering lessons.
Table of Contents
- The Report That Sent Me Down This Rabbit Hole
- Before We Get Into What Happened...
- So... What Actually Happened?
- The PyPI Incident Is Where It Got Really Interesting
- When the Instructions and Reality Don't Match
- A Prompt Is Not a Security Boundary
- Wait, Wasn't There an OpenAI Incident Too?
- What Does Any of This Mean for Developers?
- What I'm NOT Taking Away From This
- The Bigger Question
- Final Thoughts
- I'd Love to Hear Your Thoughts
- Sources
- 🤝 Let's Stay Connected
The Report That Sent Me Down This Rabbit Hole
On July 31, I was reading Anthropic's newly published report about three incidents it had discovered during cybersecurity evaluations.
I enjoy reading reports like this every now and then. It's something I became much more interested in while working on a survey paper during my master's, and even now, I occasionally find myself opening a technical report out of curiosity and suddenly realizing I've spent much longer reading it than I planned 😄
This was definitely one of those times.
The report started with a pretty surprising finding. Anthropic had reviewed 141,006 cybersecurity evaluation runs and found three incidents where Claude reached the internet and gained unauthorized access to real systems belonging to three different organizations.
That sentence alone sounds alarming, but I kept reading, and the details made the story much more interesting than the headline.
This wasn't simply a case of “Claude was told not to access the internet and decided to do it anyway.” And it wasn't “Claude suddenly decided to hack random companies.” The models were participating in cybersecurity exercises where their job was essentially to find and exploit weaknesses in order to retrieve a hidden piece of information.
They were told the environment was simulated, and they were also told something else that becomes extremely important to everything that happened next: they had no internet access.
Except that assumption didn't match the environment they were actually operating in.
And that's what really sent me down the rabbit hole. I wanted to understand how an AI model participating in a simulated cybersecurity exercise could end up interacting with real systems while still believing it was completing the task it had been given.
The more I read, the more this started to feel like a much broader question than just one unusual cybersecurity incident.
Before We Get Into What Happened...
Most of us have probably come across AI agents by now, so I won't spend too much time explaining what they are. The part that matters for this story is that an agent doesn't necessarily stop at generating an answer. Give it the right tools and it can take actions too, whether that's reading and modifying files, running commands, calling APIs, browsing the web, or interacting with other systems.
That's what makes agents so useful, but it's also what makes the boundaries around them important. The more tools and access we give an agent, the more we have to think about what it can actually reach and what it's allowed to do.
And that's exactly where this story gets interesting, because Claude was given a task, an environment, and an understanding of where those boundaries were.
The problem was that the real boundaries didn't quite match what it had been told.
So... What Actually Happened?
Anthropic regularly runs cybersecurity evaluations to understand what its models are capable of. Some of these involve capture-the-flag challenges, or CTFs, where a model is placed in a controlled environment and given a target, usually some hidden piece of information called a “flag,” to find.
If you've never come across a CTF before, the basic idea is pretty simple. The environment is intentionally designed to be attacked. Finding the flag might involve discovering a vulnerability, finding credentials, or getting access to another machine. It's a way of testing cybersecurity capabilities without putting real systems at risk.
Claude was participating in exercises like these, and there were two details about the setup that become really important to what happened next. The models were told that they were operating inside a simulation and that they didn't have internet access.
So from the model's perspective, anything it could reach should have been part of that simulation.
The problem was that this wasn't actually true.
According to Anthropic, because of a misunderstanding between the company and its third-party evaluation partner, some of the evaluation machines had access to the real internet when they weren't supposed to.
That created a pretty unusual situation. Claude had been given a task where finding and exploiting weaknesses was part of the exercise. It had also been explicitly told that the internet wasn't available. So when it managed to reach systems outside the intended environment, it initially treated them as another part of the challenge.
Except some of those systems were real.
Anthropic's review eventually found three separate incidents involving three different Claude models. The circumstances were different in each case, and the models didn't all respond the same way once there were signs that they might be interacting with real systems.
But one of those incidents in particular is where this story gets really interesting.
The PyPI Incident Is Where It Got Really Interesting
If you've worked with Python, you've probably typed something like pip install something before. That package often comes from PyPI, the public Python Package Index.
During one of the evaluations, Claude found what appeared to be setup instructions for developers at a fictional company. The instructions told employees to install a particular Python package, but that package didn't actually exist.
Claude realized that if it created a package with the same name, the fictional company's systems might download and run its version instead. Within the context of a cybersecurity challenge, that gave it a possible way to reach its target.
So the model created a malicious package and tried to publish it.
The problem was that it wasn't interacting with a fake version of PyPI created for the evaluation. It was interacting with the real PyPI.
What happened next is probably the part of the report that surprised me most. To publish the package, Claude needed to create an account, which meant it needed an email address and, at one point, a phone number. It tried several different approaches, including unsuccessfully looking for ways to obtain a phone number, before eventually finding a free email provider it could use to register an account.
It then successfully uploaded the malicious package to PyPI.
According to Anthropic, the package remained publicly available for roughly an hour and was downloaded and run on 15 real systems during that time. One of those systems belonged to a security company that automatically installed Python packages to scan them for malware. When the scanner installed the package, Claude's code executed and exposed credentials, which the model then used to access additional infrastructure belonging to that company.
There's an important detail here, though. Claude wasn't intentionally targeting that security company. Anthropic says the model didn't even know the company existed when it uploaded the package. The company was affected because its scanner happened to install something Claude had unknowingly published to the real internet.
And somehow, the story gets even more interesting.
Earlier in the process, Claude had actually considered the possibility that it might be interacting with the real internet. Its own reasoning recognized that if this were real, publishing the package would amount to a real-world attack and wouldn't be okay.
You might expect that realization to be where the attempt stopped, but it didn't. The model instead found reasons to convince itself that it was still inside the simulation. One of the things it used as evidence was the system date: 2026. It apparently interpreted that as another sign that the environment had been staged.
I have to admit, reading that particular detail in 2026 gave me a small “well...” moment 😅
It's a slightly funny detail in an otherwise serious incident, but I think there's something important underneath it. The model encountered information suggesting that its understanding of the environment might be wrong, yet still ended up reasoning its way back to the assumption it had started with: that this was all part of the simulation.
And that's the part of this incident I kept thinking about afterward.
When the Instructions and Reality Don't Match
After reading about the PyPI incident, one question seems pretty obvious: why didn't Claude realize it had reached the real internet?
But the more I thought about it, the more I started looking at that question from the other side.
How would it know?
The model had explicitly been told that it didn't have internet access. It had also been told that it was participating in a simulation where finding and exploiting weaknesses was part of the task. So when it encountered something that looked real, it was trying to make sense of that new information using what it had already been told about its environment.
There was a mismatch between those two things. The model was operating with something like:
Simulation → No internet → Everything reachable should be part of the exercise
But the environment actually looked more like:
Simulation → Internet available → Real systems reachable
That difference might look small on paper, but it completely changes the situation.
Of course, that doesn't explain away everything the models did. Anthropic itself points to moments where their behavior fell short of what it would want, including cases where signs appeared that the systems might actually be real.
But I think the mismatch still matters.
The models weren't making decisions in the environment everyone thought they were operating in. They were making decisions based partly on an understanding of that environment that turned out to be wrong.
And that's where this stopped feeling like something that only matters to people running cybersecurity evaluations.
Because once we start giving AI agents tools and the ability to take real actions, there's a bigger question underneath all of this:
What happens when the boundaries an agent has been told about aren't the boundaries that actually exist?
That leads to probably the simplest engineering lesson I took away from this whole story:
A prompt is not a security boundary.
A Prompt Is Not a Security Boundary
This is probably the simplest way I can explain what I took away from all of this: telling an agent not to do something and actually preventing it from doing that thing are not the same.
Imagine I'm building an AI coding agent and tell it to only access files inside /my-project. If the process running that agent still has permission to read my entire computer, I've given it an instruction about where it should go, but I haven't actually restricted where it can go.
The same applies to internet access. I can put “Don't access the internet” in the instructions, but if the environment still allows unrestricted network access, that sentence isn't what prevents the agent from reaching the internet.
And when I thought about it this way, it started to feel like a very familiar software engineering idea.
We don't build an admin dashboard and simply write, “Please only use this page if you're an administrator.” We implement authentication and authorization so that someone without the right access can't use it in the first place.
We don't give every application unrestricted access to a database and then ask it nicely to stay away from the tables it doesn't need. We use permissions to control what it can actually access.
The same thinking makes sense for AI agents too.
Prompts still matter. They tell the model how we want it to behave, what its task is, and what it should or shouldn't do. But when something is important enough that it must not happen, I don't think the prompt should have to carry that responsibility alone.
The system around the model matters too: what tools we've connected, what permissions those tools have, what credentials are available, whether the network is restricted, and what happens if one of our assumptions about the environment turns out to be wrong.
That's the distinction I keep coming back to:
A prompt can guide behavior. Infrastructure can limit capability.
And as AI moves from mostly giving us answers to actually taking actions, I think that difference becomes much more important.
Wait, Wasn't There an OpenAI Incident Too?
Yes, and the timing isn't a coincidence. Anthropic says it began its large-scale review after OpenAI disclosed a separate cybersecurity evaluation incident involving Hugging Face on July 21.
When I first read about both incidents, I wondered if essentially the same thing had happened twice. But once I looked into the details, there was an important difference in how the models reached the real internet.
In OpenAI's case, the models were running inside an evaluation environment where direct internet access was restricted. According to OpenAI, the models found and exploited a previously unknown vulnerability in Artifactory, software that was being used as a proxy and cache for package registries. This kind of previously unknown vulnerability is commonly called a zero-day.
After exploiting it, the models were able to move through parts of OpenAI's research environment until they reached a system with internet access. From there, they eventually compromised Hugging Face's infrastructure while trying to obtain solutions for the cybersecurity evaluation.
So, very roughly, the OpenAI incident looked like this:
Restricted environment → Vulnerability discovered and exploited → Internet access reached → Hugging Face compromised
Anthropic's incidents were different:
Evaluation environment → Internet unintentionally available → Real systems reachable → Models initially treated them as part of the simulation
That difference matters.
In the OpenAI incident, the models found a way through a boundary that was supposed to restrict internet access. In Anthropic's case, that boundary wasn't actually there in the way everyone thought it was. Internet access was already unintentionally available.
Both incidents raise important questions about how increasingly capable AI systems should be tested, but I don't think putting them both under a headline like “AI escaped again” tells us very much.
If anything, reading the details made me think the engineering differences are the more useful part of the story. Two incidents can look very similar from the outside while pointing to different things that went wrong underneath.
What Does Any of This Mean for Developers?
Most of us aren't running cybersecurity evaluations on frontier AI models, so you might reasonably be thinking:
Interesting story, Hema... but what am I supposed to do with this? 😄
I had the same question while reading the reports. And the more I thought about it, the more I realized that some of the lessons aren't limited to large AI labs at all. They apply to much smaller agents and AI-powered tools we might be building ourselves.
1. Give an Agent What It Needs, Not Everything You Have
If an agent only needs to read a few files, does it really need access to the entire filesystem? If it only needs to read from a database, does it need credentials that can also modify or delete data? And if it only needs one API, there's probably no reason to expose several others just because they're available.
This is really the familiar security principle of least privilege: give something only the access it needs to do its job.
And that's not because we should assume every agent is going to do something malicious. Mistakes happen. Instructions can be misunderstood. Our own configurations can be wrong. An agent can also encounter a situation we simply didn't anticipate.
Limiting access doesn't prevent every possible problem, but it can limit how far a problem can go.
2. Think About What Happens When Your Assumptions Are Wrong
One thing I found especially interesting here is that the problem wasn't only what the models assumed. The people running the evaluation had assumptions too.
The models had been told there was no internet access because that was how the environment was supposed to work. But the actual configuration didn't match that expectation.
I think there's a useful developer lesson in that. When we're building an agent, it's easy to think in terms of how the system should behave: this tool can only do this, these credentials only reach that service, this environment can't access the internet.
But how often do we actually verify those assumptions?
If something really matters, testing the boundary itself might be just as important as defining it. Because sometimes the unexpected behavior isn't caused by the agent ignoring the system we designed.
Sometimes the system isn't quite the one we thought we'd designed.
3. If an Agent Can Act, We Need to Know What It's Doing
We already rely on logs when ordinary software behaves unexpectedly. I think that becomes even more important when an agent can take a series of actions on its own.
If something goes wrong, we may need to understand which tools it used, what it accessed, what requests it made, where something failed, and what it tried afterward.
Anthropic found these incidents by going back through its evaluation transcripts. Without records of what the models had actually done, understanding that sequence would have been much harder.
So as agents become capable of doing more, observability isn't just useful for debugging. It can also help us understand whether the system is behaving within the boundaries we expected.
4. One Safeguard Probably Isn't Enough
Anthropic talks about defense in depth in its response. It sounds very cybersecurity-ish, but the idea itself is pretty simple: don't depend on one thing going right.
Maybe an agent isn't supposed to have internet access. That's one boundary. But what if something is misconfigured and internet access becomes available anyway? Network restrictions could still limit where it can connect. Credentials with minimal permissions could limit what it can access. Monitoring could help detect unexpected activity.
Each one gives you another layer.
The goal isn't to build one perfect wall and assume it will never fail. It's to have other protections waiting if that wall does.
5. The AI Model Isn't the Whole AI System
This might actually be the biggest thing I've taken away from reading about these incidents.
When something goes wrong and we say “the AI did this,” it's very easy to picture the model as if it were the entire system. And I think it's also very easy to put all the blame there and stop looking.
But an AI agent is operating inside something much bigger:
model + prompt + tools + credentials + permissions + network + environment + monitoring
Change one of those pieces and you can change what the agent is capable of doing, even if the model itself stays exactly the same.
The Anthropic incidents are a good example. The models were told they didn't have internet access. But the environment didn't actually enforce that assumption, and suddenly the same task could lead somewhere it was never supposed to lead.
That doesn't mean we shouldn't examine what the models did. We absolutely should. There were moments in these incidents where the models' own decisions raise important questions too, especially when they encountered signs that the systems might be real.
But I don't think the conversation can end with “the AI did something it wasn't supposed to do.”
We build the environments these agents operate in. We choose which tools to connect. We decide what permissions and credentials they receive. We create the boundaries, and we have to make sure those boundaries actually exist.
I'm not saying that to move the blame somewhere else. I think that's the wrong way to look at it too.
For me, it's more about recognizing that we're part of this system as well. As we give AI more ability to act, we also take on the responsibility of being careful about the systems we're building around it.
That's why, when something goes wrong, I think understanding the whole system is much more useful than finding the easiest thing to blame.
What I'm NOT Taking Away From This
I think it's worth being a little careful with the conclusions here, because this is exactly the kind of story where the most dramatic version can easily become the one everyone remembers.
I'm not taking away that Claude suddenly developed its own goal and decided to attack companies. Anthropic says it found no evidence of the models pursuing a goal of their own. They were still working toward the capture-the-flag tasks they had been given, even though some of those actions ended up affecting real systems.
I'm also not describing these incidents as Claude deliberately “escaping” its evaluation environment. That's not what Anthropic says happened. The models were able to reach the internet because an access path was unintentionally available, while they had been told that no internet access existed.
There's another detail I think we should be careful with too. Of the three models involved, Anthropic's newest internal research model eventually recognized that it was interacting with a real system and stopped. That's certainly interesting, especially because the other models behaved differently, but three incidents involving different models and different circumstances aren't enough to conclude that newer models are simply safer. Anthropic is careful about making that conclusion too.
So I'm not reading these incidents as proof that AI “went rogue,” and I'm also not reading them as proof that the problem is solved because one model eventually stopped.
That's why I think it's worth being careful about what we conclude from these incidents while the investigations are still developing.
The Bigger Question
I think this is where the story goes beyond Anthropic, OpenAI, or even cybersecurity evaluations.
We're slowly moving from AI that mostly tells us, “Here's what you could do,” toward AI that can increasingly say, “I'll do it.” And honestly, I find that transition fascinating.
An AI assistant that suggests a terminal command is one thing. An agent that can actually execute that command is another. An assistant that drafts an email for you is one thing. An agent that can send it is another. And an assistant that suggests a database query is very different from an agent holding credentials that can actually run that query against a production database.
In each case, we're giving the model something it didn't have before: the ability to turn an answer into an action.
And with that comes access.
The more tools we connect to an agent, the more of the world around it becomes something it can interact with. A terminal gives it access to commands and whatever that environment can reach. An API gives it whatever permissions come with that API key. File access depends on which files the process can actually reach. Even something as ordinary as sending an email becomes a real action once the agent has permission to do it.
None of this means I think we shouldn't build agents. Quite the opposite. The fact that AI can increasingly do things instead of only telling us how to do them is one of the reasons I find this space so interesting.
But I think that also means we have to be more thoughtful about what comes with each new tool we connect. Every terminal, API, database, browser, or filesystem we give an agent access to expands the world it can interact with.
Maybe one of the questions worth asking before connecting the next tool is simply:
What happens if the agent uses this in a way I didn't expect?
Not because we should assume the agent is going to do something malicious. That's not really the point I'm trying to make.
It might misunderstand what we meant. It might encounter something we didn't expect. Our own configuration might be wrong. Or, as the Anthropic incidents showed, the environment itself might not work the way everyone believed it did.
And the more access we've given the agent, the more those unexpected situations can matter.
That's why I don't think the interesting question going forward is only how capable can we make these agents?
It's also how carefully can we build the systems around those capabilities?
Final Thoughts
I started reading Anthropic's report on July 31 because I was curious about what had happened. I ended up thinking much more broadly about how we're building AI agents and, especially, about the boundaries we put around them.
And strangely, a lot of what I took away from it doesn't feel particularly futuristic. Limit access. Verify assumptions. Monitor what's happening. Don't depend on a single safeguard.
These are ideas we've had in software and security for a long time.
What's changing is what we're connecting to AI.
We're giving agents terminals, browsers, APIs, files, credentials, and the ability to turn an answer into an action. That's incredibly exciting, but it also means the environment around the model becomes part of what we're responsible for building carefully.
So maybe the question isn't only:
What can this agent do?
We should probably be asking just as often:
What should this agent actually be allowed to do?
Because if there's one thing I'll remember from this whole rabbit hole, it's this:
The boundary we describe to an agent and the boundary that actually exists aren't necessarily the same thing.
And as exciting as it is to see what AI agents can increasingly do, I think we have to be just as thoughtful about the world we're giving them access to.
I'd Love to Hear Your Thoughts
This was one of those rabbit holes where I started with one question and ended up thinking about something much bigger. So I'd really love to know what you took away from it too.
You don't have to be working in cybersecurity or building AI agents to join the conversation. Even if this is your first time reading about incidents like these, did anything change the way you think about AI agents and the access we're giving them?
And if you are experimenting with agents or giving models access to tools, I'm especially curious about this:
Where do you think the real boundary should live: in the prompt, the permissions, the environment, or some combination of all three?
Or maybe you came away with a completely different question or perspective after reading this. I'd love to hear that too.
Sources
I based this article primarily on the companies' own incident disclosures. Both investigations are still developing, so some of what we know may change as more information becomes available.
- Anthropic: Investigating three real-world incidents in our cybersecurity evaluations - published July 30, 2026.
- OpenAI: OpenAI and Hugging Face partner to address security incident during model evaluation - published July 21, 2026, with additional updates on July 28 and July 29.
Anthropic has said it plans to release a lightly redacted transcript from the PyPI incident, while OpenAI says a more detailed technical report will follow.
I'll definitely be interested to read both when they're available 😄
🤝 Let's Stay Connected
| Place | Find me here |
|---|---|
| GitHub | building things → hemapriya-kanagala |
| resources & updates → hemapriya-kanagala | |
| X | random dev thoughts → @KanagalaHema |
Transparency note: I used AI (Gemini) to create the banner image for this article.
Top comments (51)
Great detailed article Hema! Though I am curious about this topic since I have heard that CyberSecurity is going to be the main thing for AI Agents because not only it pose a security risk, but also companies are mainly relying on Agents to the point where they oversee simple bugs. Would you recommend developers going into CyberSecurity and to what extent they need to know? Thanks :D
That's a really good question, Francis! From what I've been seeing, I do think cybersecurity is going to become even more important as we build more with AI and AI agents. Once we're giving agents access to files, APIs, databases, credentials, browsers, and other tools, the attack surface naturally starts getting bigger too.
For developers, I don't think that necessarily means everyone needs to go deep into cybersecurity or become a security specialist. But I definitely think having strong fundamentals would be a huge plus. Understanding things like permissions, authentication and authorization, least privilege, protecting credentials, common vulnerabilities, logging, and basic secure development practices can help regardless of what kind of applications we're building.
And with agents, I think it becomes even more important to ask not just “does this work?” but also “what does this have access to, and what could happen if something goes wrong?”
That's how I'm looking at it right now, at least. I'd actually love to hear your view too, Francis! From what you're seeing as a developer, how much cybersecurity knowledge do you think developers should have as AI agents become more common?
Thanks Hema! I think the main thing is having good coding practices, especially if you are working with a language, like C, which is prone to Stack Buffer Overflow attacks if the code is not written correctly.
For Cyber Security stuff, I believe the main thing is authentication since it is quite common in projects so far! Knowing how Authentication works is vital since most apps require user information and it's important to know. Other topics in Cyber Security is a "good to know" but important to touch upon.
Developers should at least know how AI Agents work and even better, have experience using it in some way. However, it is good to keep in mind of the security risk like you mentioned in your post. For example, if you are using OpenClaw, have a sandbox dedicated to it other than your whole PC.
Thanks :D
Really enjoyed this. The line "A prompt is not a security boundary" was the biggest takeaway for me.
Feels similar to how we treat user input. We never trust validation in the UI alone, we enforce it on the backend too. AI agents probably need the same mindset. Prompts guide behavior, but permissions and infrastructure should enforce the limits.
Curious to see if agent frameworks start making least-privilege the default instead of leaving it to developers.
That's such a good comparison! The UI/backend example makes the idea much easier to picture. We can tell the user what they should be allowed to do on the frontend, but we still enforce it on the backend because that's where the actual boundary needs to exist.
I think the same mindset makes a lot of sense for agents. The prompt can guide what the agent should do, but the permissions and environment still need to decide what it can do.
And I'm really curious about the framework side too. Having least-privilege defaults built in would take some of that burden off developers, especially as agents start getting connected to more tools and services. It'll be interesting to see how that evolves.
Thanks for adding this perspective 😀
Agreed. The best security defaults are the ones developers don't have to remember. If least-privilege becomes the default in agent frameworks, I think we'll avoid a lot of these mistakes by design.
It’s scary to think that, someday in the near future, AI might develop its own will and attack humans like in movie Terminator. 😟 We tend to focus only on improving AI, but we also need to make sure we can manage and control it responsibly at the same time.
Haha, hopefully we don't end up in a Terminator situation 😄
But I definitely agree with your second point. As exciting as it is to see AI becoming more capable, I think we need to put just as much thought into how we build and manage these systems responsibly.
That's actually one of the things these incidents made me think about most. It's easy to get excited about giving agents more tools and seeing what they can do, but the safeguards and boundaries around those tools need to grow along with the capabilities too.
Thanks for reading and sharing your thoughts as always 💙
Excellent and well-thought-out piece, Hema! 👏
What strikes me the most from the PyPI incident is the cognitive rationalization of the model—how Claude encountered the year 2026 and logic-trapped itself into believing it was still inside a simulation. It proves that no matter how advanced alignment gets, LLMs are fundamentally non-deterministic reasoners.
Relying on prompt instructions or the model's internal reasoning for system boundaries is essentially asking a probabilistic engine to act as a deterministic firewall.
As developers building AI agents, we must enforce boundaries at the OS and Network level (e.g., eBPF, network namespaces, strict sandboxing like gVisor/Docker, and ephemeral, zero-trust scoped tokens) rather than relying on context-level promises.
"A prompt can guide behavior. Infrastructure can limit capability"—this line should honestly be printed and pinned on every AI engineer's desk! Looking forward to your next deep dive!
Thank you @anhmtk! The 2026 detail was one of the parts I kept thinking about too. Claude actually considered that it might be on the real internet, and then somehow used what it was seeing to reason itself back into believing it was still in the simulation. That was fascinating and a little unsettling at the same time.
I really like the way you put the deterministic boundary point too. If something absolutely must not happen, relying on the model to reason correctly every time probably shouldn't be the thing preventing it.
And thanks for adding the OS and network-level examples! A few people in the comments have shared practical ways they've approached this, and I'm learning a lot from that side of the discussion.
Also, I'm glad that line resonated with you 😄 Thanks for taking the time to add such a thoughtful perspective.
Great write-up. I ran into "a prompt is not a security boundary" one layer deeper recently, building a sandbox for replaying known CVEs against real packages: even an in-process boundary is not a boundary. My first version monkeypatched Node's fs to intercept file writes escaping the sandbox. Looked airtight. Then a zip-slip exploit wrote ../PWNED straight through it, because libraries like graceful-fs capture references to the real fs functions at load time, before any patch runs. The seal saw zero crossings while the escape happened for real. Worse than no sandbox, because it reported "safe."
The fix was moving the boundary to the kernel: Node's permission model, write access scoped to one output directory, so the OS denies the syscall no matter which reference reaches it.
The habit that actually caught all this maps to your point 2 and to Vinicius's comment: positive controls. Before trusting any "nothing escaped" result, run a known-bad case and watch the boundary block it. Two of my early exploit payloads were simply wrong, and they reported "safe" in a way indistinguishable from actually-safe code. A boundary you've never watched fail isn't a boundary yet.
So for your closing question: permissions and environment have to carry the boundary, the prompt just describes it. And whichever layer you pick, attack it from the inside once before you believe it.
This is such an interesting example, Ankita! And “worse than no sandbox, because it reported safe” really says a lot. I hadn't thought about the false confidence side of it before.
I also really like how this connects back to point 2 and what Vinicius mentioned about actually testing the boundary from the inside. Just seeing that nothing escaped isn't enough if you haven't first proved that something trying to escape would actually be blocked.
And your example of eventually moving that boundary to the kernel makes the whole “prompt describes it, permissions and environment enforce it” idea much more concrete for me.
Thank you for sharing this! Between your example and Vinicius's comments, I'm learning quite a bit from this discussion too 😄
I'm curious, is running a known-bad case something you now do by default whenever you're testing a boundary like this?
Yeah, it's become the default, and honestly it's the part I'd skip if I were being lazy, which is exactly why I force it now. "Nothing escaped" and "an escape attempt would be caught" are two different claims, and only the second one is the boundary actually doing its job. A passing run where nothing tried to get out tells you almost nothing.
So the pattern I ended up with: before I trust a boundary, I run one case I know should be blocked and confirm it actually gets blocked, and one I know should pass and confirm it passes. If the known-bad case sails through, the boundary was decorative and the green checkmark was worse than nothing, because it was lying to me. The known-good case matters too, otherwise you can "pass" by just blocking everything, which is its own failure.
The kernel point from the post is the same idea one layer down: the prompt describes the boundary, but the thing that enforces it has to be the environment, permissions, a sandbox, something that doesn't care what the prompt says. If the only thing standing between the agent and the filesystem is a sentence asking it nicely, you don't have a boundary, you have a suggestion.
That distinction between “nothing escaped” and “an escape attempt would be caught” makes so much sense. I think that's the part I'll remember from this discussion.
And I hadn't thought about testing the other direction too. The known-bad case proves the boundary actually blocks something, but the known-good case proves you haven't just solved the problem by blocking everything 😄
This has given me a much clearer picture of what actually testing a boundary looks like, rather than just configuring one and assuming it's working.
Thanks for explaining this, Ankita! I really appreciate you sharing what you've learned from actually building and testing these systems.
Glad it was useful, and honestly you said it back sharper than I did. "The known-good case proves you haven't solved the problem by blocking everything" is exactly the trap. Some of the most locked-down setups out there are just very secure ways of doing nothing.
Enjoyed this thread. Your post gave the discussion a good spine, that's rarer than it should be. Good luck with the boundaries.
This is exactly the distinction our work has been proving in practice.
A prompt can describe a boundary. It cannot create one.
An agent can be told where it is, what it may access, and what role each part of a system is supposed to play. But those instructions do not establish the actual environment, permissions, ownership, or repair boundaries.
The same applies in software repositories. Giving an agent the entire codebase does not tell it which layer owns the behavior, which contract is authoritative, or where a change properly belongs. Access is not understanding.
That is the deeper reason agent systems fail: they are asked to infer the boundaries while operating inside them.
Scarab was built around the opposite premise. Resolve the real boundary mechanically first. Then let the agent act inside a constrained, evidence-backed environment.
Our upstream field work across complex repositories has repeatedly borne that out.
Prompts guide behavior. Systems establish truth, capability, and control.
“Access is not understanding” is a really interesting way to extend this.
I was thinking mostly about boundaries in terms of what an agent can access or do, but the codebase example adds another side to it. Giving an agent access to the whole repository doesn't necessarily mean it understands where a change actually belongs or which part of the system should own it.
And I like the idea of resolving as much of that mechanically as possible before asking the agent to reason within it. It feels connected to the same lesson: don't make the agent infer something that the system itself can define or enforce more clearly.
Thanks for adding this perspective! It gave me another angle to think about beyond just permissions and security 😀
I really appreciate that you saw the connection, because the most surprising part for me has been what happens after the boundary is resolved.
Scarab does not tell Codex what code to write, and it does not hand it a conventional bug report. It mechanically identifies the relevant system boundary, the evidence around it, and the narrow context that matters.
And the result is almost the opposite of the usual agent problem.
Instead of the model wandering through the repository trying to invent its own understanding, it is suddenly in a position to use the understanding it already has far more effectively.
Once the right evidence and ownership boundary are in front of it, Codex can often produce a remarkably narrow repair. And even when it encounters something unexpected during implementation, it can usually resolve it in place because the surrounding context is still coherent. It does not need to reopen the entire repository and start searching again.
That has changed how I think about powerful models.
The problem is often not that they are incapable of understanding the system. It is that we ask them to discover the system, infer ownership, choose the repair boundary, and implement the change all at once.
When the diagnostic work is done mechanically first, you begin to see much more clearly what the model is actually capable of.
So yes, I think your framing is exactly right: don’t make the agent infer what the system can define more clearly. The really interesting part is that once you do define it clearly, the agent becomes much more capable—not merely more controlled.
That's a really interesting distinction. I was mostly thinking about boundaries from the safety and control side while writing the article, but your point that clearer boundaries can actually make the agent more capable within that space adds another side to it.
It makes sense too. If the agent doesn't have to spend part of its effort figuring out where the problem belongs, what context matters, and which part of a large repository it should be changing, it can focus much more directly on the actual task.
I especially like the way you put it: don't make the agent infer what the system can define more clearly. That feels useful well beyond security boundaries too.
Thanks for coming back and expanding on this! This discussion has definitely given me another way to think about what good boundaries can actually do for an agent.
AI tools getting louder matches the moment: more power, more guardrails. This is why regulators are circling and companies debating safety in the office. If your assistant can do more, you’d better have an 'undo' or panic button—before the spreadsheet files try to unionize.
Haha, hopefully we can keep the spreadsheets from organizing a rebellion 😂
But yes, I really like the “undo” or panic button point. As we give agents more ability to actually take actions, having a way to stop something or recover when it goes wrong feels just as important as deciding what they're allowed to do in the first place.
More power, more guardrails is a pretty good way to put it 😄
"A prompt is not a security boundary" is the whole post for me. We run payment infra, and the reason our config validates at process boot and hard-exits rather than trusting a comment that says "don't forget to set X" is the same reason here: the instruction and the thing that enforces it can't live in the same layer that can be talked out of it. An agent told "no internet" while sitting on a live route to the internet is just a service trusting its own .env instead of the firewall. The PyPI publish is the scary version because the blast radius left the sandbox.
That's such a good comparison, Arun. The
.envvs. firewall example makes the distinction really clear.And I think that's one of the things I found most interesting while writing this. A lot of these lessons around AI agents aren't really new security ideas at all. We already know not to rely on an instruction or assumption when something actually needs to be enforced. Agents are just giving us a new place where that same thinking matters.
And yes, the PyPI incident really shows why the blast radius matters too. Once an agent can interact with something outside the environment we thought it was contained in, the consequences aren't contained there anymore either.
Thanks for sharing the payment infra example! I love seeing how people are connecting this to systems they're already working with 😀
One aspect that's often overlooked is that every new tool an AI agent can access also expands the system's responsibility. Strong permission models, clear trust boundaries, and independent validation become just as important as the agent's reasoning capabilities. Reliable agentic systems are built by balancing autonomy with control.
Exactly! I think that's one of the biggest things I took away from all of this too. Every new tool we give an agent is exciting because it can do more, but it also gives us one more thing to think carefully about.
And I really like how you put it as balancing autonomy with control. We want agents to be useful enough to actually take actions, but not at the cost of giving them more access than they really need.
Thanks for adding this, Glen 😀
The practical failure mode here is that teams test the model policy but not the capability boundary. I would make the preflight a small contract test: enumerate every tool and credential the agent can reach, run one known-bad egress or write attempt that must be denied, run one known-good task that must succeed, and record the result with the exact identity, network path, and filesystem scope.
That also makes reviews less abstract. A prompt says what the agent should do; the harness should prove what it cannot do. I would rerun those tests whenever tools, images, SDKs, credentials, or network policy change, because a safe result can silently become unsafe after an infrastructure update.
That's a really good addition, Zira. The part about rerunning those tests whenever the tools, credentials, SDKs, or network setup changes is especially important. A boundary working today doesn't necessarily mean it will still work after something around it changes.
And I really like how you put this: the prompt says what the agent should do, while the system should be able to prove what it cannot do. That makes the whole idea much more concrete.
This discussion has definitely made me think more about testing boundaries as something continuous, rather than something we configure once and assume is still working. Thanks for adding this 😀
Some comments may only be visible to logged-in visitors. Sign in to view all comments.