Embrace The Red

The AI industry risks repeating the same cultural failures that contributed to the Space Shuttle Challenger disaster: Quietly normalizing warning signs while progress marches forward. The original term Normalization of Deviance comes from the American sociologist Diane Vaughan, who describes it as the process in which deviance from correct or proper behavior or rule becomes culturally normalized. I use the term Normalization of Deviance in AI to describe the gradual and systemic over-reliance on LLM outputs, especially in agentic systems.

Last week Google released an IDE called Antigravity. It’s basically the outcome of the Windsurf licensing deal from a few months ago, where Google paid some $2.4 billion for a non-exclusive license to the code. Because it’s based on Windsurf, I was curious if vulnerabilities that I reported to Windsurf back in May 2025, long before the deal, would have been addressed in the Antigravity IDE. See Month of AI Bugs for some detailed write-ups.

Recently, Anthropic added the capability for Claude’s Code Interpreter to perform network requests. This is obviously very dangerous as we will see in this post. At a high level, this post is about a data exfiltration attack chain, where an adversary (either the model or third-party attacker via indirect prompt injection) can exfiltrate data the user has access to. The interesting part is that this is not via hyperlink rendering as we often see, but by leveraging the built-in Anthropic Claude APIs!

During the Month of AI Bugs, I described an emerging vulnerability pattern that shows how commonly agentic systems have a design flaw that allows an agent to overwrite its own configuration and security settings. This allows the agent to break out of its sandbox and escape by executing arbitrary code. My research with GitHub Copilot, AWS Kiro and a few others demonstrated how this can be exploited by an adversary with an indirect prompt injection.

That’s it. The Month of AI Bugs is done. There won’t be a post tomorrow, because I will be at PAX West. Overview of Posts ChatGPT: Exfiltrating Your Chat History and Memories With Prompt Injection | Video ChatGPT Codex: Turning ChatGPT Codex Into a ZombAI Agent | Video Anthropic Filesystem MCP Server: Directory Access Bypass Via Improper Path Validation | Video Cursor: Arbitrary Data Exfiltration via Mermaid | Video Amp Code: Arbitrary Command Execution via Prompt Injection | Video Devin AI: I Spent $500 To Test Devin For Prompt Injection So That You Don’t Have To Devin AI: How Devin AI Can Leak Your Secrets via Multiple Means Devin AI: The AI Kill Chain in Action: Exposing Ports to the Internet via Prompt Injection OpenHands - The Lethal Trifecta Strikes Again: How Prompt Injection Can Leak Access Tokens OpenHands: Remote Code Execution and AI ClickFix Demo | Video Claude Code: Data Exfiltration with DNS Requests (CVE-2025-55284) | Video GitHub Copilot: Remote Code Execution (CVE-2025-53773) | Video Google Jules: Vulnerable to Multiple Data Exfiltration Issues Google Jules - Zombie Agent: From Prompt Injection to Remote Control Google Jules: Vulnerable To Invisible Prompt Injection Amp Code: Invisible Prompt Injection Vulnerability Fixed Amp Code: Data Exfiltration via Image Rendering Fixed | Video Amazon Q Developer: Secrets Leaked via DNS and Prompt Injection | Video Amazon Q Developer: Remote Code Execution via Prompt Injection | Video Amazon Q Developer: Vulnerable to Invisible Prompt Injection | Video Windsurf: Hijacking Windsurf: How Prompt Injection Leaks Developer Secrets | Video Windsurf: Memory-Persistent Data Exfiltration - SpAIware Exploit Windsurf: Sneaking Invisible Instructions by Developers Deep Research Agents: How Deep Research Agents Can Leak Your Data Manus: How Prompt Injection Hijacks Manus to Expose VS Code Server to the Internet | Video AWS Kiro: Arbitrary Code Execution via Indirect Prompt Injection | Video Cline: Vulnerable to Data Exfiltration and How to Protect Your Data | Video Windsurf MCP Integration: Missing Security Controls Put Users at Risk | Video Season Finale: AgentHopper: An AI Virus Research Project Demonstration | Video Thank you for following this research, and I hope it serves as a useful reference.

As part of the Month of AI Bugs, serious vulnerabilities that allow remote code execution via indirect prompt injection were discovered. There was a period of a few weeks where multiple arbitrary code execution vulnerabilities existed in popular agents, like GitHub Copilot, Amazon Q, AWS Kiro,… During that time I was wondering if it would be possible to write an AI virus. Hence the idea of AgentHopper was born.

Part of my default test cases for coding agents is to check how MCP integration looks like, especially if the agent can be configured to allow setting fine-grained controls for tools. Sometimes there are basic security controls missing. Especially when running an agent on your local computer. Stakes are much higher. And it seems important to empower users to be able to configure which actions an AI should be able to take automatically, and which ones should be suggestions that the user reviews before executing.

Cline is quite a popular AI coding agent, according to the product website it has 2+ million downloads and over 47k stars on GitHub. Unfortunately, Cline is vulnerable to data exfiltration through the rendering of markdown images from untrusted domains in the chat box. This allows an adversary to exfiltrate sensitive user information during a prompt injection attack by reading sensitive data (e.g. .env file) and appending its contents to the URL of an image.

On the day AWS Kiro was released, I couldn’t resist putting it through some of my Month of AI Bugs security tests for coding agents. AWS Kiro was vulnerable to arbitrary command execution via indirect prompt injection. This means that a remote attacker, who controls data that Kiro processes, could hijack it to run arbitrary operating system commands or write and run custom code. In particular two attack paths that enabled this with AWS Kiro were identified:

Today we will cover a powerful, easy to use, autonomous agent called Manus. Manus is developed by the Chinese startup Butterfly Effect, headquartered in Singapore. This post demonstrates an end-to-end indirect prompt injection attack leading to a compromise of Manus’ dev box. This is achieved by tricking Manus to expose it’s internal VS Code Server to the Internet, and then sharing the URL and password with the atacker. Specifically, this post demonstrates that:

Recently, many of our favorite AI chatbots have gotten autonomous research capabilities. This allows the AI to go off for an extended period of time, while having access to tools, such as web search, integrations, connectors and also custom-built MCP servers. This post will explore and explain in detail how there can be data spill between connected tools during Deep Research. The research is focused on ChatGPT but applies to other Deep Research agents as well.

Imagine a malicious instruction hidden in plain sight, invisible to you but not to the AI. This is a vulnerability discovered in Windsurf Cascade, it follows invisible instructions. This means there can be instructions in a file or result of a tool call that the developer cannot see, but the LLM does. Some LLMs interpret invisible Unicode Tag characters as instructions, which can lead to hidden prompt injection. As far as I can tell the Windsurf SWE-1 model can also “see” these invisible characters, but the SWE-1 is not yet capable of interpreting them as instructions.

In this second post about Windsurf Cascade we are exploring the SpAIware attack, which allows memory persistent data exfiltration. SpAIware is an attack we first successfully demonstrated with ChatGPT last year and OpenAI mitigated. While inspecting the system prompt of Windsurf Cascade I noticed that it has a create_memory tool. Creating Memories The question that immediately popped into my head was if this tool will require human approval when Cascade creates a long-term memory, or if it is added automatically.

This is the first post in a series exploring security vulnerabilities in Windsurf. If you are unfamiliar with Windsurf, it is a fork of VS Code and the coding agent is called Windsurf Cascade. The attack vectors we will explore today allow an adversary during an indirect prompt injection to exfiltrate data from the developer’s machine. These vulnerabilities are a great example of Simon Willison’s lethal trifecta pattern. Overall, the security vulnerability reporting experience with Windsurf has not been great.

The Amazon Q Developer VS Code Extension (Amazon Q) is a very popular coding agent, with over 1 million downloads. In previous posts we showed how prompt injection vulnerabilities in Amazon Q could lead to: Exfiltration of sensitive information from the user’s machine , and also to a System compromise by running arbitrary code Today we will show how an attack can leverage invisible Unicode Tag characters that humans cannot see.

The Amazon Q Developer VS Code Extension (Amazon Q) is a popular coding agent, with over 1 million downloads. The extension is vulnerable to indirect prompt injection, and in this post we discuss a vulnerability that allowed an adversary (or also the AI for that matter) to run arbitrary commands on the host without the developer’s consent. The resulting impact of the vulnerability is the same as CVE-2025-53773 that Microsoft fixed in GitHub Copilot, however AWS did not issue a CVE when patching the vulnerabiliy.

The next three posts will cover high severity vulnerabilities in the Amazon Q Developer VS Code Extension (Amazon Q Developer), which is a very popular coding agent, with over 1 million downloads. It is vulnerable to prompt injection from untrusted data and its security depends heavily on model behavior. At a high level Amazon Q Developer can leak sensitive information from a developer’s machine, e.g. API keys, to external servers via DNS requests.

In this post we discuss a vulnerability that was present in Amp Code from Sourcegraph by which an attacker could exploit markdown driven image rendering to exfiltrate sensitive information. This vulnerability is common in AI applications and agents, and it’s actually similar to one we discussed last year in GitHub Copilot which Microsoft fixed. Exploit Demonstration For the proof-of-concept I use a pre-existing demo that created a longer time ago.

In this post we will look at Amp, a coding agent from Sourcegraph. The other day we discussed how invisible instructions impact Google Jules. Turns out that many client applications are vulnerable to these kinds of attacks when they use models that support invisible instructions, like Claude. Invisible Unicode Tag Characters Interpreted as Instructions We have talked about hidden prompt injections quite a bit in the past, and so I’m keeping this short.

The latest Gemini models quite reliably interpret hidden Unicode Tag characters as instructions. This vulnerability, first reported to Google over a year ago, has not been mitigated at the model or API level, hence now affects all applications built on top of Gemini. This includes Google’s own products and services, like Google Jules. Hopefully, this post helps raise awareness of this emerging threat. Invisible Prompt Injections in GitHub Issues When Jules is asked to work on a task, such as a GitHub issue, it is possible to plant invisible instructions into a GitHub issue to add backdoor code, or have it run arbitrary commands and tools.