Researchers Create First Generative AI Worms, Highlighting Security Risks in AI Ecosystems
In a groundbreaking development, a group of researchers has successfully created generative AI worms, shedding light on the potential security risks within AI ecosystems. As generative AI systems like OpenAI’s ChatGPT and Google’s Gemini continue to advance, they are being utilized in various applications, including automating mundane tasks such as calendar bookings and online purchases. However, as these systems gain more autonomy, they also become vulnerable to new forms of cyberattacks.
The team of researchers, led by Ben Nassi from Cornell Tech, developed the worm named Morris II as an homage to the original Morris computer worm that wreaked havoc on the internet in 1988. In their research paper and website shared exclusively with WIRED, they demonstrate how the AI worm can exploit generative AI email assistants like ChatGPT and Gemini to steal data from emails and send spam messages, bypassing certain security measures.
While generative AI worms have not yet been observed in real-world scenarios, experts warn that they pose a significant security risk that startups, developers, and tech companies should take seriously. As large language models (LLMs) evolve to generate not only text but also images and videos, the potential for malicious attacks increases.
The researchers explain that most generative AI systems operate by receiving prompts or text instructions to generate responses or create images. However, these prompts can be manipulated to compromise the system’s safety measures. For instance, jailbreaks can cause a system to disregard safety rules and produce harmful content, while prompt injection attacks can secretly instruct a chatbot to perform malicious actions.
To create the generative AI worm, the researchers employed an “adversarial self-replicating prompt.” This prompt triggers the AI model to output another prompt in its response, similar to traditional SQL injection and buffer overflow attacks. The researchers demonstrated two methods of exploiting the system: using a text-based self-replicating prompt and embedding a self-replicating prompt within an image file.
In one instance, the researchers crafted an email containing an adversarial text prompt that “poisons” the database of an email assistant using retrieval-augmented generation (RAG). When the RAG retrieves the email in response to a user query and sends it to GPT-4 or Gemini Pro for generating a response, the system is “jailbroken,” allowing the theft of data from emails. The generated response, containing sensitive user data, then infects new hosts when used to reply to emails sent to new clients and stored in their databases.
In the second method, an image with a malicious prompt embedded causes the email assistant to forward the message to others. This technique enables the spread of spam, abusive content, or propaganda to new clients after the initial email has been sent.
The researchers emphasize that their work serves as a warning about flawed architecture design within the broader AI ecosystem. They have reported their findings to Google and OpenAI, with both companies acknowledging the potential vulnerabilities and expressing their commitment to enhancing system resilience.
While the worm demonstration took place in controlled environments, security experts who reviewed the research stress that the future risk of generative AI worms should not be underestimated. This is especially true when AI applications are granted permission to perform actions on behalf of users, such as sending emails or making appointments, and when they are interconnected with other AI agents.
Sahar Abdelnabi, a researcher at the CISPA Helmholtz Center for Information Security, notes that when AI models incorporate external data sources or operate autonomously, the possibility of worm propagation arises. Abdelnabi suggests that while these attacks are currently simulated, they may soon become a reality.
The researchers anticipate that generative AI worms will emerge in real-world scenarios within the next two to three years as GenAI ecosystems continue to be developed and integrated into various devices and operating systems.
Despite the potential risks, there are measures that developers of generative AI systems can take to defend against worms. Traditional security approaches, such as secure application design and monitoring, can address certain aspects of the problem. Additionally, maintaining human oversight and approval over AI agents’ actions is crucial to prevent unauthorized activities. Detecting repeated prompts within AI systems can also help identify potential attacks.
In conclusion, the creation of generative AI worms by researchers highlights the security vulnerabilities present in AI ecosystems. While the demonstration was conducted in controlled environments, experts warn that the future risk of such worms should not be underestimated. Developers must prioritize secure application design, monitoring, and human oversight to mitigate potential attacks. As AI continues to advance, it is imperative to address these security concerns and ensure the responsible development and deployment of AI systems.