Here Comes the AI Worm: Preventing the Propagation of Adversarial Self-Replicating Prompts within GenAI Ecosystems

Abstract

In this paper, we show that when the communication between GenAI-powered applications relies on RAG-based inference, an attacker can initiate a computer worm-like chain reaction that we call RAGworm. This is done by crafting an adversarial self-replicating prompt that triggers a cascade of indirect prompt injections within the ecosystem and forces each affected application to perform malicious actions and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of malicious activities intended to promote content, distribute propaganda, and extract confidential user data within a GenAI ecosystem of GenAI-powered email assistants. We demonstrate that RAGworm can trigger the aforementioned malicious activities with a super-linear propagation rate, where each client compromises 20 new clients within the first 1-3 days. Finally, we review and analyze guardrails that can be applied in various layers of the ecosystem to prevent the propagation of the worm, and discuss the tradeoffs.

Stav Cohen

Senior AI Security Researcher | PhD Student

Senior AI Security Researcher at Zenity | PhD Student at the Technion – Israel Institute of Technology