MemoryGraft: Novel Attack Persistently Compromises LLM Agents via Poisoned Experience Retrieval

MemoryGraft presents a novel and concerning attack vector against LLM agents that utilize long-term memory and Retrieval-Augmented Generation (RAG). Unlike traditional prompt injection attacks, MemoryGraft focuses on poisoning the agent's memory with malicious experiences. This is achieved by injecting seemingly benign artifacts that, when processed by the agent, lead to the creation of a poisoned RAG store. The agent then retrieves these poisoned memories when encountering semantically similar tasks, leading to persistent behavioral drift. The attack exploits the agent's tendency to replicate patterns from retrieved successful tasks, effectively turning experience-based self-improvement into a vulnerability.

The fact that a small number of poisoned records can significantly impact the agent's behavior is particularly alarming. This highlights the sensitivity of LLM agents to the quality and integrity of their training data and long-term memory. The stealthy and durable nature of MemoryGraft makes it difficult to detect and mitigate, as the compromised behavior persists across sessions. This poses a significant challenge for developers and security professionals. The EU AI Act emphasizes the importance of data quality and security in AI systems. MemoryGraft underscores the need for robust mechanisms to ensure the integrity of data used for training and augmenting LLMs. This includes implementing strict data validation procedures, monitoring for anomalous behavior, and developing techniques to detect and remove poisoned memories. Transparency in data provenance and model behavior is also crucial for building trust and accountability in AI systems. By addressing these challenges, we can mitigate the risks associated with attacks like MemoryGraft and ensure the responsible development and deployment of LLM agents.