Khaos: Open-Source Framework Exposes Vulnerabilities in AI Agents

Khaos is an open-source chaos engineering framework designed to adversarially test AI agents for vulnerabilities. The framework addresses the growing concern that many AI agents, even those processing payments or handling sensitive data, can be easily tricked into bypassing their own safety policies. Khaos tests for a range of vulnerabilities, including prompt injection, tool misuse, data exfiltration, and infrastructure faults.

The framework includes six intentionally vulnerable example agents, such as a support bot, SQL agent, and payment processor, with real attack scenarios demonstrating how they can be compromised. Khaos is compatible with various LLMs and agent frameworks, including OpenAI/Anthropic, Gemini, LangGraph, CrewAI, and AutoGen. It works by auto-patching LLM calls to inject faults and log telemetry, allowing developers to observe how agents respond to adversarial inputs.

Khaos distinguishes itself by focusing on testing the agent's environment, rather than just the model in isolation. This approach provides a more realistic assessment of an agent's security posture. The framework also includes tutorials using the free Gemini API, making it accessible to developers who want to learn about AI agent security without incurring significant costs. While Khaos offers a valuable tool for identifying and mitigating vulnerabilities, it also underscores the inherent risks associated with deploying AI agents. The framework's ease of use could potentially be exploited by malicious actors to identify weaknesses in production systems.