AI Agents vs. Web Security: Testing Offensive Capabilities

This study evaluates the performance of AI agents in offensive security tasks, using a series of lab challenges modeled after real-world vulnerabilities. Three models, Claude Sonnet 4.5, GPT-5, and Gemini 2.5 Pro, were tested on their ability to identify and exploit vulnerabilities in web applications. The challenges were designed as a Capture the Flag (CTF) setup, with clear 'win conditions' (flags) indicating successful exploitation.

The results indicate that AI agents are generally proficient in directed tasks, but their effectiveness decreases in less structured environments. The presence of clear success metrics, such as the flags, significantly improved agent performance by reducing false positives and encouraging continued effort. The study highlights the importance of providing AI agents with clear objectives and feedback mechanisms to enhance their capabilities in real-world security scenarios.

While AI agents show promise in automating certain aspects of security testing, they are not yet a replacement for human security professionals. Further research is needed to improve their ability to handle unguided environments and complex vulnerabilities. The findings suggest that a hybrid approach, combining the strengths of AI and human expertise, is likely to be the most effective strategy for enhancing web security.