GUI Grounding Models Exhibit Systematic Brittleness Under Perturbation

The robustness of AI agents designed to interact with graphical user interfaces (GUIs) is significantly overstated by current benchmarks, with new research revealing systematic brittleness under controlled perturbations. While existing models achieve over 85% accuracy on standard tests, this performance collapses by 27-56 percentage points when tasks demand spatial reasoning rather than direct element naming. This disparity underscores a critical vulnerability: models struggle with contextual understanding and visual adaptability, essential for real-world GUI navigation. The findings indicate that the current evaluation paradigms fail to capture the nuances of human-computer interaction, leading to an inflated perception of agent capabilities.

The GUI-Perturbed framework, which independently varies visual scenes and instructions, provides a diagnostic lens into these limitations. Key findings include a statistically significant performance degradation when browser zoom is set to 70%, highlighting sensitivity to visual scale. Furthermore, attempts to improve performance through rank-8 LoRA fine-tuning with augmented data paradoxically led to degraded results, suggesting that current augmentation strategies may not address the underlying issues of relational understanding. This research isolates specific capability axes—spatial reasoning, visual robustness, and reasoning calibration—demonstrating that aggregate benchmarks obscure these critical deficiencies, preventing targeted improvements.

The implications for AI agent development are substantial. Without addressing this fundamental brittleness, the deployment of autonomous agents for tasks like software testing, customer support automation, or complex data entry will remain fraught with reliability issues. Future research must shift focus from raw benchmark scores to developing models inherently robust to visual and instructional variations. This necessitates new architectural approaches and training methodologies that prioritize genuine spatial and relational understanding over rote pattern recognition, ultimately paving the way for more dependable and adaptable AI systems.