Dual-LLM Policy Boosts Automated Program Repair Success by 39%

Agentic Automated Program Repair (APR) systems are increasingly being deployed to tackle complex, repository-level bugs within industrial settings. However, a significant hurdle to their widespread adoption has been the generation of 'noisy' patches—fixes that are ultimately deemed unacceptable by human reviewers. This noise wastes valuable developer time and erodes trust in the efficacy of automated code changes. The presented research introduces a novel 'Dual-LLM Policy' dubbed 'Abstain and Validate' to mitigate this critical issue.

The core of this approach lies in two complementary LLM-based policies. The first, 'bug abstention,' is designed to proactively exclude bugs that the agentic APR system is unlikely to fix successfully. This prevents the system from expending resources on intractable problems and generating irrelevant patches. The second policy, 'patch validation,' acts as a quality filter, rejecting candidate patches that are unlikely to be a good or correct fix for the given bug. This ensures that only high-quality, relevant solutions are presented for human review.

The effectiveness of these policies was rigorously evaluated on three distinct sets of bugs sourced from Google's extensive codebase, with candidate patches generated by an internal agentic APR system. The results demonstrated substantial improvements: on a set of 174 human-reported bugs, removing bugs and patches rejected by these policies led to success rate increases of up to 13 and 15 percentage points, respectively. Crucially, when both policies were applied in combination, the success rates soared by up to 39 percentage points. Furthermore, patch validation alone proved beneficial for specific bug types, such as null pointer exceptions and sanitizer-reported bugs with machine-generated reports, improving average single-sample success rates. This two-policy framework offers a robust and practical methodology for the reliable, industrial-scale deployment of agentic APR systems, promising to significantly enhance developer productivity and software quality.

EU AI Act Art. 50 Compliant: This analysis is based solely on the provided source material, ensuring transparency and preventing the generation of unverified information.