LLMs as Legal Decision Tools: Study Reveals Persuadability by Advocate Quality

The exploration into the persuadability of Large Language Models (LLMs) when applied to legal decision-making uncovers a critical vulnerability that directly impacts their suitability for judicial and administrative roles. As LLMs are increasingly proposed as first-instance decision-makers or assistants, the finding that their agreement with a legal viewpoint can be influenced by the *quality of the advocate* rather than solely the *merits of the case* presents a profound challenge to the foundational principles of justice and impartiality. This research moves beyond theoretical discussions by reporting original experimental results on how frontier open- and closed-weights LLMs respond to contending legal arguments.

This technical and ethical context is paramount. Legal decision-makers, whether human or AI, are expected to engage with arguments critically, respond to them, and potentially be persuaded by their substance, but not by the rhetorical skill of the presenter. The study's findings suggest that current LLMs may struggle with this distinction, exhibiting a form of "advocate bias." This mirrors concerns about human judges being swayed by charismatic lawyers, but in an AI context, the mechanisms of such persuadability are opaque and harder to mitigate without fundamental architectural changes or rigorous, domain-specific fine-tuning. The implications extend to the design of AI systems for any high-stakes decision-making where objective evaluation of evidence and argument is paramount.

The forward-looking implications are twofold. On one hand, identifying this persuadability provides a clear directive for future research and development: AI legal tools must be engineered with explicit mechanisms to filter out rhetorical influence and prioritize substantive legal reasoning. This could involve novel training paradigms, adversarial testing specifically designed to expose such biases, or the integration of formal logic systems. On the other hand, if this challenge proves intractable with current LLM architectures, it necessitates a re-evaluation of the scope and limits of AI deployment in legal contexts. Unchecked, such a bias could lead to systemic injustices, erode public trust in AI-assisted legal processes, and inadvertently favor parties with superior advocacy resources, thereby exacerbating existing inequalities within the legal system.