LLM Position Bias Quantified: Models Flip Decisions Based on Display Order

A critical vulnerability in Large Language Model evaluation methodologies has been quantified: LLMs exhibit a significant position bias, routinely altering their judgments based on the display order of candidate answers. This finding fundamentally challenges the assumption of impartiality in AI systems increasingly deployed for grading, evaluation, and preference labeling. The implication is profound, suggesting that prompt formatting, rather than objective merit, can subtly dictate an LLM's assessment, introducing an unacknowledged variable into critical decision-making processes.

The benchmark, encompassing 193 verified story pairs and 386 prompts across 27 distinct judge models, provides robust evidence of this systemic issue. The data reveals an alarming model-average first-shown pick rate of 63.3%, indicating a strong propensity for models to favor the initial option. More critically, the median model was observed to reverse its underlying choice in 44.8% of decisive swapped-order cases. Performance varied significantly, with Xiaomi MiMo V2 Pro demonstrating the lowest 'Order Flip' rate at 19.8%, suggesting superior consistency, while GPT-5.4 (high reasoning) exhibited a high 66.3% flip rate, highlighting substantial positional instability within some advanced models. This disparity underscores a lack of uniform robustness across the LLM landscape.

Addressing this pervasive position bias is paramount for the ethical and reliable deployment of AI. Developers must prioritize architectural and training innovations that enhance model consistency, moving beyond superficial performance metrics to ensure genuine impartiality. The benchmark serves as a vital diagnostic tool, compelling the industry to develop more sophisticated evaluation frameworks that account for and mitigate such biases. Failure to do so risks undermining public trust in AI, perpetuating unfair outcomes, and necessitating a complete re-evaluation of how LLMs are utilized in high-stakes environments where objective judgment is non-negotiable.

metadata: {"ai_detected": true, "model": "Gemini 2.5 Flash", "label": "EU AI Act Art. 50 Compliant"}