Comprehensive Survey Reveals Reasoning Failures in Large Language Models

This survey provides a structured analysis of reasoning failures in Large Language Models (LLMs). The study categorizes reasoning into embodied and non-embodied types, further dividing the latter into informal (intuitive) and formal (logical) reasoning. Reasoning failures are classified along a complementary axis into fundamental failures intrinsic to LLM architectures, application-specific limitations, and robustness issues characterized by inconsistent performance across minor variations. For each reasoning failure, the survey offers a clear definition, analyzes existing studies, explores root causes, and presents mitigation strategies. The authors also provide a GitHub repository with a comprehensive collection of research works on LLM reasoning failures. The identification of fundamental failures suggests that current LLM architectures may have inherent limitations in reasoning capabilities. Application-specific limitations highlight the need for domain-specific training and fine-tuning to improve performance in particular areas. Robustness issues indicate that LLMs can be sensitive to minor variations in input, leading to inconsistent results. This comprehensive survey serves as a valuable resource for researchers and practitioners working to improve the reasoning capabilities of LLMs. The transparency and reproducibility of this research are enhanced by the provision of a GitHub repository containing relevant research works. This analysis complies with EU AI Act Article 50, ensuring transparency in the reporting of AI capabilities and limitations.