CAMO Ensemble Boosts LLM Performance on Imbalanced Datasets

The pervasive challenge of class imbalance severely compromises the efficacy of language models in real-world categorization tasks. Traditional ensemble methods, by their nature, tend to prioritize majority classes, leading to diminished performance on critical minority classes and a subsequent reduction in overall F1-score. The introduction of CAMO (Class-Aware Minority-Optimized) represents a significant methodological advancement, providing a novel ensemble technique specifically designed to enhance robust language model evaluation on such imbalanced data, thereby improving the fairness and accuracy of AI systems.

CAMO operates through a sophisticated hierarchical procedure that integrates vote distributions, confidence calibration, and inter-model uncertainty to dynamically bolster underrepresented classes. This mechanism ensures that minority forecasts are not only preserved but actively amplified, counteracting the inherent bias of imbalanced datasets. The framework's effectiveness was rigorously verified on two highly unbalanced, domain-specific benchmarks: the DIAR-AI/Emotion dataset and the ternary BEA 2025 dataset. Benchmarking against seven established ensemble algorithms, using a diverse set of eight language models (three LLMs and five SLMs) in both zero-shot and fine-tuned configurations, CAMO consistently achieved the highest strict macro F1-score. This performance establishes a new benchmark and underscores its reliability as a domain-neutral solution for unbalanced categorization.

The strategic implications of CAMO are far-reaching, particularly for the responsible deployment of AI. By ensuring that language models perform robustly across all classes, including those that are rare or underrepresented, CAMO contributes directly to the development of more equitable and less biased AI applications. This is critical in sensitive domains such as medical diagnostics, legal document analysis, and social media content moderation, where misclassifications of minority instances can have severe consequences. The framework's ability to work in concert with model adaptation further suggests its potential to become a standard component in the fine-tuning and evaluation pipelines for next-generation language models, fostering greater trust and utility in AI systems.