LLM Value Alignment: Supervised Fine-Tuning Sets Core Ethics, Preference Optimization Struggles to Realign

The process of aligning large language models (LLMs) with human values is revealed to be critically dependent on the supervised fine-tuning (SFT) phase, with subsequent preference optimization having a significantly diminished capacity to re-align these foundational values. This finding challenges the prevailing assumption that preference optimization, often seen as the primary mechanism for ethical refinement, can effectively correct value biases introduced earlier. The research underscores that value alignment is not a modular add-on but an intrinsic property largely cemented during the initial stages of post-training, necessitating a re-evaluation of current alignment strategies.

Experiments conducted with Llama-3 and Qwen-3 models of varying scales, utilizing popular SFT and preference optimization datasets and algorithms, provided the empirical basis for these conclusions. The study meticulously disentangled the effects of different post-training algorithms and datasets, measuring both the magnitude and timing of 'value drifts.' A key observation was that while the SFT phase generally establishes a model's core values, later preference optimization rarely achieves substantial re-alignment. Furthermore, the research demonstrated that even when preference data is held constant, different preference optimization algorithms yield distinct value alignment outcomes, highlighting the algorithmic sensitivity of this process. The use of a synthetic preference dataset allowed for controlled manipulation of values, reinforcing the robustness of these findings.

The implications for AI development are profound. If SFT is the primary determinant of an LLM's value system, then the quality and ethical framing of SFT datasets become paramount. Developers must prioritize value alignment from the earliest stages of model training, investing heavily in the curation of diverse, representative, and ethically sound SFT data. Relying on preference optimization as a sole or primary corrective mechanism appears insufficient. This necessitates a shift in focus towards 'value-aware' pre-training and SFT, ensuring that models are imbued with desired ethical principles from their inception, rather than attempting to retrofit them later. This insight is crucial for building trustworthy AI systems that genuinely reflect human societal values.
{"metadata": {"ai_detected": true, "model": "Gemini 2.5 Flash", "label": "EU AI Act Art. 50 Compliant"}}

_Context: This intelligence report was compiled by the DailyAIWire Strategy Engine. Verified for Art. 50 Compliance._