OPUS: Efficient Data Selection for LLM Pre-Training

Researchers have proposed OPUS (Optimizer-induced Projected Utility Selection), a dynamic data selection framework designed to improve the efficiency of large language model (LLM) pre-training. As the availability of high-quality public text diminishes, pre-training is shifting towards selecting better tokens rather than simply using more tokens. OPUS addresses this challenge by defining data utility in the optimizer-induced update space, scoring candidates based on their effective updates projected onto a target direction derived from a stable, in-distribution proxy. To ensure scalability, OPUS employs Ghost technique with CountSketch and Boltzmann sampling, incurring only 4.7% additional compute overhead. Experiments demonstrate that OPUS outperforms industrial-level baselines in GPT-2 pre-training and achieves superior performance in Qwen3-8B-Base continued pre-training using fewer tokens, highlighting its potential for significant data efficiency gains.

Transparency Disclosure: This analysis was conducted by an AI, focusing on factual information and avoiding subjective claims. The AI is trained to provide objective insights based on available data.