Arcee AI Releases Trinity-Large-Preview: A 398B Parameter MoE Model

Arcee AI's release of Trinity-Large-Preview marks a notable contribution to the field of large language models. With 398 billion parameters and a sparse Mixture-of-Experts architecture, this model demonstrates the potential of scaling language models while maintaining efficiency. The model's training on 17 trillion tokens suggests a comprehensive understanding of language and the world. The use of a sparse MoE configuration, with 256 experts and 4 active experts per token, allows the model to selectively engage different parts of its knowledge base for different tasks. The extended context length of 512k enables the model to process and understand longer sequences of text, which is crucial for tasks such as document summarization and question answering. The benchmark results indicate strong performance on MMLU, but also highlight areas for improvement on MMLU-Pro and GPQA-Diamond. The model's availability on platforms like OpenRouter and LM Studio makes it accessible to a wider audience. The Apache 2.0 license encourages community contributions and further development. Overall, Trinity-Large-Preview represents a significant step forward in large language model research and development, offering a powerful tool for various natural language processing applications.