Mappa: Fine-Tune Multi-Agent LLMs with AI Coaches

Mappa introduces a novel approach to fine-tuning multi-agent LLM systems by employing an external LLM as a coach. The core problem addressed is the difficulty in assigning responsibility when multiple agents are working together and an error occurs. Traditional reinforcement learning provides a single reward at the end, making it challenging to identify which agent was at fault. Mappa solves this by having an external LLM, such as Gemini, observe each agent's actions and tool outputs and assign per-action scores. This provides a dense training signal without requiring ground truth labels. The framework is designed to be general, allowing users to plug in their own agents, tasks, and coach models. The trained models can be run offline, reducing reliance on API calls. The results demonstrate significant improvements in performance, including a +17 percentage point increase on the AIME math competition and a +38% F1 score on Kaggle-style data science tasks. The hardware requirements, however, may be a barrier for some users. Overall, Mappa represents a promising step towards more effective and efficient training of multi-agent AI systems.