Calibrate-Then-Delegate Enhances LLM Safety Monitoring with Cost Guarantees

The imperative for scalable and cost-effective safety monitoring in large language models (LLMs) is addressed by Calibrate-Then-Delegate (CTD), a model-cascade approach that provides probabilistic guarantees on computation cost while enabling instance-level decisions. This innovation is critical for the responsible deployment of LLMs, particularly in high-stakes environments where balancing accuracy with operational expenditure is paramount. Current methods, which often rely on probe uncertainty for delegation, have proven insufficient due to their poor correlation with the actual benefit of escalating a case to a more expensive expert.

CTD introduces a novel delegation value (DV) probe, a lightweight model that operates on the same internal representations as the initial safety probe. Crucially, this DV probe directly predicts the benefit of escalation, moving beyond mere uncertainty. To enforce budget constraints, CTD calibrates a threshold on the DV signal using held-out data via multiple hypothesis testing, thereby yielding finite-sample guarantees on the delegation rate. Evaluated across four safety datasets, CTD consistently outperforms uncertainty-based delegation at every budget level, demonstrating its ability to avoid harmful over-delegation and adapt budget allocation dynamically to input difficulty without requiring explicit group labels.

The strategic implications of CTD are significant for the future of AI safety and governance. By optimizing the trade-off between cost and accuracy, CTD enables organizations to deploy LLMs more confidently, knowing that critical safety issues are being managed efficiently and within predefined budgetary limits. This approach fosters greater trust in AI systems and could influence the development of new regulatory frameworks that mandate transparent and auditable safety monitoring protocols. Ultimately, CTD represents a crucial step towards making advanced AI both powerful and practically governable, ensuring that the economic realities of large-scale AI deployment do not compromise safety standards.