LASE: Language-Adversarial Speaker Encoding for Cross-Script Voice Cloning

The introduction of LASE (Language-Adversarial Speaker Encoder) represents a critical advancement in addressing the persistent challenge of speaker identity drift in cross-script voice cloning. Traditional speaker encoders often fail to maintain consistent speaker identity when the same voice is rendered across different linguistic scripts, leading to measurable degradation in cosine similarity. LASE tackles this by employing a language-adversarial training approach, ensuring that speaker embeddings are language-uninformative while remaining highly speaker-informative. This is crucial for developing robust multilingual text-to-speech (TTS) systems that can produce natural and consistent synthetic voices across a global linguistic landscape. The current state of off-the-shelf encoders, particularly when projecting non-Indic trained voices into Indic scripts, highlights a significant gap that LASE aims to bridge.

LASE's technical foundation involves a small projection head layered over a frozen WavLM-base-plus backbone, trained with a dual-loss mechanism. This includes a supervised contrastive loss to reinforce voice identity and a gradient-reversal cross-entropy loss against a 4-language classifier. This adversarial component actively strips language-specific signals from the speaker embedding, thereby enhancing cross-script consistency. Empirical results demonstrate substantial improvements: on a 1118-pair Indian-voice held-out dataset, LASE r1 reduces cross-script drift by a factor of five and achieves three times the headroom over the within-script ceiling. This performance significantly surpasses baselines like WavLM-base-plus-sv and ECAPA-TDNN, which exhibit notable drops in cosine similarity when voices change script. The contribution of both the WavLM backbone and the Gradient-Reversal Layer (GRL) objective is validated through ablation studies, confirming the efficacy of the combined approach.

The implications of LASE extend beyond mere technical improvement; they impact the practical utility and adoption of multilingual AI. By ensuring high fidelity in speaker identity across diverse scripts, LASE enables more convincing and personalized AI agents, virtual assistants, and content localization solutions. This technology is particularly impactful for regions with high linguistic diversity, such as India, where cross-script communication is common. The release of the r1 checkpoint, corpora, and bootstrap recipe also fosters open science, allowing broader research and development. Future applications could include more natural voice interfaces, enhanced accessibility tools for diverse language users, and more authentic digital avatars, pushing the boundaries of human-computer interaction in a globally connected world. However, the ethical considerations surrounding highly realistic voice cloning remain paramount, necessitating robust safeguards against misuse.