Provable Speech Attributes Conversion
via Latent Independence

Model

Indepedence-based Voice Conversion (IVC) consists of an encoder predicting content-like latent variable Ŝ and a decoder conditioned on speaker, emotion, or loudness embeddings C. The model is trained with discrepancy loss R and independence loss I

• Provides theoretical guarantees: recovering the latent variable up to an invertible map is sufficient to guarantee correct conversion.
• Derives an explicit conversion error bound under imperfect reconstruction & independence.
• Simplified training: trains using only two MSE losses — no GANs, no VQ codebooks, no adversarial training.
• Supports speaker, emotion, [pitch] and loudness conversion — individually or jointly.
• Performs one-shot inference from a single reference utterance (≥ 3 seconds) .
• Linear-time inference, improving KNN-VC’s O(n²) complexity.
• Achieves competitive or better results vs. KNN-VC, Free-VC, and emotion conversion baselines.

While signal conversion and disentangled representation learning have shown promise for manipulating data attributes across domains such as audio, image, and multimodal generation, existing approaches, especially for speech style conversion, are largely empirical and lack rigorous theoretical foundations to guarantee reliable and interpretable control. In this work, we propose a general framework for speech attribute conversion, accompanied by theoretical analysis and guarantees under reasonable assumptions. Our framework builds on a non-probabilistic autoencoder architecture with an independence constraint between the predicted latent variable and the target controllable variable. This design ensures consistent signal transformation conditioned on an observed style variable, preserving the original content while modifying the desired attribute. We further demonstrate the versatility of our method by evaluating it across a range of speech styles beyond speaker identity, including emotion, loudness, and pitch contour. Quantitative evaluations confirm the effectiveness and generality of the proposed approach.