Improved prosody from Learned F0 Codebook representations for VQ-VAE speech waveform reconstruction

Zhao, Yi, Li, Haoyu, Lai, Cheng-I, Williams, Jennifer, Cooper, Erica and Yamagishi, Junichi (2020) Improved prosody from Learned F0 Codebook representations for VQ-VAE speech waveform reconstruction. Proceedings of the Annual Conference of the International Speech Communication Association: INTERSPEECH 2020, Shanghai, Shanghai, China. 25 Oct 2020 - 29 Oct 2022 . pp. 4417-4421 .

Abstract

Vector Quantized Variational AutoEncoders (VQ-VAE) are a powerful representation learning framework that can discover discrete groups of features from a speech signal without supervision. Until now, the VQ-VAE architecture has previously modeled individual types of speech features, such as only phones or only F0. This paper introduces an important extension to VQ-VAE for learning F0-related suprasegmental information simultaneously along with traditional phone features. The proposed framework uses two encoders such that the F0 trajectory and speech waveform are both input to the system, therefore two separate codebooks are learned. We used a WaveRNN vocoder as the decoder component of VQ-VAE. Our speakerindependent VQ-VAE was trained with raw speech waveforms from multi-speaker Japanese speech databases. Experimental results show that the proposed extension reduces F0 distortion of reconstructed speech for all unseen test speakers, and results in significantly higher preference scores from a listening test. We additionally conducted experiments using single-speaker Mandarin speech to demonstrate advantages of our architecture in another language which relies heavily on F0. Index Terms: VQ-VAE, speech synthesis, prosody, representation learning

This record has no associated files available for download.

Contributors

Author: Jennifer Williams

Download statistics