Exploring Disentanglement with Multilingual and Monolingual VQ-VAE

Williams, Jennifer, Fong, Jason, Cooper, Erica and Yamagishi, Junichi (2021) Exploring Disentanglement with Multilingual and Monolingual VQ-VAE. 11th ISCA Speech Synthesis Workshop, , Budapest, Hungary. 26 - 28 Aug 2021.

Abstract

This work examines the content and usefulness of disentangled
phone and speaker representations from two separately trained
VQ-VAE systems: one trained on multilingual data and another
trained on monolingual data. We explore the multi- and monolingual models using four small proof-of-concept tasks: copysynthesis, voice transformation, linguistic code-switching, and content-based privacy masking. From these tasks, we reflect on how disentangled phone and speaker representations can be
used to manipulate speech in a meaningful way. Our experiments demonstrate that the VQ representations are suitable for these tasks, including creating new voices by mixing speaker representations together. We also present our novel technique to conceal the content of targeted words within an utterance by
manipulating phone VQ codes, while retaining speaker identity and intelligibility of surrounding words. Finally, we discuss recommendations for further increasing the viability of disentangled representations

This record has no associated files available for download.

Contributors

Author: Jennifer Williams

Download statistics