An investigation into facial depth data for audio-visual speech recognition

Bleeck, Stefan and Ralph-Donaldson, Travis James Francis Paul (2022) An investigation into facial depth data for audio-visual speech recognition. Hearing, Audio and Audiology Sciences Meeting, , Southampton, United Kingdom. 12 - 13 Sep 2022.

Abstract

Recent SOTA (state of the art) AVSR (Audio Visual Speech Recognition) systems such as Meta’s AV-Hubert have highlighted the superior efficacy of multi-modal speech recognition when compared to audio-based implementations, especially in noisy conditions. However, planar feature extraction methods are still susceptible to variable lighting conditions and skin tones. Moreover, these AVSR systems are currently unable to accurately classify visemes (visual phonemes) to phonemes with a one-one taxonomy. One potential avenue of research to prevent both these shortcomings is the application of newer RGB-D cameras (analogous to the Microsoft’s Kinect Sensor) to extract more comprehensive facial speech data that is both lighting and skin tone invariant. Depth data also includes additional more differentiable speech information pertaining to phonemes that involve lip protrusion, such as rounded vowels, that may allow for a more accurate discrimination between visemes. The current RGB-D AVSR literature has yet to thoroughly explore the applicability of the depth modality in more challenging classification tasks, such as continuous and free speech and has been limited to mostly smaller speaker-dependent datasets containing only individual words or phrases. This study will investigate the depth modality's influence on speech classification, using a bespoke broadly generalisable multi-modal speaker-independent dataset. This will contain both continuous and free speech, in a rigorous attempt to assess the depth modality's robustness against these more challenging classification tasks. This paper will then compare the proposed RGB-D facial dataset with current planar AVSR implementations and robustly evaluate the inherent benefits and potential shortcomings of each multi-modal AVSR system.

This record has no associated files available for download.

Contributors

Author: Stefan Bleeck

Download statistics