DNN Multimodal Fusion Techniques for Predicting Video Sentiment

Williams, Jennifer, Comanescu, Ramona, Radu, Oana and Tian, Leimin (2018) DNN Multimodal Fusion Techniques for Predicting Video Sentiment. ACL 2018: 56th Annual Meeting of the Association for Computational Linguistics, Melbourne Convention and Exhibition Centre, Melbourne, Australia. 15 Jul 2018 - 20 Jul 2020 . 64–72 .

Record type: Conference or Workshop Item (Paper)

Abstract

We present our work on sentiment prediction using the benchmark MOSI dataset from the CMU-MultimodalDataSDK. Previous work on multimodal sentiment analysis have been focused on input-level feature fusion or decision-level fusion for multimodal fusion. Here, we propose an intermediate-level feature fusion, which merges weights from each modality (audio, video, and text) during training with subsequent additional training. Moreover, we tested principle component analysis (PCA) for feature selection. We found that applying PCA increases unimodal performance, and multimodal fusion outperforms unimodal models. Our experiments show that our proposed intermediate-level feature fusion outperforms other fusion techniques, and it achieves the best performance with an overall binary accuracy of 74.0% on video+text modalities. Our work also improves feature selection for unimodal sentiment analysis, while proposing a novel and effective multimodal fusion architecture for this task.

This record has no associated files available for download.

More information

Published date: 1 July 2018

Additional Information: Proceedings of Grand Challenge and Workshop on Human Multimodal Language (Challenge-HML), pages 64–72, Melbourne, Australia. Association for Computational Linguistics.

Venue - Dates: ACL 2018: 56th Annual Meeting of the Association for Computational Linguistics, Melbourne Convention and Exhibition Centre, Melbourne, Australia, 2018-07-15 - 2020-07-20

Related URLs: