The University of Southampton
University of Southampton Institutional Repository

Video modeling and learning on Riemannian manifold for emotion recognition in the wild

Video modeling and learning on Riemannian manifold for emotion recognition in the wild
Video modeling and learning on Riemannian manifold for emotion recognition in the wild
In this paper, we present the method for our submission to the emotion recognition in the wild challenge (EmotiW). The challenge is to automatically classify the emotions acted by human subjects in video clips under real-world environment. In our method, each video clip can be represented by three types of image set models (i.e. linear subspace, covariance matrix, and Gaussian distribution) respectively, which can all be viewed as points residing on some Riemannian manifolds. Then different Riemannian kernels are employed on these set models correspondingly for similarity/distance measurement. For classification, three types of classifiers, i.e. kernel SVM, logistic regression, and partial least squares, are investigated for comparisons. Finally, an optimal fusion of classifiers learned from different kernels and different modalities (video and audio) is conducted at the decision level for further boosting the performance. We perform extensive evaluations on the EmotiW 2014 challenge data (including validation set and blind test set), and evaluate the effects of different components in our pipeline. It is observed that our method has achieved the best performance reported so far. To further evaluate the generalization ability, we also perform experiments on the EmotiW 2013 data and two well-known lab-controlled databases: CK+ and MMI. The results show that the proposed framework significantly outperforms the state-of-the-art methods.
113–124
Liu, Mengyi
675d70e6-dc60-47f2-8da3-4fd7f6b6e297
Wang, Ruiping
d44a3866-4f48-4323-bba6-3bcb145ed34a
Li, Shaoxin
a371fce9-d471-4020-9fc0-2a21a7e49d19
Huang, Zhiwu
84f477cd-9097-44dd-a33e-ff71f253d36b
Shan, Shiguang
72278811-5f18-4dc9-ab05-64668aaee9ad
Chen, Xilin
094f7c27-74a6-44e2-80c3-c1a2b0db0a40
Liu, Mengyi
675d70e6-dc60-47f2-8da3-4fd7f6b6e297
Wang, Ruiping
d44a3866-4f48-4323-bba6-3bcb145ed34a
Li, Shaoxin
a371fce9-d471-4020-9fc0-2a21a7e49d19
Huang, Zhiwu
84f477cd-9097-44dd-a33e-ff71f253d36b
Shan, Shiguang
72278811-5f18-4dc9-ab05-64668aaee9ad
Chen, Xilin
094f7c27-74a6-44e2-80c3-c1a2b0db0a40

Liu, Mengyi, Wang, Ruiping, Li, Shaoxin, Huang, Zhiwu, Shan, Shiguang and Chen, Xilin (2015) Video modeling and learning on Riemannian manifold for emotion recognition in the wild. Journal On Multimodal User Interfaces, 113–124. (doi:10.1007/s12193-015-0204-5).

Record type: Article

Abstract

In this paper, we present the method for our submission to the emotion recognition in the wild challenge (EmotiW). The challenge is to automatically classify the emotions acted by human subjects in video clips under real-world environment. In our method, each video clip can be represented by three types of image set models (i.e. linear subspace, covariance matrix, and Gaussian distribution) respectively, which can all be viewed as points residing on some Riemannian manifolds. Then different Riemannian kernels are employed on these set models correspondingly for similarity/distance measurement. For classification, three types of classifiers, i.e. kernel SVM, logistic regression, and partial least squares, are investigated for comparisons. Finally, an optimal fusion of classifiers learned from different kernels and different modalities (video and audio) is conducted at the decision level for further boosting the performance. We perform extensive evaluations on the EmotiW 2014 challenge data (including validation set and blind test set), and evaluate the effects of different components in our pipeline. It is observed that our method has achieved the best performance reported so far. To further evaluate the generalization ability, we also perform experiments on the EmotiW 2013 data and two well-known lab-controlled databases: CK+ and MMI. The results show that the proposed framework significantly outperforms the state-of-the-art methods.

This record has no associated files available for download.

More information

Published date: 11 November 2015

Identifiers

Local EPrints ID: 501113
URI: http://eprints.soton.ac.uk/id/eprint/501113
PURE UUID: dcf9ec74-3d07-4b0f-a885-015826ab7661
ORCID for Zhiwu Huang: ORCID iD orcid.org/0000-0002-7385-079X

Catalogue record

Date deposited: 23 May 2025 17:18
Last modified: 25 May 2025 05:21

Export record

Altmetrics

Contributors

Author: Mengyi Liu
Author: Ruiping Wang
Author: Shaoxin Li
Author: Zhiwu Huang ORCID iD
Author: Shiguang Shan
Author: Xilin Chen

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×