Covariance pooling for facial expression recognition

Abstract

Classifying facial expressions into different categories
requires capturing regional distortions of facial landmarks.
We believe that second-order statistics such as covariance is
better able to capture such distortions in regional facial features. In this work, we explore the benefits of using a manifold network structure for covariance pooling to improve
facial expression recognition. In particular, we first employ
such kind of manifold networks in conjunction with traditional convolutional networks for spatial pooling within individual image feature maps in an end-to-end deep learning
manner. By doing so, we are able to achieve a recognition
accuracy of 58.14% on the validation set of Static Facial
Expressions in the Wild (SFEW 2.0) and 87.0% on the validation set of Real-World Affective Faces (RAF) Database1
.
Both of these results are the best results we are aware of.
Besides, we leverage covariance pooling to capture the temporal evolution of per-frame features for video-based facial
expression recognition. Our reported results demonstrate
the advantage of pooling image-set features temporally by
stacking the designed manifold network of covariance pooling on top of convolutional network layers

More information

Identifiers

ORCID for Zhiwu Huang: orcid.org/0000-0002-7385-079X

Catalogue record

Export record

Altmetrics