文献类型：期刊文献

英文题名：Multimodal emotion recognition based on feature selection and extreme learning machine in video clips

作者：Pan, Bei[1]; Hirota, Kaoru[1]; Jia, Zhiyang[1]; Zhao, Linhui[2,3]; Jin, Xiaoming[2,3]; Dai, Yaping[1]

第一作者：Pan, Bei

通讯作者：Jia, Zhiyang

机构：[1] School of Automation, Beijing Institute of Technology, Beijing, 100081, China; [2] College of Robotics, Beijing Union University, Beijing, 100020, China; [3] Beijing Engineering Research Center of Smart Mechanical Innovation Design Service, Beijing, 100020, China

第一机构：School of Automation, Beijing Institute of Technology, Beijing, 100081, China

年份：2021

外文期刊名：Journal of Ambient Intelligence and Humanized Computing

收录：EI(收录号：20213110709805);Scopus(收录号：2-s2.0-85111486941)

基金：This work was supported by the Open Foundation of Beijing Engineering Research Center of Smart Mechanical Innovation Design Service under Grant No. KF2019302, the General Projects of Science and Technology Plan of Beijing Municipal Commission of Education under Grant No. KM202011417005, and the National Talents Foundation under Grant No. WQ20141100198.

语种：英文

外文关键词：Knowledge acquisition - Machine learning - Optimization

摘要：Multimodal fusion-based emotion recognition has attracted increasing attention in affective computing because different modalities can achieve information complementation. One of the main challenges for reliable and effective model design is to define and extract appropriate emotional features from different modalities. In this paper, we present a novel multimodal emotion recognition framework to estimate categorical emotions, where visual and audio signals are utilized as multimodal input. The model learns neural appearance and key emotion frame using a statistical geometric method, which acts as a pre-processer for saving computation power. Discriminative emotion features expressed from visual and audio modalities are extracted through evolutionary optimization, and then fed to the optimized extreme learning machine (ELM) classifiers for unimodal emotion recognition. Finally, a decision-level fusion strategy is applied to integrate the results of predicted emotions by the different classifiers to enhance the overall performance. The effectiveness of the proposed method is demonstrated through three public datasets, i.e., the acted CK+ dataset, the acted Enterface05 dataset, and the spontaneous BAUM-1s dataset. An average recognition rate of 93.53% on CK+, 91.62% on Enterface05, and 60.77% on BAUM-1s are obtained. The emotion recognition results acquired by fusing visual and audio predicted emotions are superior to both recognition of unimodality and concatenation of individual features. ? 2021, The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature.