基于辅助模态监督训练的情绪识别神经网络

为了解决多模态数据中数据样本不平衡的问题，利用资源丰富的文本模态知识对资源贫乏的声学模态建模，构建一种利用辅助模态间相似度监督训练的情绪识别神经网络。首先，使用以双向门控单元为核心的神经网络结构，分别学习文本与音频模态的初始特征向量；其次，使用SoftMax函数进行情绪识别预测，同时使用一个全连接层生成2个模态对应的目标特征向量；最后，利用该目标特征向量计算彼此之间的相似度辅助监督训练，提升情绪识别的性能。结果表明，该神经网络可以在IEMOCAP数据集上进行情绪4分类，实现了82.6%的加权准确率和81.3%的不加权准确率。研究结果为人工智能多模态领域的情绪识别以及辅助建模提供了参考依据。

Emotion recognition neural network based on auxiliary modal supervised training

In order to solve the problem of imbalance of data samples in multi-modal data, the resource-rich text modal know-ledge was used to model the resource-poor acoustic mode, and an emotion recognition neural network was constructed by using the similarity between auxiliary modes to supervise training. Firstly, the neural network with bi-GRU as the core was used to learn the initial feature vectors of the text and acoustic modalities. Secondly, the SoftMax function was used for emotion recognition prediction, and simultaneously a fully connected layer was used to generate the target feature vectors corresponding to the two modalities. Finally, the target feature vector assisted the supervised training by calculating the similarity between each other to improve the performance of emotion recognition. The results show that this neural network can perform four emotion classifications on the IEMOCAP data set to achieve a weighted accuracy of 82.6% and an unweighted accuracy of 81.3%. The research result provides a reference and method basis for emotion recognition and auxiliary modeling in the multi-modal field of artificial intelligence.