基于唇重构与三维耦合CNN的多视角音唇一致性判别

针对传统音唇一致性判别方法主要对正面唇动视频进行处理，未考虑视频采集角度变化对结果的影响，且容易忽略唇动过程中的时空特性等不足，文中以唇部角度变化对一致性判别的影响为研究重心，结合三维卷积神经网络在非线性表示和时空维度特征提取上的优势，提出了基于正面唇重构与三维耦合卷积神经网络的多视角音唇一致性判别方法。该方法先通过在生成器中引入自映射损失来提高正面重建效果，并采用基于自映射监督循环一致性生成对抗网络（SMS-CycleGAN）的唇重构方法对多视角唇图进行角度分类及正面重构；然后设计两个异构三维卷积神经网络，分别用来描述音频和视频信号，并提取包含长时时空关联信息的三维卷积特征；最后引入对比损失函数作为音视频信号匹配的相关度鉴别度量，将音视频网络输出耦合到同一表示空间，并进行一致性判别。实验结果表明，文中方法能重建出更高质量的正面唇图，一致性判别性能优于多种不同类型的比较方法。

Multi-View Lip Motion and Voice Consistency Judgment Based on Lip Reconstruction and Three-Dimensional Coupled CNN

The traditional consistency judgment methods of lip motion and voice mainly focus on processing the frontal lip motion video，without considering the impact of angle changes on the result during the video acquisition process. In addition, they are prone to ignoring the spatio-temporal characteristics of the lip movement process．Aiming at these problems, this paper focused on the influence of lip angle changes on consistency judgment，combined the advantages of three dimensional convolutional neural networks for non-linear representation and spatio-temporal dimensional feature extraction, and proposed a multi-view lip motion and voice consistency judgment method based on frontal lip reconstruction and three dimensional（3D） coupled convolutional neural network．Firstly，the self-mapping loss was introduced into the generator to improve the effect of frontal reconstruction, and then the lip reconstruction method based on self-mapping supervised cycle-consistent generative adversarial network (SMS-CycleGAN) was used for angle classification and frontal reconstruction of multi-view lip image．Secondly，two heterogeneous three dimensional convolution neural networks were designed to describe the audio and video signals respectively, and then the 3D convolution features containing long-term spatio-temporal correlation information were extracted．Finally, the contrastive loss function was introduced as the correlation discrimination measure of audio and video signal matching, and the output of the audio-video network was coupled into the same representation space for consistency judgment. The experimental results show that the method proposed in this paper can reconstruct frontal lip images of higher quality， and it is better than a variety of comparison methods on the performance of consistency judgment．