深度学习中对比散度算法的有偏性分析

为了给对比散度算法的进一步优化提供理论指导,尝试从理论上分析对比散度算法的收敛性。首先从仅含4个结点的玻尔兹曼机入手,利用单纯形表征模型的概率空间,以及流形表征概率空间与模型参数的关系,形象地表示了对比散度算法和极大似然算法的收敛过程,并从理论上推导出对比散度算法的收敛集与极大似然算法的收敛集之差不为空,从而证明了对比散度算法的有偏性。基于该结论,设计了一种先利用对比散度算法进行预训练,再利用极大似然算法调优的训练策略。实验结果表明,在应用该策略获得同等收敛效果的条件下,训练迭代步骤降低了83.3%。

Biasness of contrastive divergence in deep learning

Some theoretical problems on the convergence property of the contrastive divergence (CD) algorithm were investigated,providing theoretical guidance for optimizing the CD algorithm. Simplex was used to represent the probability space of the model, and manifold used to represent the relationship between the probability space and parameters of the model. Both of them help to reveal the convergence process visually. Compared with the results from normal maximum likelihood estimation (MLE) for a Boltzmann machine with only 4 nodes, the CD algorithm actually has biasness. Based on this conclusion, a new training strategy of CD pre-training followed by MLE fine-tuning was designed. The experimental results show that, in the same convergence condition, the procedure of the algorithm with the new strategy is reduced by 83.3% compared with the traditional algorithm.