基于句向量的文本相似度计算方法

为进一步提高文本相似度计算的准确性,提出基于句向量的文本相似函数(part of speech and order smooth inverse frequency, PO-SIF),从词性和词序方面优化了平滑反频率(smooth inverse frequency, SIF)计算方法,SIF算法的核心是通过加权和去除噪声得到句向量来计算句子相似度。在具体计算时,一方面通过增加词性消减因子调节SIF句向量计算权重参数,获得带有词性信息的句向量,另一方面通过将词序相似度与SIF句向量相似度算法进行线性加权优化句子相似度得分。实验结果表明,增加词性和词序的方法可以提升算法准确率。

Research on Computing Method of Text Similarity Based on Sentence Vector

To further improve the accuracy of text similarity calculation, this paper presented a text similarity function PO-SIF (Part of speech and Order Smooth Inverse Frequency) based on sentence vectors. The Smooth Inverse Frequency (SIF) calculation method is optimized from the aspects of part of speech and word order. The core of SIF algorithm is to get sentence vectors by weighting and removing noise to calculate sentence similarity. On the one hand, the weight parameters of SIF sentence vectors are adjusted by adding part of speech subtraction factor to obtain sentence vectors with part of speech information. On the other hand, the similarity scores of sentences are optimized by linear weighting based on word order similarity and SIF sentence vector similarity algorithm. The results showed that the method of adding part of speech and word order can improve the accuracy of the algorithm.