一种频率增强的语句语义相似度计算

目前,在基于HowNet进行语句语义相似度计算的算法中,没有考虑语句中的不同词语对语句之间相似度值的不同贡献程度,以致计算结果不理想.为了更好地解决上述缺陷,提出了一种频率增强语句语义相似度算法.该算法利用HowNet作为词典库,在同时考虑义原距离和义原深度的条件下,进行词语相似度计算;在此基础上算法进一步将词语在语料库中的频率函数作为权重值,引入至语句的语义相似度计算中,以降低高频率词语在语句相似度值中的比重.实验表明,改进的算法在语句相似度计算结果上与人们的主观判断更接近,结果更合理.

A Frequency Enhanced Algorithm of Sentence Semantic Similarity

Sentence semantic similarity algorithms based on HowNet ignored the fact that different words have different contribution weight to sentence similarity value, and therefore, the similarity result is not quite reasonable. In order to solve this problem, we proposed an improved algorithm based on word frequency. The algorithm calculates the similarity between words based on HowNet, both considering the distance and the height of primitives. Then, a frequency function of words in corpus as a weight factor is embedded into the sentence semantic similarity algorithm, which reduces the proportion value that the high frequency words devote to sentence similarity calculation. The sentence semantic similarity experiment results show that the improved algorithm is much better in rationality as well as in matching with people''s subjective judgment.