基于语境相似度的中文分词一致性检验研究

提出一种基于语境相似度的中文分词一致性检验方法。首先利用词法和句法层面的特征, 设计基于构词、词性和依存句法的分类规则, 再使用预训练词向量, 对不一致字串所在语境的语义信息进行编码, 通过语境间的语义相似度对不一致字串进行分类。在人工构建的36万字分词语料库中进行分词一致性检验, 结果表明该方法能够有效地提高中文分词一致性检验的准确率。进一步地, 使用3 种主流中文分词模型在修正一致性后的分词语料中重新训练和测试, 结果表明该方法可以有效地提高分词语料库的质量, 3种中文分词模型的F1值分别提高1.18%, 1.25%和1.04%。

Consistency Check for Chinese Word Segmentation via Contextual Similarity

The authors propose a method of consistency check for Chinese word segmentation based on contextual similarity. First, the classification constraints based on word formation, part of speech and dependency syntax are designed by using the features of morphology and syntax. Then, the semantic information of the context in which the inconsistent strings are located is encoded by using pretrained word embeddings, and the inconsistent strings are classified by semantic similarity between contexts. Experimental results show that proposed method can effectively improve the accuracy of consistency check for Chinese word segmentation. Further, three mainstream Chinese word segmentation models are used to re-implement in the revised Chinese word segmentation corpus. The result shows that proposed method can effectively improve the quality of Chinese word segmentation corpus, and the F1 scores of three Chinese word segmentation models are improved by 1.18%, 1.25% and 1.04% respectively.