语料库自然标注信息与中文分词应用研究

以中文分词为应用目标, 将大规模语料库上存在的自然标注信息分为显性标注信息与隐性标注信息, 分别考察了它们的分布和对大数据集上语言计算的影响。结果表明, 两者都直接或间接地表达了作者对语言的分割意志, 因而对分词具有积极的影响。通过词语抽取测试, 发现在缺乏丰富显性标注信息的文本中, 来自语言固有规律的自然标注信息对字符串有着强大的分割性能。

Natural Annotation Research in Large-Scale Corpora with a Focus on Chinese Word Segmentation

The distribution and meaning of natural annotations on large datasets are discussed. The proposed research on word extraction shows the positive potential of both implicit and explicit natural annotation in word segmentation. Experiments on word extraction indicates that the implicit natural annotation derived from language laws and patterns are more powerful in splitting character strings in raw corpora.