基于延迟决策和斜率的新词识别方法

采用词典分词时会遇到未登录词的识别问题.本文提出一种新的中文新词识别方法，用于全文信息检索系统索引的建立.在索引切分过程中遇到无法切分字串时，暂存为未切分串，并生成统计信息.待未切分串达到一定数量时，再利用生成的统计信息结合斜率（加速度）的方法来切分暂存的未切分串.切分的结果可以进入索引，对于出现频率高的切分片段可以提取加入词典.

A method of Chinese new word identification based on delayed decision-making and slope

The problem of identification of OOV (out of Vocabulary) words can be met when segmenting Chinese document with a dictionary.The paper provides a new method of new word identification that is used to index for IR (Information Retrieval) system.When meeting character sequence that cannot be segmented by dictionary,we first store them as uncut sequence and distill statistic information,and then,as the amount of uncut sequence grows large enough,segment the uncut sequence using the statistic information and slope (acceleration) method.The result of the segmentation can be put into indexes,while the sequence with high frequency can be added to dictionary.