基于词条组合的中文文本分词方法

针对汉语自动分词后词条的特征信息缺失的问题,提出以词串为分词单位的中文文本分词方法,将整个分词过程分解为三个子过程:首先,采用逆向最大匹配法对文本进行切分;第二,对切分结果进行停用词消除;第三,计算第一次分词得到的词条互信息和相邻共现频次,根据计算结果判定相应的词条组合成词串。实验结果表明,词条组合后的词串的语义信息更丰富,有助于文本特征选择效果的改善和文本分类性能的提高。

Word Combination Based Chinese Word Segmentation Methodology

Since the automatic of Chinese word will bring the lack of information,method of word segmentation according to lexical chunk as segmentation unit are proposed,such segmenting process divided are into three sub-process:firstly,text segmentec by means of Backward Maximum Matching.Second,the stop-words is deleted from the segmentation result.At last,count words mutual information and adjacency by the first time segment words,and then,according to this counting result the lexical chunk can judge and sign by re...