基于组合型中文分词技术的改进

在分词过程中如何处理歧义切分是中文分词算法要解决的难点之一.文中提出了一种改进的基于词典和基于统计的组合中文分词算法,能够发现并处理交集型歧义,该算法在发现歧义的过程中没有采取传统的双向匹配法而是采用双栈的结构,减少了匹配花费时间,并分别采取长词优先和最大概率方法处理一般的交集型歧义和特殊的同词长交集型歧义.最后通过实例对文中所提出的算法进行实验验证,结果表明该算法比传统的分词算法有更好的准确率.

Improvement of Chinese Word Segmentation Based on Combination Method

How to deal with ambiguity in the segmentation process is a challenging issue that requires Chinese word segmentation algorithms to solve it.This paper proposes an improved dictionary and statisticsbased Chinese word segmentation combination algorithm that can discovery and solve the crossing ambiguity.This algorithm adopts dual stack structure rather than traditional bidirectional matching method to discover ambiguity with less matching time.Furthermore,the algorithm takes methods ＂choosing longer word＂ and ＂choosing word with maximum probability＂ respectively to deal with general crossing ambiguity and special crossing ambiguity with equal length.Finally,it was verified by case studies that the proposed algorithm has better accuracy than traditional word segmentation algorithms.