一种基于SVM和规则消除组合型歧义的算法

歧义的处理是影响分词系统切分精度的重要因素,也是中文自动分词系统中较为困难的问题.提出一种基于支持向量机(SVM)和规则(Rules)相结合的算法(SR算法),对中文分词过程中的组合型歧义字段进行排歧.SR算法主要思想是利用支持向量机分类的原理,结合词性搭配的一些规则排除组合型歧义.实验表明,用此算法对几种不同的语料进行排歧测试,排除组合型歧义字段的切分正确率可达83%左右,为解决中文自动分词难题提供了一条新路径.

Algorithm Based on SVM and Rules for the Disambiguation of Combinatorial Ambiguous Phrases

Ambiguity processing is an essential but difficult problem in word segmentation.The authors combine Support Vector Machines(SVM)with rules to propose a new algorithm(SR algorithm)to deal with the combinatorial ambiguous phrases in Chinese word segmentation.The key idea of the SR algorithm is to solve combinatorial ambiguous phrases making use of the theory of SVM and rules of parts of speech.In a test of several kinds of Chinese corpus,it indicates that the accuracy of segmentation for combinatorial ambiguous phrases reach 83%.It provides a new method for solving Chinese word segmentation problems.