基于改进BP网络的中文歧义字段分词方法研究

文本挖掘中中文歧义字段的自动分词是计算机科学面临的一个难题.针对汉语书写时按句连写,词间无间隙,歧义字段分词困难的特点,对典型歧义中所蕴含的语法现象进行了归纳总结,建立了供词性编码使用的词性代码库.以此为基础,通过对具有特殊语法规则的歧义字段中的字、词进行代码设定,转化为神经网络能够接受的输入向量表示形式,然后对样本进行训练,通过改进BP神经网络的自学习来掌握这些语法规则.训练结果表明:算法在歧义字段分词上达到了93.13%的训练精度和92.50%的测试精度.

Research on ambiguous words segmentation algorithm based on improved BP neural network

In the text mining, the technology of Chinese automatic word segmentation is a difficult problem that the computer science has to face. Aiming at the characteristics of Chinese writing, such as no space between words, continuous writing in sentences and difficulty of segmenting the ambiguous words, the grammatical phenomena are summarized which lie in the typical ambiguity, and the codes library of different parts of speech used for coding is built up. On this basis, words in ambiguity fields with special grammatical rules are set with codes and transformed to the representation form of inputting vector which can be accepted by the neural network. Then the samples are trained and the grammatical rules can be obtained by improving the self-learning of BP neural network. After a lot of training through adopting the BP network, the algorithm reaches 93. 13% of training precision and 92.50% of test precision on ambiguous words segmentation.