一种高性能英文词性标注器的设计与实现

针对统计和规则方法各自的优点和局限,提出运用Viterbi和FTBL(fast transformation-based learning)算法相级联的算法,实现一种英文自动词性标注器.该级联方法以FTBL算法为整体算法,在它的规则学习和最终标注两个阶段,均以Viterbi算法作为其初始化过程.实验结果表明此算法优于其中任何一种单独的算法,达到了98%的高准确率,验证了自然语言处理中统计与规则并举的主流设计思想.

Design and Realization of a High-Performance Part of Speech Tagger for the English Language

In view of the respective strength and weakness of the statistical method and rule governed method, a kind of English part of speech tagger based on a cascade of Viterbi and FTBL algorithms is proposed. Its design idea and realization process are discussed. This cascade method views the FTBL algorithm as its whole algorithm, applying Viterbi algorithm to its initialization process during its two phases rule learning and final tagging. The results show that this method excels either of the separate algorithms, achieving high accuracy of over 98%, and validates the mainstream design idea as a combination of statistical and rule governed methods in natural language processing.