支持多语言的自然语言倒序分词最大成词算法

提出一种支持多语言的分词算法,该算法可以按照以下层次来理解：首先,对不同编码的源词库文件编码转换,生成Unicode编码的源词库文件;然后,用Unicode编码的词库文件生成Unicode词库索引;最后,将待分词的自然语句转换成Unicode编码的语句并按照索引倒序分词。该算法已经用C＋＋语言实现,基于此算法实现的分析系统能够自动探测词库的更新并确定是否需要更新索引,能够支持多种编码方式,其编码转换和分词代码与平台无关,分词效率在9MB/s以上,正确率在90%以上。

Maximum Term Segmentation Algorithm in Reverse Order for Multiform Natural Language

The word segmentation algorithm of support multi-language is proposed. The algorithm can be understood according to the following levels: The first is the code conversion,the different source thesaurus documents are turned to Unicode thesaurus documents; Then,Unicode thesaurus file index is generated based on Unicode thesaurus documents; Finally, the natural language will be converted into Unicode encoding, and the words begin to be segmented in a reverse order according to the Unicode thesaurus file index. The algorithm has been completed by using C++ language, and the system can detect the changes of the source thesaurus documents automatically to determine whether there is a need to update the Unicode thesaurus index. The system can support a variety of coding types. The process of code conversion and word segmentation is independent on the platform. The efficiency of the segmentation is more than 9 MB/s , the accuracy rate is more than 90%.