基于统计词典和特征加强的多语言文本分类

在统计双语词典的基础上,提出一种特征加强的多语言文本分类方法.在执行文本分类时,考虑到其他语言的训练文本,使得多种语言的文本集合中均存在训练文本,放松了MLTC的要求.特征加强是一种交叉检查过程,即获取两种语言所有特征的卡方统计后,通过语言中相关特征的辨识力,再次对语言的特征辨识力进行评估,以提高分类的可信度.实验选择汉语或英语作为目标语言.实验结果表明:提出的方法具有更高的分类精度,且对训练集规格的敏感度更低.

Multiple Language Text Classification Method Based on Statistical Dictionary and Feature Enhancing

Aiming at the problem that multiple language text classification (MLTC) can only solve single language text classification problem of multiple independent, on the basic of statistical bilingual dictionary, multiple language text classification based on feature enhancing has been proposed. In the implementation of text classification, the training texts of other languages have been taken into account, which makes the text of a variety of languages in the training texts. And it relaxes MLTC requirements. Feature enhancing is a processing of cross examination. After chi square statistics of all the features for the two languages is obtained, the identification of language feature is reassessed through the feature identification to improve the reliability of classification. Chinese or English is chosen as the target language in the experiment. Experimental results show that the proposed method has a higher classification accuracy, and the sensitivity of the training set is lower.