文本分类中基于CHI和PCA混合特征的降维方法

中文文本数据的半结构化甚至非结构化的特点使得其分类存在着特征高维的问题,传统单一的特征降维方法难以满足大数据时代的文本分类需求.基于此,提出了一种基于卡方统计(Chi-square statistics,CHI)和主成分分析(principal component analysis,PCA)的混合特征降维方法(CHI-...

Research on dimension reduction method based on mixed features of CHI and PCA in text classification

The semi-structured and even unstructured characteristics of Chinese text data make its classification have high-dimensional features. The traditional single feature dimensionality reduction method is difficult to meet the text classification needs in the era of big data. Based on this, a hybrid feature dimension reduction method (CHI-PCA) based on Chi-square statistics (CHI) and principal component analysis(PCA) is proposed. This method uses the CHI method to initially screen out category-related feature words, and then the PCA method is used to perform two-dimensional dimensionality reduction on the feature word space. While reducing the feature dimensions, it still retains the most feature information of the original feature space. After comparison experiments with traditional feature dimensionality reduction methods document frequency (DF), Information gain (IG), CHI and PCA methods, the results show that under different feature dimensions, the overall classification effect of the proposed method under Softmax regression, support vector machines(SVM) classification and KNN classifier is better than the comparison method. The Macro-F1 value is improved by up to 2.7%, and the classification performance in each category is also considerable. This shows that the two-stage feature dimensionality reduction method based on CHI-PCA is feasible. It improves the classification performance while reducing the dimensionality of features.