采用并行遗传算法的文本分割研究

针对短篇幅文本数据稀疏的特性,提出了一种利用外部语料库知识提高短篇幅文本分割准确率的方法.该方法分2个步骤完成:①利用Gibbs采样方法估计语料库对应的潜在狄利克雷分配(LDA)模型,并利用该模型推断目标文本的潜在语义结构信息;②通过定义语义段落内凝聚性和语义段落间发散性2个目标函数,将文本分割问题转化为多目标优化问题.采用一种针对文本分割的并行遗传算法,获得全局最优解.通过实验,在文本数据稀疏的情况下,该算法在准确率方面优于多元判别分析(MDA)方法和基于LDA的文本分割方法,对于提高文本分割的准确率是可行和有效的.

Research of Text Segmentation Based on Parallel Genetic Algorithm

Focusing on the data sparseness of short texts,an algorithm based on knowledge from external corpus iS proposed to improve the accuracy of text segmentation.which contains two steps:Gibbs sampling is adopted to estimate the LDA model;corresponding to the corPus and the latent semantic structure information of the text iS inferred based on the LDA model.Two objective functions of internal cohesion and external dissimilarity are then defined to transform text segmentation into a multi-objective optimization problem.A parallel genetic algorithm based on the objective functions is emloyed to obtain the global optimal solution for text segmentation.According to the experiments,the proposed algorithm achieves higher accuracy than the MDA and LDA-based methods in the case of data sparseness.