基于MapReduce的并行LAD模型评论主题提取算法研究

针对传统的潜在狄利克雷分析(LDA)模型在提取评论主题时存在着计算时间长、计算效率低的问题,提出基于MapReduce架构的并行LAD模型建立方法.在文本预处理的基础上,得到文档-主题分布和主题-特征词分布,分别计算主题相似度和特征词权重,结合k-均值聚类算法,实现评论主题提取的并行化.通过Hadoop并行计算平台进行实验,结果表明,该方法在处理大规模文本时能获得接近线性的加速比,对主题模型的建立效果也有提高.

Research on topic extraction algorithm based on MapReduce parallel LAD model

In the face of non structured massive comment text, Latent Dirichlet Allocation (LDA) model in extracting thematic reviews exist long computing time and computing the problem of low efficiency. To solve this problem, a method for establishing the parallel LDA model based on MapReduce framework has been putforwardand the parallelization of the extraction of topic comment has been realized. The experimental results show that the method can achieve near linear speedup in processing large scale text, and the effect of the model is improved by Hadoop parallel computing platform.