PDM:基于Hadoop的并行数据分析系统

提出了一款基于Hadoop的并行数据分析系统——PDM.该系统拥有大量以MapReduce为计算框架的并行数据分析算法,不仅包括传统的ETL、数据挖掘、数据统计和文本分析算法,还引入了基于图理论的SNA(社会网络分析)算法.详细阐述了并行多元线性回归算法和多源最短路径算法的原理和实现,其中,提出的消息传递模型能有效解决MapReduce难以处理邻接矩阵的问题;介绍了基于电信数据的典型应用,如采用并行k均值和决策树算法实现的套餐推荐,利用并行PageRank算法实现的营销关键点发现等;最后通过性能测试,说明该系统适合高效地处理大规模数据.

PDM:A Parallel Data Analysis System Based on Hadoop

A PDM(Parallel Data Mining) system was built based on Hadoop.PDM contains a large number of parallel data analysis algorithms based on MapReduce computational framework.These algorithms not only contain the classic algorithms of ETL,data mining,data statistical and text analysis,but also introduce SNA(social network analysis) based on graph mining.The principle and implementation of the parallel multiple linear regression algorithm and the multi-source shortest path algorithm were described and the Message-passing model proposed can effectively solve the problem that MapReduce is difficult to deal with the adjacency matrix structure.This paper also illustrates some typical applications of telecommunications,such as the Business recommendation based on parallel k-means and decision tree algorithms,the Marketing key points discovery based on parallel PageRank algorithm and the like.Finally,the results of performance test show that the proposed system is suitable for dealing with large scale data efficiently.