基于Spark的大规模社交网络社区发现原型系统

 为有效发掘大规模社交网络上的用户信息，提高对用户之间关系的深入了解，设计开发了基于Spark的大规模社交网络社区发现原型系统。系统利用ActiveMQ实现对大量用户数据的抓取，使用基于Spark的MLlib提供的朴素贝叶斯算法对用户数据进行清洗，利用Spark的GraphX提供的PageRank算法和MLlib提供的Z-Score算法计算用户排名，最终应用并优化LPA算法，将特征相近、联系较密切的用户快速地划分到同一社区中，为进一步分析利用社区用户数据打下了基础。

A large scale social networking community detection prototype system based on Spark

In order to effectively explore the user information in large-scale social networks and improve the understanding of the relationship between users, a community detection prototype system based on Spark is designed and developed. The ActiveMQ is used to acquire a large amount of the user data, taking advantage of the naive Bayesian algorithm provided by Spark-based MLlib to clean the user data, and using the PageRank algorithm provided by Spark-based GraphX and the Z-Score algorithm provided by MLlib to calculate the user ranking. In the prototype system, the LPA algorithm is finally used and optimized, to group the users of similar features and close ties into the same community quickly, as a foundation for further analysis and utilization of the community user data.