基于Spark框架的大数据K-prototypes聚类算法

大数据具有数据量大及混合类型的属性,基于MapReduce的K-prototypes并行大规模混合数据方案的缺点是时间和内存的限制,导致这些方案不适合处理大数据.为了解决这个问题,该文提出一种新的基于Spark的k-prototypes聚类方法,该方法使用了重新聚集技术,利用Spark框架的内存操作来构建大规模混合数据分组.在模拟和实际数据集上的实验表明,该文方法可行,且提高了现有K-prototypes方法的效率.

K-prototypes Clustering Algorithm Based on Spark Framework for Big Data

Big data has a large amount of data and mixed types of attributes. The disadvantages of the current MapReduce-based K-prototypes parallel large-scale hybrid data plan are the limitations of time and memory, making these solutions unsuitable for processing big data. To solve this problem, a new Spark-based K-prototypes clustering method has been proposed in this paper. In this method, the re-aggregation technique and Spark''s memory operations have been used to build large-scale mixed data groups. Experiments on simulated and actual datasets show that this method is feasible and improves the efficiency of the existing K-prototypes method.