面向R语言的分布式流处理系统设计与实现

为了使得在数据分析领域广泛使用的R语言能够应用于大规模数据的分布式流处理计算,基于Spark Streaming框架,设计和实现了一个面向R语言的分布式流处理系统。系统提出了一套完备的R/Java跨语言交互模型,实现了R语言程序和Java程序的互相调用,从而使得R语言程序可以运行在Java平台上,改进了R语言在大规模数据处理和并发计算等方面的不足。系统也提供了Spark Streaming的R语言编程接口,提高了后者的易用性。通过实例验证了该系统的可行性和实用性。

Design and Implementation of the Distributed Stream Processing System for R Language

R language is widely used in data analysis. In order to make R jobs run on large-scale streaming datasets, we design and implement a distributed stream processing system based on the Spark Streaming framework. The system provides an R/Java cross-language interaction interface, which makes the R code and the Java code able to invoke each other and eventually makes the R codes run on the Java platform. The system also provides an R programming interface for Spark Streaming, which both improves the ability of R on the large-scale data processing and parallel computing and makes the Spark Streaming framework easier to use. Finally, a network work-count example is showed to demonstrate the feasibility and practicability of the system.