支持云计算环境的MapReduce模拟器设计

针对云计算环境下很难确定出有助于实现良好性能的Hadoop参数集的问题,设计了一种MapReduce模拟器.首先,对Hadoop的各种参数进行建模;然后,集群读取元件从集群中读取参数,从而创建模拟的Hadoop集群环境;最后,利用作业跟踪器跟踪模拟的作业,并利用任务跟踪器运行单个任务.所设计的模拟器从多角度研究Hadoop应用的性能,集中模拟Hadoop的映射和化简行为,弥补了MRPerf设计的不足.在基准测试结果和用户自定义MapReduce应用中验证了模拟器的有效性.

Design of a MapReduce Simulator Supporting Cloud Computing Environment

For the issue that Hadoop parameter sets helpful for achieving good performance are difficult to be confirmed in cloud environment,a MapReduce simulator was designed in this paper. Firstly,various parameters of Hadoop were modeled. Then,the parameters were readed from cluster by cluster read element so that simulative Hadoop cluster environment was established. Finally,the simulative job is tracked by using job tracker,and task tracker was used to run a single task. Performance of Hadoop application with multi-angles was researched by the designed simulator,and it concentrated at simulating the Hadoop map and reduction behavior so that it made up the deficiency of the MRPerf designing. The effectiveness of designed simulator was verified by the baseline testing results and MapReduce application of customized definition.