基于MapReduce的网络爬虫设计与实现

针对单机爬虫效率低、可扩展性差等问题，本文设计并实现了一种基于MapReduce的网络爬虫系统。该系统首先采用HDFS和HBase对网页信息进行存储管理，基于行块分布函数的方法进行网页信息抽取；然后通过URL和网页信息相似度分析相结合的去重策略，采用Simhash算法对抓取的网页信息进行相似度度量。实验结果表明,该系统具有良好的性能和可扩展性，较单机爬虫相比平均抓取速度提高了4.8倍。

MapReduce based web crawler design and implementation

We design and implement a MapReduce based web crawler system for such issues as low efficiency and bad scalability of a single crawler system. It employs HDFS and HBase to store web information and extracts web information through a row block distribution function. It then measures similarity for acquired web information by Simhash algorithm and deduplication strategy of similarity analysis of URL and web information. Experimental results show that it has better performance and scalability, and increases average crawling speed by 4.8 times, as compared with single crawling system.