移动互联网用户行为分析系统中聚焦爬虫的设计与实现

在移动互联网用户行为分析系统中,为了使深度包检测(deep packet inspection,DPI)进行有效地数据匹配,对用户行为进行更深层次的分析,达到不仅能识别出业务网站类型而且还识别出业务网站访问具体内容的目标,设计出一种能进行具体内容级别上特征爬取和提炼的爬虫模块.针对特定业务网站,考虑广义爬取对技术和存储要求高的缺点以及针对某一行业的爬虫系统得到数据有限的不足,设计并实现了一种基于特定页面分析的聚焦爬虫模块.该爬虫模块采用模块化的思想,使用多线程多任务,精确高效地爬取特定业务网站信息,为DPI匹配提供数据支持.经过测试验证,该爬虫模块达到了预期的要求,可维护性、可扩展性和实时性强,满足移动互联网用户行为分析系统对特征数据提取的需求.

Design and implementation of focused crawler in mobile internet user behavior analysis system

In the mobile Internet user behavior analysis system, to make the DPI (deep packet inspection) data matching effective, to do deeper user behavior analysis, to not only identify the business site types but also identify the specific content of business site visiting, a crawler module that can extract and refine specific content was designed. For specific business website, considering the disadvantage that generalized crawler need high technology and storage requirements and the deficiency that crawler system for a particular industry only can get limited data, a focused crawler module based on the analysis of the specific page was designed and implemented. The crawler module adopts the idea of modular, uses multithreading and multi-tasking, accurately and efficiently gets data information from specific business website, and provides data support for the DPI matching. Trough testing, the crawler module has reached the expected requirement, has strong maintainability, expansibility and timeliness, and meets the demand of feature data extraction in the mobile Internet user behavior analysis system.