网页分类中的数据预处理方法研究

讨论了网页分类过程中数据预处理的相关技术,提出一种网页预处理方法,将网页解析为DOM树结构,通过分析、整理,得到噪音信息的特征,并依据判定规则,找出噪音信息并删除处理。网页去噪后,提取页面中的文本信息和每个相关链接目标URL、源URL及锚文本并存入数据库。实验结果证明,该去噪方法可以有效去除网页噪音,利用所提取的超链接结构信息分类网页,可以有效提高分类精度。

Research on the Data Preprocessing of Webpage Categorization

This paper discussed the technology of data preprocessing in webpage categorization process and put forward a method of webpage pretreatment which parse the page into a DOM tree structure.The characteristics of the webpage noise was extracted through analysising and sorting operation.We finally found out the noise in accordance with the decision rules and removed them.Then,we extracted the text information and links for each target URL,source URL and anchor text and stored them in the database.This method has been proved to be effective on improving the accuracy of classification by our experiment.