基于DOM树及行文本统计去噪的网页文本抽取技术

首先对网页源码文本统一编码转为UTF格式,然后把HTML网页文档转换为XML文档并解析为一棵DOM树。依据XML语言特点及噪声特征规则先对DOM树的噪声节点进行过滤删除,然后依据中文标点符号统计方法提取网页正文内容,并在此基础上利用行文本统计方法去除提取出的正文中存在的噪声信息,最后得到网页正文文本。对来自结构完全不同的主流与非主流的中英文新闻网站上的2 000篇网页进行实验,结果表明本文提出的方法具有较高的抽取准确率,并具有很好的通用性和实现简单的特点,适用于针对互联网中不同网站新闻文本信息的自动采集。

Content extraction from web page based on the DOM tree and line-text statistical noise-elimination

As different web pages have different codes,the HTML web page first need to be encoded with the uniform code UTF8,and then translated into an XML document which is parsed into the DOM tree.After removing some noise nodes from the DOM tree according to the features of XML language and the rules of the noise characteristics,text contents are extracted from the DOM tree by the method of statistics of punctuation and noise information is continued to be eliminated from contents extracted above by the method of statistics of line-text.The result of experiments on 2000 web pages obtained from different web sites shows that our method has high accuracy,great generality,and simplicity,and can be automatically used to extract the right contents from different web sites.