利用超链接信息改进网页爬行器的搜索策略

网页爬行器在Web空间中爬行时,要面对如下两个问题:1)由于Internet上的信息量十分巨大,网络搜索引擎不可能包含整个Web网页;2)受到硬件资源的限制,它所能存储的网页是有限的.爬行器如果按照传统的宽度优先搜索策略在Web空间中爬行,它对所有的网页都采取一视同仁的态度,这样爬行的结果就导致了它所爬行回来的网页质量不高.为此,给出了利用超链接信息改进网页爬行器搜索策略的算法.该算法充分考虑了网页之间的超链接信息,克服了传统的宽度优先搜索策略的盲目性爬行.实验表明,利用该算法爬行得到的网页与某一特定主题相关的网页超过50%.

Using Hyperlink Information to Improve Crawler's Searching Strategy

A crawler must face two problems when it searches pages in internet. One is that an internet search engine cannot contain entire Web pages due to huge volumes of data in internet. Because of the constraint of hardware resource, the other is that the Web pages stored in the internet search engine are limited. Crawling in Web space according to the strategy of the traditional breadth-first search, if a crawler respects the importance of every page equally, the quality of Web pages collected by the crawler is not high. The algorithm proposed in the present paper makes the best use of hyperlink information contained in the Web pages to the great extent, overcomes the limitations of the blind crawling strategy owned by the traditional breadth-first search. It is improved by using the hyperlink information contained in the Web pages on the base of the traditional breadth-first search. The experiment results show the Web pages crawled by using this algorithm those are relevant to a pre-defined set of topics are over 50%.