基于改进Single-Pass算法的网络新闻话题发现

通过改进的Single Pass增量文本聚类算法, 以话题为粒度对新闻信息进行组织, 实现网络新闻话题的发现. 该方法考虑了新闻的动态性和时间特性, 在特征词项权重计算中从词项在标题和正文中的位置信息及词项的增量文档频率两方面进行优化, 同时在相似度的计算中添加了时间因素及聚类中动态更新话题的质心向量. 应用 基于主题的网络爬虫构建的新闻等语料作为测试数据集, 实验结果表明, 改进算法较传统算法在耗费代价和错检率上分别降低0.34%和1.57%, 验证了改进算法的有效性和准确性.

Network News Topics Discovery Based on Improved Single-Pass Algorithm

By improved Single Pass incremental text clustering algorithm, we organized news information with granularity of topics, and achieved the discovery of network news topics. Considering the dynamic and time characteristics of news, the position information of terms in the headlines and texts and the frequency of incremental documents of terms in the feature terms weight calculation were optimized, meanwhile, time factor was added in similarity calculation and the topics centroid vectors were updated dynamically in clustering. Through the topic based Web crawler to construct news corpus as the test data set, the experimental results show that, compared with the traditional algorithm, the improved algorithm reduces the cost and fallout ratio by 0.34% and 1.57% respectively, which verify the validity and accuracy of the improved algorithm.