基于MapReduce的网络爬虫设计与实现

针对单机爬虫效率低、可扩展性差等问题,本文设计并实现了一种基于MapReduce的网络爬虫系统。该系统首先采用HDFS和HBase对网页信息进行存储管理,基于行块分布函数的方法进行网页信息抽取;然后通过URL和网页信息相似度分析相结合的去重策略,采用Simhash算法对抓取的网页信息进行相似度度量。实验结果表明,该系统具有良好的性能和可扩展性,较单机爬虫相比平均抓取速度提高了4.8倍。     

基于coroutine模型的网络爬虫设计与实现

网络爬虫在中文信息处理中被大量使用,根据待处理的问题定向爬取相关领域的数据,为后续中文信息处理提供基础.传统多线程模型在处理高并发和大量I/O阻塞操作时,存在较为明显的限制和不足.针对以上问题,提出了一种基于coroutine模型的解决方案.从coroutine的基本原理和实现方法上作了较为详细的阐述,并给出基于coroutine网络爬虫的完整实现.实验表明,该方案能够有效地降低系统负荷,提高爬虫的爬取效率.     

基于语义相似聚合的主题爬虫算法研究

互联网的迅速发展,数据不断增加,使得个性化数据的获取难度越来越大.主题爬虫作为一种垂直检索方式,已经成为一个热门研究领域.传统的主题爬虫往往是通过网页链接之间的关系下载网页,然后再计算下载的网页与给定主题之间的相关关系.传统的主题爬虫一方面割裂了网页链接结构和网页内容主题之间的关系,使得两个部分分开计算; 另一方面下载过程的网页主题相关性不强,会下载大量的主题无关网页.本文提出一种新的基于PageRank 算法主题爬虫算法将网页主题相似度计算与传统的PageRank 算法相结合,将网页链接结构与网页主题相关性结合在一起.另外本文将语义相似性引入到主题爬虫里,实验结果表明本文提出的基于语义相似聚合的主题爬虫算法大大提高了主题爬虫的查全率.     

基于Python的网页信息爬取技术分析

基于大规模数据下载的需求,针对以往人工下载数据过程中存在的影响效率的问题,结合Python语言提出一种能够用于网页信息数据爬取的技术,用于减少数据下载前期的繁琐工作,提高网络信息系统和技术的开发效率。该文以网页信息爬取技术为主要研究对象,基于现代化信息数据的应用和发展需求,着重对融合了Python语言的网页信息爬取技术进行研究和分析。     

搜索服务中基于云计算的垃圾网页识别研究

为了快速、有效地对垃圾网页进行识别,提出了一种基于云计算平台的垃圾网页识别方案.在Hadoop分布式平台下构建一个基于页面内容和连接结构的支持向量机分类器,对垃圾网页进行识别,通过对数据进行分析.结果表明:垃圾网页的识别准确率较高,处理时间随着集群节点数的增加而明显减少.     

聚焦爬虫爬行性能研究

爬虫作为网页搜索下载程序,其网络爬行性能决定了搜索引擎的性能和数据质量。本文通过分析聚焦爬虫的特点和网络环境,总结出三类制约爬行性能的主要问题,分别是DNS查询及缓存设置、内外部分布式爬行特点和网页URL静态与动态分配策略。结论为聚焦爬虫使用URL静态分配策略结合异步DNS查询及缓存设置,在内部分布式爬行时可有效提高网络爬行性能。     

基于Hadoop集群的分布式日志分析系统研究

当数据存储和计算遇到瓶颈时,分布式技术相对于传统的向上扩展技术在伸缩性和成本上都具有巨大的优势。本文介绍了开源的分布式编程框架Hadoop,并通过具体的代码说明了基于Hadoop集群的分布式日志分析系统的工作方式。     

基于Hadoop的改进遗传算法

利用Hadoop的并行式处理技术改进了遗传算法.通过将种群划分为多个子种群,并将每个子种群由一个单一的MapReduce任务来处理,实现了遗传算法的并行化.通过解决OneMax问题的一系列实验,验证了基于Hadoop的遗传算法提高了运行效率与正确性.     

基于Scrapy的分布式爬虫系统的设计与实现

随着互联网的快速发展,其信息量和相关服务也随之快速增长.如何从海量的信息中快速、准确地抓取所需要的信息变得越来越重要,因此负责互联网信息收集工作的网络爬虫将面临着巨大的机遇和挑战.目前国内外一些大型搜索引擎只给用户提供不可制定的搜索服务,而单机的网络爬虫又难当重任,因此可定制性强、信息采集速度快和规模大的分布式网络爬虫便应运而生.通过对原有Scrapy框架的学习和研究,将Scrapy和Redis结合改进原有的爬虫框架,设计并实现了一个基于Scrapy框架下的分布式网络爬虫系统,然后将从安居客、58同城、搜房等网站抓取的二手房信息存入MongoDB中,便于对数据进行进一步的处理和分析.结果表明基于Scrapy框架下的分布式网络爬虫系统同单机网络爬虫系统相比效率更高且更稳定.     

基于Map/Reduce的网页消重并行算法

网页消重模块是搜索引擎系统的重要组成部分,其作用是对搜索引擎的爬虫系统下载的网页进行过滤,去除重复内容的网页,从而提高搜索引擎爬虫系统的性能和检索的质量。提出了一种网页消重的并行算法以及基于Map/Reduce的实现机制,并通过实际网站的实验验证了该消重算法的稳定性和处理大量网页时的并行性能。     

Markedness and its Interpretation in Language Use

Language markedness is a common phenomenon in languages, and is reflected from hearing, vision and sense, i.e. the variation in the three aspects such as phonology, morphology and semantics. This paper focuses on the interpretation of markedness in language use following the three perspectives, i.e. pragmatic interpretation, psychological interpretation and cognitive interpretation, with an aim to define the function of markedness.     

Characterization of Ordovician carbonate reservoirs, southeastern Saskatchewan, Canada

The discovery of the prolific Ordovician Red River reservoirs in 1995 in southeastern Saskatchewan was the catalyst for extensive exploration activity which resulted in the discovery of more than 15 new Red River pools. The best yields of Red River production to date have been from dolomite reservoirs. Understanding the processes of dolomitization is, therefore, crucial for the prediction of the connectivity, spatial distribution and heterogeneity of dolomite reservoirs.The Red River reservoirs in the Midale area consist of 3～4 thin dolomitized zones, with a total thickness of about 20 m, which occur at the top of the Yeoman Formation. Two types of replacement dolomite were recognized in the Red River reservoir: dolomitized burrow infills and dolomitized host matrix. The spatial distribution of dolomite suggests that burrowing organisms played an important role in facilitating the fluid flow in the backfilled sediments. This resulted in penecontemporaneous dolomitization of burrow infills by normal seawater. The dolomite in the host matrix is interpreted as having occurred at shallow burial by evaporitic seawater during precipitation of Lake Almar anhydrite that immediately overlies the Yeoman Formation. However, the low δ18O values of dolomited burrow infills (-5.9‰～ -7.8‰, PDB) and matrix dolomites (-6.6‰～ -8.1‰, avg. -7.4‰ PDB) compared to the estimated values for the late Ordovician marine dolomite could be attributed to modification and alteration of dolomite at higher temperatures during deeper burial, which could also be responsible for its 87Sr/86Sr ratios (0.7084～0.7088) that are higher than suggested for the late Ordovician seawaters (0.7078～0.7080). The trace amounts of saddle dolomite cement in the Red River carbonates are probably related to "cannibalization" of earlier replacement dolomite during the chemical compaction.     

低渗透非均质砂岩油藏启动压力梯度研究

理论推导与室内实验相结合,建立了低渗透非均质砂岩油藏启动压力梯度确定方法。首先借助油藏流场与电场相似的原理,推导了非均质砂岩油藏启动压力梯度计算公式。其次基于稳定流实验方法,建立了非均质砂岩油藏启动压力梯度测试方法。结果表明:低渗透非均质砂岩油藏的启动压力梯度确定遵循两个等效原则。平面非均质油藏的启动压力梯度等于各级渗透率段的启动压力梯度关于长度的加权平均;纵向非均质油藏的启动压力梯度等于各渗透率层的启动压力梯度关于渗透率与渗流面积乘积的加权平均。研究成果可用于有效指导低渗透非均质砂岩油藏的合理井距确定,促进该类油藏的高效开发。     

On Tragic Consciousness In The Works By Ernest Hemingway

As an American modern novelist who were famous in the literary world, Hemingway was not a person who always followed the trend but a sharp observer. At the same time, he was a tragedy maestro, he paid great attention on existence, fate and end-result. The dramatis personae's tragedy of his works was an extreme limit by all means tragedy on the meaning of fearless challenge that failed. The beauty of tragedy was not produced on the destruction of life, but now this kind of value was in the impact activity. They performed for the reader about the tragedy on challenging for the limit and the death.     

A computer generator for randomly layered structures

AcomputergeneratorforrandomlylayeredstructuresYUJia shun1,2,HEZhen hua2(1.TheInstituteofGeologicalandNuclearSciences,NewZealand;2.StateKeyLaboratoryofOilandGasReservoirGeologyandExploitation,ChengduUniversityofTechnology,China)Abstract:Analgorithmisintrod…     

An Index mechanism for multi-scale view in spatial databases

There are numerous geometric objects stored in the spatial databases. An importance function in a spatial database is that users can browse the geometric objects as a map efficiently. Thus the spatial database should display the geometric objects users concern about swiftly onto the display window. This process includes two operations:retrieve data from database and then draw them onto screen. Accordingly, to improve the efficiency, we should try to reduce time of both retrieving object and displaying them. The former can be achieved with the aid of spatial index such as R-tree, the latter require to simplify the objects. Simplification means that objects are shown with sufficient but not with unnecessary detail which depend on the scale of browse. So the major problem is how to retrieve data at different detail level efficiently. This paper introduces the implementation of a multi-scale index in the spatial database SISP (Spatial Information Shared Platform) which is generalized from R-tree. The difference between the generalization and the R-tree lies on two facets: One is that every node and geometric object in the generalization is assigned with a importance value which denote the importance of them, and every vertex in the objects are assigned with a importance value,too. The importance value can be use to decide which data should be retrieve from disk in a query. The other difference is that geometric objects in the generalization are divided into one or more sub-blocks, and vertexes are total ordered by their importance value. With the help of the generalized R-tree, one can easily retrieve data at different detail levels.Some experiments are performed on real-life data to evaluate the performance of solutions that separately use normal spatial index and multi-scale spatial index. The results show that the solution using multi-scale index in SISP is satisfying.     

Achieving Unusual States of Matter Under High Pressure

正The periodicity of the elements and the non-reactivity of the inner-shell electrons are two related principles of chemistry,rooted in the atomic shell structure.Within compounds,Group I elements,for example,invariably assume the+1 oxidation state,and their chemical properties differ completely from those of the p-block elements.These general rules govern our understanding of chemical structures and reactions.Using first principles calcula-     

Exchange-Correlation and Electronic Excitation Energies from Pairing Matrix Fluctuations

We have developed an adiabatic connection to formulate the ground-state exchange-correlation energy in terms of pairing matrix linear fluctuations.This formulation of the exchange-correlation energy opens a new channel for density functional approximations based on the many-body perturbation theory.We illustrate the potential of such approaches with an approximation based on the particle-particle Random Phase Approximation(pp-RPA).This re-     

Density Functional Theory for the Response of Periodic Systems to Electric Fields Based on the Vector Potential Approach

正The electronic and nuclear(structural/vibrational)response of 1D-3D nanoscale systems to electric fields gives rise to a host of optical,mechanical,spectral,etc.properties that are of high theoretical and applied interest.Due to the computational difficulty of treating such large systems it is convenient to model them as infinite and periodic(at least,in first approximation).The fundamental theoretical/computational problem in doing so is that     

On the Thermodynamic Limit for Responses to Static Electromagnetic Fields

For molecular systems,the quantum-mechanical treatment of their responses to static electromagnetic fields usually employs a scalar-potential treatment of the electric field and a vector-potential treatment of the magnetic field.Although the potential for each field separately is associated with the choice of an(unphysical)origin,the precise choice of the origin for the electrostatic field has little consequences for the results.This is different for the