一种基于混合索引的HDFS小文件存储策略

Hadoop分布式文件系统(hadoop distributed file system,HDFS)因其稳定高效、低成本等优势,已被很多大型企业广泛使用.针对HDFS海量小文件存储时元数据服务器节点内存开销过大,合并文件中小文件访问效率不高的问题,提出一种改进的基于混合索引的小文件存储策略,应用分类器分类标记小文件,并在元数据服务器建立H-B+树索引,在存储节点根据小文件大小建立不同的块内索引,达到提高小文件访问效率的目标.实现中采用缓存结构,以提高客户端访问的响应速度,同时也有利于元数据服务器节点的内存负载.实验结果表明,基于混合索引的小文件存储策略能有效提高小文件访问效率,并显著降低元数据节点内存开销.     

Construction of Smart City Spatio-Temporal Information Cloud Platform in Weifang,China

On the basis of the digital Weifang geospatial framework,Smart Weifang spatio-temporal information cloud platform(WFCP) integrated legal person information,population,place name and address data,macroeconomic data and so on.And it also expanded the data contents,such as the indoor and outdoor data,the overground and underground data,panoramic data and real data.It also introduced the contents of historical geographical information in different periods and real-time location information,address information of sensing equipment,real-time perception and interpreting information.It has overcome the difficulties of real-time access of Internet of Things(IoT) perception,multi-node collaboration,64-bit support,cluster deployment and has the characteristics of spatio-temporal management,ondemand service,large data analysis and micro-service architecture.It built spatio-temporal information big data center and spatio-temporal information cloud platform,realized the convergence and management of the distributed big data,deeply applied for land,transportation,environmental protection,police and subdistrict five areas,by supporting the integrated application of multi-source information and supporting intelligent deep application.In the aspect of hardware environment construction,according to the top-level design and unified arrangement of Smart Weifang,the WFCP was migrated to Weifang cloud computing center,to achieve the on-demand computing resources and dynamic scheduling load-based computing resources,to support the generalizing load map application.     

A hybrid cache placement scheme for multi-hop wireless service network

In this paper,a hybrid cache placement scheme for multihop wireless service networks is proposed. In this scheme,hot nodes in data transferring path are mined up by means of rout-ing navigation graph,and whole network is covered with network clustering scheme. A hot node has been chosen for cache place-ment in each cluster,and the nodes within a cluster access cache data with no more than two hops. The cache placement scheme reduces data access latency and workload of the server node. It also reduces the average length of data transferring,which means that fewer nodes are involved. The network system energy con-sumption decreased as involved relay nodes reduced. The per-formance analysis shows that the scheme achieves significant system performance improvement in network environment,with a large number of nodes.     

File assignment policy in network storage system

0　IntroductionStorageareanetworks(SAN)andNetworkattachedstorage(NAS)aretwoprovenapproachestonetworkingstorage.Ingeneral,bothofSANandNASemploytheswitchtechnologyforinterconnectthatcanefficientlyreducedelayandcompetitionofnetwork[1 3] .Inthissystem ,anaccesstoonedeviceisinde pendentofotherdevices.Fromviewofabstract,eachnetworkstoragedeviceisviewedasnetwork attacheddisk .SAN (FibreChannel attacheddisks)andNAS (Ethernet \|attacheddisks)areexamplesofsuchanapproach .Sointhispaper,thenetw…     

共享存储器型超并行多处理机系统的模拟性能评价

在共享存储器超并行多处理机系统中，访问共享存储器的吞吐量相当大，而存取延迟是与多级互连网络的级数成正比的．为防止这种延迟而产生的处理效率下降，把整个系统视为一个流水线处理体系，同时配合使用高速缓冲存储器．本文给出超并行多处理机系统的模拟性能评价结果．     

基于HBase的小文件高效存储方法

基于 Hadoop 平台的相关系统得到了广泛应用。Hadoop 分布式文件系统（Hadoop distributed file system, HDFS）通过分布式的工作方式,负责处理海量文件数据。对 HDFS 而言,海量数据中的小文件存储问题制约着系统高效工作的能力。针对海量数据中小文件读写效率低的情况,提出一种基于 HBase（Hadoop database）的海量小文件高效存储方法,利用 HBase 的存储优势,将小文件直接存储于 HBase,从而有效减少元数据节点服务器（Name-Node）的负载,并对上层应用系统提供透明的访问接口。实验结果表明,该方法可以实现海量小文件的高效存储,提高 HDFS 环境下小文件的读写效率。     

GDSS系统中文件缓存副本策略及其性能研究

首先针对GDSS系统中引入文件缓存副本模型的必要性进行了相关探讨,接着针对域内情况提出了一种基于访问综合评价值的文件缓存副本创建管理策略,并结合Java编写的模拟器测试工具,得到了缓存命中率和数据访问集中度,缓存时间临界值等之间的关系,然后针对该模型,详细说明了其在GDSS系统中的实现方法.     

基于云计算的一种全新元数据管理方案

为有效减少云计算过程中不同元数据的工作量,本研究提出了一种全新的管理方案.通过改进的Chord算法和协同双层缓存机制使得元数据服务性能随着聚类上元数据服务器数量增加而增强;采用亲密算法控制客户端与服务器端的协作以便对元数据集进行缓存.基于以上方案能表现出较高性能和良好的可扩展元数据服务,实验表明本文提高的方案性能优于其它方案,更能体现出元数据访问性能与亲密值之间变化的关系.     

A Cache Considering Role-Based Access Control and Trust in Privilege Management Infrastructure

PMI （privilege management infrastructure） is used to perform access control to resource in an E-commerce or E-government system. With the ever-increasing need for secure transaction, the need for systems that offer a wide variety of QoS （quality-of-service） features is also growing. In order to improve the QoS of PMI system, a cache based on RBAC （Role-based Access control） and trust is proposed. Our system is realized based on Web service. How to design the cache based on RBAC and trust in the access control model is deseribed in detail. The algorithm to query role permission in cache and to add records in cache is dealt with. The policy to update cache is introduced also.     

A Novel Architecture of Metadata Management System Based on Intelligent Cache

This paper introduces a novel architecture of metadata management system based on intelligent cache called Metadata Intelligent Cache Controller （MICC）. By using an intelligent cache to control the metadata system, MICC can deal with different scenarios such as splitting and merging of queries into sub-queries for available metadata sets in local, in order to reduce access time of remote queries. Application can find results patially from local cache and the remaining portion of the metadata that can be fetched from remote locations. Using the existing metadata, it can not only enhance the fault tolerance and load balancing of system effectively, but also improve the efficiency of access while ensuring the access quality.     

面向网络GIS的最小价值空间数据缓存替换算法研究

海量空间数据的网络传输是影响网络GIS服务质量的重要因素.数据缓存机制能有效提高数据响应速度,提升网络GIS服务质量.FIFO、LRU、LFU等常用的缓存替换算法侧重于数据的访问时间和频率,而未能兼顾空间数据特有的空间位置特性,在空间数据缓存的应用上存在着局限性.该文面向网络GIS提出了一种最小空间数据价值缓存替换算法GDLVF(Lowest-Value FirstCache Replacement for Geospatial Data),除考虑数据的访问时间和频率外,还兼顾数据空间位置与可视区域间的面积和位置关系,并利用加权数据大小来降低数据大小差异对价值计算的影响.实验表明,GDLVF较传统的缓存替换算法能够同时兼顾请求命中率和字节命中率,减少数据的网络传输,显著改善网络GIS的服务质量.     

Web智能代理的预取技术和缓存技术

针对同一个工人组中成员可能对Web有相似的兴趣点和访问习惯问题,研究了主要包括缓存和预取两个部分的智能代理技术,研究了代理缓存三种替换算法：LRU和LRU的两种变种,仿真试验表明,LRU算法的命中率极差,将LRU的两种变种相结合则是一种较好的方案,通过引入预测算法和门限算法,代理服务器可以预测最近将要访问到的页面,并存在客户际请求提出之前有选择地下载,因为单个客户访问某个页面的历史次数往往不够多,将预测算算放在代理服务器上比放在客户端的预测概率更准确,通过在代理服务器上实现缓存和预取技术,可减少用户访问Web的响应时间,还可减少实际访问Web服务器的总次数,并提出了一个为发展中国家中国家用户缩减Web访问代价的有效方法。     

层次化网络策略管理关键技术的研究与实现

提出了层次化的网络策略管理模型，其特点是引入了多级策略决策点(PDP)，在策略执行点(PEP)中引入本地PDP和缓存，PDP中也引入缓存，扩展了IETF策略管理框架．实现了将高层策略规范转换为各种运行期描述的策略编译器及其后端．将策略作为运行期对象实现，可以在域服务中方便地访问策略对象并将其自动分发到执行组件，减轻了PDP的负担，提高了被管设备的智能和网络管理的自动化程度，提高了系统的可扩展性和健壮性。     

Multiple Routes Recommendation System on Massive Taxi Trajectories

This paper presents a cloud-based multiple-route recommendation system, xGo, that enables smartphone users to choose suitable routes based on knowledge discovered in real taxi trajectories. In modern cities, GPS-equipped taxicabs report their locations regularly, which generates a huge volume of trajectory data every day. The optimized routes can be learned by mining these massive repositories of spatio-temporal information. We propose a system that can store and manage GPS log files in a cloud-based platform, probe traffic conditions, take advantage of taxi driver route-selection intelligence, and recommend an optimal path or multiple candidates to meet customized requirements. Specifically, we leverage a Hadoop-based distributed route clustering algorithm to distinguish different routes and predict traffic conditions through the latent traffic rhythm. We evaluate our system using a real-world dataset(100 GB) generated by about 20 000 taxis over a 2-month period in Shenzhen, China. Our experiments reveal that our service can provide appropriate routes in real time and estimate traffic conditions accurately.     

Design and Implementation of Hierarchy Cache Using Pagefile

This paper presents a novel hierarchy cache architecture for the purpose of optimizing IO performance. The main idea of the hierarchy cache is to use a few megabytes of RAM and a pagefile to form a two-level cache architecture. The pagefile is equivalent to the cache disk in DCD(Disk Caching Disk). The pagefile outperforms data disks, because data are accessed in different units and different ways. Small writes are collected in the RAM cache first, and data will be transferred to the pagefile in large writes later. When the system is idle, it will destage data from the pagefile to data disks. The performance test results show that the hierarchy cache can improve IO performance dramatically for small writes, and the mail server using the hierarchy cache driver can handle transactions about 2.2 times faster than the normal mail server. The hierarchy cache is implemented as a filter driver, so it‘s transparent to the current Windows 2000/Windows XP operating system.     

基于.NET Cache Memcached Web缓存技术的研究与应用

缓存技术在Web开发中得到了广泛的应用,合理的缓存方案可以减轻服务器负担,提升系统性能。采用Asp.net Cache Memcached缓存技术相结合的方式能够有效地增强系统的缓存性能。本文首先介绍了Asp.net 自身Cache类与Memcached缓存各自的工作原理,结合Asp.net自身缓存的高速访问性和Memcached的数据共享等各自优势,提出了.Net内部缓存与Memcached外部缓存相结合的缓存机制,最后通过测试数据得出结论,这种内外相结合的缓存方式有效地解决了Asp.net自身缓存在分布式应用下不能提供数据共享的不足,提高了Web应用的整体性能。     

Architecture Design of a Variable Length Instruction Set VLIW DSP

The cost of the central register file and the size of the program code limit the scalability of very long instruction word (VLIW) processors with increasing numbers of functional units. This paper presents the architectural design of a six-way VLIW digital signal processor (DSP) with clustered register files. The architecture uses a variable length instruction set and supports dynamic instruction dispatching. The one-level memory system architecture of the processor includes 16-KB instruction and data caches and 16-KB instruction and data on-chip RAM. A compiler based on the Open64 was developed for the system. Evaluations show that the processor is suitable for high performance applications with a high code density and small program code size.     

A prediction mechanism of file access popularity and optimal strategy of replicas deployment in cloud computing

In cloud computing,the number of replicas and deployment strategy have extensive impacts on user's requirement and storage efficiency.Therefore,in this paper,a new definition of file access popularity according to users' preferences,and its prediction algorithm are provided to predict file access trend with historical data.Files are sorted by priority depending on their popularity.A mathematical model between file access popularity and the number of replicas is built so that the reliability is increased efficiently.Most importantly,we present an optimal strategy of dynamic replicas deployment based on the file access popularity strategy with the overall concern of nodes' performance and load condition.By this strategy,files with high priority will be deployed on nodes with better performance therefore higher quality of service is guaranteed.The strategy is realized in the Hadoop platform.Performance is compared with that of default strategy in Hadoop and CDRM strategy.The result shows that the proposed strategy can not only maintain the system load balance,but also supply better service performance,which is consistent with the theoretical analysis.     

预约统计无线分组多址接入控制方法研究

本文提出了一种基于分组传输的无线接入系统,它具有一定的综合业务接入功能,并提出了相应的无线分组预约统计轮询（ＰＲＳＰ） 多址接入控制（ＭＡＣ） 方法,经计算机模拟研究结果显示,ＰＲＳＰ在各接入点突发数据业务不平衡的一般情况和相对小业务时,系统的延时性能较通常轮询多址接入控制方式优越．     

对象存储系统的合作缓存方案

结合对象存储系统的数据访问模式,综合设计客户端和元数据服务的缓存,构造存储系统的合作缓存方案.该方案将客户端和元数据服务器的缓存作为整体进行设计,以达到提高缓存利用率的目的;通过缓存准入策略合理选择数据传送模式,减少数据传送的通信量;同时,合作缓存方案根据数据对象的大小、访问成本和网络负载动态地调整缓存策略,提高存储系统的服务质量.实验显示,合作缓存方案能较好地适应不同的工作负载,有效提高了系统的输入输出性能.