Data Location Mechanism in Agent-Based Autonomic Storage Systems

This paper introduces agent-based methodology to build a distributed autonomic storage system infrastructure, and an effectively negotiation mechanism based on agent is applied for data location. We present Availability-based Data Allocation （ADA） algorithm as a data placement strategy to achieve high efficient utilization of storage resources by employing multiple distributed storage resources. We use Bloom filter in each storage device to track the location of data. We present the data lookup strategy that small size of read request is handled directly, and large size of read request is handled by cooperation with storage devices.The performance evaluation shows that the data location mechanism is high available and can work well for heterogeneous autonomic storage systems.     

一种分布式文件系统的推送式数据预取机制

为了克服传统的数据预取机制的不足,提出了一种在分布式文件系统中存储服务器端的推送式数据预取机制.在达到减少网络通信的数据量和通信时延目的的同时,可以让运行客户端文件系统的计算节点从跟踪I/O操作和预测I/O操作的工作中脱离出来,减轻了客户端的工作负担,提高了存储系统的性能.     

纠删码存储系统中数据修复性能优化研究进展与展望

纠删码被广泛应用于分布式存储系统以保存在线应用的用户数据。当部分存储节点发生故障时,纠删码存储系统需使用新的存储节点替换原有失效节点,并恢复失效的用户数据。由于需要执行数据编码、传输和读写等操作,纠删码存储系统通常需要消耗较长的时间执行数据修复操作,存储的用户数据将长期处于不可靠状态。为了保障存储数据的可靠性,研究学者提出了多种数据修复性能优化方案以减少数据修复时间。本文介绍了数据修复性能优化问题,分析了各个应用场景下主要的性能瓶颈和性能优化难点,总结了提升数据修复性能的主要技术方案和研究工作,并对数据修复性能优化研究领域的未来发展方向进行展望,为纠删码存储系统设计人员准确选择适合特定应用场景的数据修复性能优化方案提供思路。     

云计算下异构数据集成模型的构建

为使用户能够将现有的应用系统向云计算环境迁移,将原有的异构数据处理无缝的映射为云计算环境下通用的数据服务和行为,从云计算和云存储的理论和模型出发,面向大规模的数据密集应用,针对各种异构数据库存储系统和云存储系统,构建云计算环境下异构数据模型,为用户提供透明、统一的异构数据集成和访问接口服务。实践证明:云计算下基于本体的异构数据集成模型的构建,可以为云计算环境下各业务应用提供统一的数据管理和处理方法,方便、快捷地实现异构数据统一的检索与查询以及业务应用所处理的各种异构数据之间实质性关联与映射。     

Load Feedback-Based Resource Scheduling and Dynamic Migration-Based Data Locality for Virtual Hadoop Clusters in OpenStack-Based Clouds

With cloud computing technology becoming more mature, it is essential to combine the big data processing tool Hadoop with the Infrastructure as a Service(Iaa S) cloud platform. In this study, we first propose a new Dynamic Hadoop Cluster on Iaa S(DHCI) architecture, which includes four key modules: monitoring,scheduling, Virtual Machine(VM) management, and VM migration modules. The load of both physical hosts and VMs is collected by the monitoring module and can be used to design resource scheduling and data locality solutions. Second, we present a simple load feedback-based resource scheduling scheme. The resource allocation can be avoided on overburdened physical hosts or the strong scalability of virtual cluster can be achieved by fluctuating the number of VMs. To improve the flexibility, we adopt the separated deployment of the computation and storage VMs in the DHCI architecture, which negatively impacts the data locality. Third, we reuse the method of VM migration and propose a dynamic migration-based data locality scheme using parallel computing entropy. We migrate the computation nodes to different host(s) or rack(s) where the corresponding storage nodes are deployed to satisfy the requirement of data locality. We evaluate our solutions in a realistic scenario based on Open Stack.Substantial experimental results demonstrate the effectiveness of our solutions that contribute to balance the workload and performance improvement, even under heavy-loaded cloud system conditions.     

An Energy-Saving Task Scheduling Strategy Based on Vacation Queuing Theory in Cloud Computing

High energy consumption is one of the key issues of cloud computing systems. Incoming jobs in cloud computing environments have the nature of randomness, and compute nodes have to be powered on all the time to await incoming tasks. This results in a great waste of energy. An energy-saving task scheduling algorithm based on the vacation queuing model for cloud computing systems is proposed in this paper. First, we use the vacation queuing model with exhaustive service to model the task schedule of a heterogeneous cloud computing system.Next, based on the busy period and busy cycle under steady state, we analyze the expectations of task sojourn time and energy consumption of compute nodes in the heterogeneous cloud computing system. Subsequently, we propose a task scheduling algorithm based on similar tasks to reduce the energy consumption. Simulation results show that the proposed algorithm can reduce the energy consumption of the cloud computing system effectively while meeting the task performance.     

A hybrid cache placement scheme for multi-hop wireless service network

In this paper,a hybrid cache placement scheme for multihop wireless service networks is proposed. In this scheme,hot nodes in data transferring path are mined up by means of rout-ing navigation graph,and whole network is covered with network clustering scheme. A hot node has been chosen for cache place-ment in each cluster,and the nodes within a cluster access cache data with no more than two hops. The cache placement scheme reduces data access latency and workload of the server node. It also reduces the average length of data transferring,which means that fewer nodes are involved. The network system energy con-sumption decreased as involved relay nodes reduced. The per-formance analysis shows that the scheme achieves significant system performance improvement in network environment,with a large number of nodes.     

A Fault Detection Mechanism in Erasure-Code Byzantine Fault-Tolerance Quorum

Fault-tolerance is increasingly significant for large-scale storage systems in which Byzantine failure of storage nodes may happen. Traditional Byzantine Quorum systems that tolerate Byzantine failures by using replication have two main limitations： low space-efficiency and static quorum variables. We propose an Erasure-code Byzantine Fault-tolerance Quorum that can provide high reliability with far lower storage overhead than replication by adopting erasure code as redundancy scheme. Through read/write operations of clients and diagnose operation of supervisor, our Quorum system can detect Byzantine nodes, and dynamically adjust system size and fault threshold. Simulation results show that our method improves performance for the Quorum with relatively small quorums.     

云存储及其分布式文件系统研究

云存储专注于解决云计算系统中海量数据文件的存储问题;文件系统用来管理大量的文件信息,并负责对文件的存储空间进行组织分配;分布式文件系统不仅要管理本地计算机的数据文件,同时还要管理整个存储系统中的数据文件。文章首先对云计算技术进行介绍,然后引出云存储的概念,并通过对云存储的技术原理、性能优势等进行分析,指出分布式文件系统对于云存储技术的重要性;最后对现有的云存储分布式文件系统进行详细论述。全文工作将指导我们进一步深入研究云存储技术。     

面向大数据应用挑战的超级计算机设计

在大数据时代, 大数据处理给面向高性能计算设计的超级计算机带来计算、存储、通信、编程等软硬件体系结构方面的挑战. 对此在分析目前主流学术界和工业界相关研究成果的基础上, 介绍了天河二号超级计算机系统针对大数据处理应用所采用的新型异构多态体系结构、自主定制高速互连网络、混合层次式存储系统、混合计算框架等多种优化措施, 为大数据时代的超级计算机设计提供有意义的参考.     

一种不依赖访问热度信息的分布式文件放置算法

文件放置问题一直是分布式存储领域的研究热点。分布式文件存储系统HDFS随机选择节点完成文件放置,存在访问负载分布不均衡的缺点。研究人员提出大量基于文件访问热度信息的放置算法;但是,文件的访问热度信息是动态变化的,难以准确预测。提出一种不依赖访问热度信息的分布式文件放置算法;该算法仅使用文件的创建时间信息,利用文件已创建时间与访问热度之间的相关性,首先将时间进行区间划分,然后统计出各节点在不同时间区间内所创建文件的数据量,放置过程中保持同一时间区间的数据量在不同节点间大致相同。实验结果表明,该算法不仅可以使各节点的存储负载达到均衡,还能够提升访问负载的均衡,消除因文件访问热度不均而导致的性能瓶颈。     

云存储结构模型及云存储架构的比较研究

云存储的概念是一个新的概念,它所利用的技术包括集群计算、分布式文件系统以及网格计算技术等。这些异构的存储系统通过彼此的协调、配合,构成一个虚拟的、巨大的存储系统,作为一个整体共同对外提供数据存储和业务访问功能。文章阐述了云存储结构模型,并对几种典型云存储的架构进行了比较研究。     

异构任务图的实现模型

任意异构程序映射到任意异松系统上执行,是并行程序计算最复杂的情况,为此提出了异构任务图的定义,它是异构程度直观和有效描述方法,还研究了异构任务图的实现模型,它是异构计算环境中软件和硬件的桥梁,分别给出了任务图的分簇算法、系统图的分簇算法和映射算法及应用实例,利用该模型可设计出可移植的并行程序,然后调度到可动态配置的异构计算机组织上运行。     

Fault-Tolerant Mechanism of the Distributed Cluster Computers

The distributed system with high performance and stability is commonly adopted in large scale scientific and engineering computing. In this paper, we discuss a fault-tolerant mechanism under Linux circumstance to improve the fault-tolerant ability of the system, namely a scheme and frame to form the stable computing platform. In terms of the structure and function of the distributed system, active list and file invocation strategies are employed in the task management. System multilevel fault-tolerance can be achieved by repeated processes in a single node and task migration on multi-nodes. Manager node agent introduced in this paper administrates the nodes using the list, disposes of the tasks according to the nodes' performance, and hence, to be able to make full use of the cluster resources. An evaluation method is proposed to appraise the performance. The analyzed results show the usefulness of the scheme proposed except for some additional overhead of memory consumption.     

GPU加速高性能计算平台上容器性能评估

容器是近年来新兴的虚拟化工具,可以实现资源和系统环境的隔离.容器能够帮助高性能计算应用程序将依赖打包进轻量级可移植的环境中,解决因软件配置无法在高性能计算资源上运行的问题.容器在虚拟化宿主机过程中具有性能开销,为了解GPU加速高性能计算平台上容器虚拟化技术的性能特征,使用标准基准测试工具对Docker容器进行了全面的性...     

数据网格中存储I/O的SPN建模与性能评价

针对数据网格中存储资源的结构、协议和I/O模式的复杂性问题,分析了存储网络的并行存储I/O请求模型,采用随机Petri网(SPN)建立了SPN并行存储I/O传输的数学模型,描述了系统请求响应时间与系统负载、网络带宽和节点处理能力之间的关系,然后通过排队论进行了并行I/O的性能分析,最后归纳了影响存储网络系统I/O性能的因素,在并行策略和数据组织与分配等方面提出了提高整体存储性能的改进方法.为改进提高数据网格的整体存储I/O性能提供了有效的建模方法和具体的理论依据.     

无线传感网中基于压缩感知的高效数据传输方案

传统的无线传感网技术中,数据源节点将采集到的原始数据序列直接传送给中间转发节点进行处理,因此当需采集的数据序列很长或者数据源节点的数目很多时中间转发节点需处理的数据量非常大。为此,基于近年来发展迅速的压缩感知理论,本文将压缩感知技术应用于无线传感网,提出一种新型高效的数据采集传输方案。采用系统所需存储转发寄存器的容量、节点耗能、时延来衡量无线传感网的网络性能。仿真结果表明,压缩感知理论可在近乎完美地重建信号的前提下有效地减少处理的数据量进而降低对硬件的要求并改善无线传感网网络性能。     

SCStore: Managing Scientific Computing Packages for Hybrid System with Containers

Managing software packages in a scientific computing environment is a challenging task, especially in the case of heterogeneous systems. It is error prone when installing and updating software packages in a sophisticated computing environment. Testing and performance evaluation in an on-the-fly manner is also a troublesome task for a production system. In this paper, we discuss a package management scheme based on containers. The newly developed method can ease the maintenance complexity and reduce human mistakes. We can benefit from the self-containing and isolation features of container technologies for maintaining the software packages among intricately connected clusters. By deploying the Super Computing application Strore(SCStore) over the WAN connected world-largest clusters, it proved that it can greatly reduce the effort for maintaining the consistency of software environment and bring benefit to achieve automation.     

面向云端融合计算的防御性数据销毁机制

云端融合计算将云计算和对等计算有机地融合,充分挖掘网络中心集群服务器端和网络边缘终端所蕴含的各种可利用的资源。引入了动态性比较强的网络边缘节点之后,云端融合计算环境下的数据安全保护和远程数据销毁更加困难,特别是恶意的异地主体可能对被托管的数据实施非法篡改等攻击行为。为此,本文提出了基于移动Agent的防御性数据销毁机制,利用移动Agent对云端数据进行检测,将待上传密文数据的HASH值拼接在密文后面,并定时对数据进行双重检测,有效区分云端数据的正常修改和非法篡改。通过预测相关数据受攻击的可能性,对已被非法篡改或有被非法篡改可能性的数据进行有效处理。设计并构建了基于移动Agent的云端数据防御性数据销毁原型系统。实验结果表明,基于移动Agent的防御性数据销毁机制在检测准确性、安全性、时间开销等方面具有良好的性能表现。      

无线Ad hoc网络的连通性与抗摧毁性

为了使Ad hoc网络能持续保持连通,覆盖性与抗摧毁性必须加以考虑.在任何两个一跳覆盖区必须通过公用信道传输信息的前提下,通过分析给定区域内信道节点的特征,提出了网络覆盖优化模型和规划方法,完成了区域的完全覆盖.针对模型的抗摧毁性能,提出了利用频率代替直接计算概率的算法,并在此基础上分析了随机抽取若干信道节点后整个网络的抗摧毁性.模拟实验结果证明,采用这种划分方法可以利用较少的信道完成对网络的覆盖,并能够得到较好的网络连通性和抗摧毁性能.