大型文本数据库中分布式数据去重备份方法研究

数据去重可删除备份过程中的冗余数据,节约存储资源与与网络带宽,是目前数据存储领域研究的重点问题。针对传统方法去重率和吞吐量低的问题,提出一种新的大型文本数据库中分布式数据去重备份方法。介绍了大型文本数据库中分布式数据去重方法基本思想,通过文件管理部分对数据片组中的扶贫数据进行查询和统计,给出数据片组路由策略,分析了数据预取过程。通过最终权值完成对数据块的排列,以恢复时间与恢复成本达到最小化为目的设计适应度函数。通过改进遗传方法对大型文本数据库中分布式扶贫数据进行备份。实验结果表明,所提方法有很高的去重率和吞吐量,且成本低,恢复速度快。

Research on distributed data deduplication method in large text database

Data deduplication can remove redundant data in the backup process and save storage resources and network bandwidth. It is the key point of data storage research. Aiming at the problem of low throughput and low throughput of traditional methods, a new method of distributed data deduplication in large text databases is proposed. This paper introduces distributed data of large text databases to the basic thought way through the file management part query and statistical data on poverty alleviation sheet group, and gives the data group routing strategy, analyzes the data prefetching process. Finally, the fitness function is designed by minimizing the recovery time and the recovery cost by arranging the data blocks with the final weights. Genetic algorithm is used to backup the distributed poverty reduction data in large text databases. Experimental results show that the proposed method has high throughput and low throughput, and it has low cost and fast recovery speed.