基于HDFS的分布式文件系统

在现有的开源分布式文件存储系统HDFS上，构建一个智能大数据存储系统IHDFS.该系统提出了大数据去重模块、大数据放置模块、大数据智能迁移模块和大数据编码模块，构造了智能分布式文件存储系统，可以提高用户访问效率，节省集群的存储空间.实验结果表明，数据去重模块很好地节省了存储空间;数据放置模块合理地分配文件上传的存储层，使数据上传速度提高一倍;数据智能迁移模块提高了用户在高等存储层上文件的命中率，提高了用户获取数据的效率;数据编码模块节省了集群的储存空间，节省了大约原来存储空间的三分之一.

A Distributed File System Based on HDFS

This paper establishes an intelligent big data storage system IHDFS， based on the existing open source distributed file storage system HDFS. The system proposes and implements big data de-duplication module， big data placement module， big data intelligent migration module， and big data encoding module， which improves the efficiency of user visits and saves the storage space of the cluster. Experimental results show that the data de-duplication module can save the storage space. The data placement module provides a reasonable distribution of file upload storage layer， which twice the uploading speed; the data intelligent migration module improves the hit rate of files on the upper storage layer， which improves the efficiency of obtaining data; the data encoding module saves the storage space of the cluster about one third of the original.