一种基于混合索引的HDFS小文件存储策略

Hadoop分布式文件系统(hadoop distributed file system,HDFS)因其稳定高效、低成本等优势,已被很多大型企业广泛使用.针对HDFS海量小文件存储时元数据服务器节点内存开销过大,合并文件中小文件访问效率不高的问题,提出一种改进的基于混合索引的小文件存储策略,应用分类器分类标记小文件,并在元数据服务器建立H-B+树索引,在存储节点根据小文件大小建立不同的块内索引,达到提高小文件访问效率的目标.实现中采用缓存结构,以提高客户端访问的响应速度,同时也有利于元数据服务器节点的内存负载.实验结果表明,基于混合索引的小文件存储策略能有效提高小文件访问效率,并显著降低元数据节点内存开销.

A kind of HDFS small files storage strategy based on hybrid index

Benefiting from its stability and efficiency, low-cost storage capability, Hadoop distributed file system HDFS has been widely used in many large enterprises. However, processing massive small files on HDFS costs too much memory overhead of NameNode, and the efficiency of accessing small files from merged file is not satisfactory. To deal with these two issues, this paper proposes an optimizing strategy for storing and accessing small files, that is Small Files Storage Strategy Based on Hybrid Index.Firstly, the strategy classifies and marks small files by a classifier. Secondly, H - B + _tree index on NameNode and different block index on DataNode are created to improve small file access efficiency. Finally, to enhance the response speed of the client access request, cache structure is used to release memory load of NameNode. The experimental results indicate that the strategy is able to improve small file access efficiency, and significantly reduce the memory overhead of NameNode.