| 基于关键属性组的相似重复记录检测方法研究 |
| |
| 引用本文： | 宋国兴,周 喜,马 博,赵 凡. 基于关键属性组的相似重复记录检测方法研究[J]. 科学技术与工程, 2017, 17(19) |
| |
| 作者姓名： | 宋国兴 周 喜 马 博 赵 凡 |
| |
| 作者单位： | 中科院新疆理化技术研究所,中科院新疆理化技术研究所,中科院新疆理化技术研究所,中科院新疆理化技术研究所 |
| |
| 基金项目： | 新疆维吾尔自治区高技术计划项目(201512103)；新疆维吾尔自治区重点实验室项目(2016D03019)；新疆维吾尔自治区高层次人才引进工程 (Y639401201)；中国科学院西部博士项目(XBBS201315) |
| |
| 摘 要： | 对相似重复记录的检测是数据预处理中的关键环节。针对海量数据相似重复记录检测过程中,记录属性维度过高导致的查准率和时间效率较低的问题,提出了基于统一互信息的关键属性组查找算法,通过过滤噪音属性,降低属性维度,达到准确检测相似重复记录和提高时间效率的目的。最后,在真实数据集上通过对该算法和基于原始数据所有属性的相似重复记录检测算法在准确率和效率方面的比较,验证了所提出算法的有效性。
|
| 关 键 词： | 统一互信息 关键属性组 降低维度 相似重复记录 噪音属性 |
| 收稿时间： | 2016-12-28 |
| 修稿时间： | 2017-02-09 |
Similar duplicate records detection based on key attribute group |
| |
| songguoxing,zhouxi,mabo and zhaofan. Similar duplicate records detection based on key attribute group[J]. Science Technology and Engineering, 2017, 17(19) |
| |
| Authors: | songguoxing zhouxi mabo zhaofan |
| |
| Affiliation: | Xinjiang Institute of physical and chemical technology Chinese Academy of Sciences,,, |
| |
| Abstract: | The detection of similar duplicate records is a key link in data preprocessing. In the process of similar duplicate records detection for massive data, for the problem of low precision and low time efficiency caused by high attribute dimension of a record, a key attribute group search algorithm based on unified mutual information is proposed. The attribute dimension can be reduced by filtering the noise attribute. And the purpose of accurately detecting similar duplicate records and improving time efficiency is achieved. Finally, on the real data set, the accuracy and efficiency of the algorithm are compared with the similar duplicate records based on all attributes of the original data, and the validity of the proposed algorithm is verified. |
| |
| Keywords: | unified mutual information key attribute group dimension reduction similar duplicate records noise attribute |
|
| 点击此处可从《科学技术与工程》浏览原始摘要信息 |
|
点击此处可从《科学技术与工程》下载全文 |
|
