基于动态行为特征加权聚类的加壳恶意软件未知变种检测方法

攻击者为了逃避检测，常利用加壳技术对恶意软件进行加密或压缩，使得安全分析人员以及传统基于静态分析的恶意软件检测方法在恶意软件运行前难以利用反汇编等逆向工具对其进行静态分析。为检测加壳恶意软件，当前主要采用动态分析方法检测加壳恶意软件，然而受限于加壳工具种类和样本规模，以及恶意软件加壳行为带来的混淆噪声，导致传统基于机器学习检测方法存在准确率不足等问题。研究提取并分析加壳恶意软件运行时的系统调用行为特征，识别并筛选出敏感行为，旨在过滤脱壳行为噪声产生的影响；通过对系统调用行为特征加权降维，提升行为特征的有效性；通过对加权降维的行为特征进行聚类分析，最终实现加壳恶意软件未知变种检测和检测模型增量更新。实验结果表明，提出的基于动态行为特征加权聚类的加壳恶意软件未知变种检测方法检测误报率3.9%,相较几种典型机器学习检测方法呈显著降低。

A packed malware variants detection method based on weighted dynamic behaviour feature clustering

In order to avoid malware detection, attackers often use packing techniques to encrypt or compress malware binaries, which makes it difficult for security analysts and malware detectors based on traditional static analysis to use reverse tools, such as disassembly tools, to statically analyze malware before it runs. Currently, to detect packed malware, dynamic analysis methods are mainly used. However, due to the limitation of the types of packing tools and packed samples, as well as the confusion noise caused by malware packers, traditional machine learning based detection methods have insufficient accuracy. In this paper, to filter the packing behavior, the system call behavior features of packed malware are extracted and analyzed, and then sensitive behaviors are identified and filtered out. Next, the feature dimensions of system call behaviours are reduced by weighting to improve the contribution of each feature. Finally, these behaviours are analyzed by using density-based clustering, realizing the detection of unknown variants of packed malware and the update of the detection model. The experimental results show that the proposed packed malware variants detection method based on weighted clustering of sensitive behavior features achieves 3.9 % false alter rate and significantly reduces the false alter rate compared with that of some other machine learning-based detection methods.