Spark日志整合与FCM-DNN的网络流量分析算法

为了解决网络设备类型划分粒度粗,导致网络流量无法准确分类的问题,提出了一种基于Spark日志集成与FCM-DNN的流量分析算法。首先,该方法使用Spark集成会话日志以获取可分析的结构化数据；然后对同一网站的行为数据进行聚类,提取网站的多类簇特征集合,以解决单个会话连接特征维度较少、特征相似且不平衡的问题；最后,构建DNN网络,将统一化后的聚类特征与原始特征结合进行训练,并从聚类分组长度和损失函数等多个方面进行算法优化。仿真实验结果表明,对于特征较少的会话日志数据,该算法有效提高了网站分类的准确性,同时在保留学生上网特征的前提下将日志压缩了700倍,从而节省了存储开销。

Network flow traffic analysis algorithm based on Spark log integration and FCM-DNN

To address the challenge of accurately classifying network traffic, attributed to the limited granularity in categorizing network devices, a novel traffic analysis algorithm leveraging Spark log integration and FCM-DNN is proposed. Firstly, the method employs Spark to consolidate session logs, yielding structured and analyzable data. Subsequently, clustering is applied to group behavior data from the same website, thereby extracting a multi-cluster feature collection. This approach mitigates issues stemming from insufficient connection feature dimensions and imbalanced and similar features of a single session. Finally, the method constructs a DNN network and combine the unified cluster features with original features for training, optimizing the algorithm across various dimensions, such as cluster grouping length and loss functions. Simulation experiments demonstrate that, even with session log data containing fewer features, our algorithm significantly enhances website classification accuracy and reduces storage overhead by compressing logs by a factor of 700, while retaining critical student online features.