海量数据广义线性模型变量选择算法研究

首先推导出了用于求解一般广义线性模型变量选择问题的非凸惩罚迭代估计算法，并利用分治思想对算法进行修正，使其能够适用于海量数据情形，以解决海量数据下进行变量选择时可能存在的内存溢出等问题。考虑到当前处理海量数据实际使用的工具，进一步给出了算法在分布式并行下的计算步骤，大幅提高了计算速度。在数值模拟中，通过单机和集群两种方式对算法进行数值计算，结果表明本文方法有效解决了数据存储问题且适用于分布式环境。最后，通过所提算法来完成Probit模型的变量选择，并将其用于新闻数据集的分类问题。

A variable selection algorithm for generalized linear models of massive data

We derive a non-convex penalty iterative estimation algorithm for solving the generalized linear model variable selection problem, and then use the divide-and-conquer principle to modify the algorithm so that it can be applied to massive data problems and solve the problem of memory overflow bottlenecks which are common with massive data. Compared with the tools currently used to process massive amounts of data, our algorithm's computation steps utilize distributed parallelism, which greatly improves the calculation speed. In subsequent numerical simulations, the algorithm was numerically calculated in two ways:stand-alone and cluster. The results show that our method effectively solves the data storage problem and is suitable for distributed environments. Finally, this algorithm was used to complete the variable selection of the Probit model and used for classification of news datasets.