基于大数据技术的AI岗位需求分析研究

近年来,我国人才市场出现供需失配的结构性矛盾,尤其是在人工智能(AI)领域。准确感知并描述劳动力市场的需求是解决该问题的重要手段。本研究首先使用网络爬虫抓取智联招聘网站发布的AI岗位相关招聘信息,通过中文分词、K-means等大数据分析方法对招聘岗位名称进行聚类处理,识别出软件工程师、算法工程师、产品经理及产品架构师等4个岗位簇;然后利用概率主题模型(Latent Dirichlet Allocation, LDA)对招聘岗位要求继续进行聚类处理,得到数据库、机器学习、模式识别、大数据、程序设计等5个技能集;最后利用LDA求得岗位簇对其技能集的需求矩阵,以分析各岗位簇对其岗位技能的需求程度。结果表明:程序设计能力对AI软件工程师最重要,模式识别的理论与技术对算法工程师最重要;产品经理岗位对数据库、机器学习和大数据技术等均有较强的技能需求;机器学习的理论与技术对产品架构师最重要。本研究成果可为高校、企业常态化或实时准确感知并描述AI劳动力市场需求提供技术支持。

Research on AI Job Demand Analysis Based on Big Data Technology

In recent years, there has been a structural contradiction of mismatch between supply and demand in China''s talent market, especially in the field of artificial intelligence (AI). Accurately perceiving and describing the demand of the labor market is an important method to solve this problem. This study first takes a web crawler to capture AI job-related recruitment information published by the recruitment website of Zhilian. Through Chinese word segmentation, K-means and other big data analysis methods, the recruitment job names are clustered to identify four job clusters:Software engineers, algorithm engineers, product managers and product architects. Then, the model of Latent Dirichlet Allocation (LDA) is used to cluster the job requirements, and five skill sets including database, machine learning, pattern recognition, big data and program design are obtained. Finally, LDA is used to obtain the demand matrix of job clusters for their skill sets to analyze the demand degree of each job cluster to their job skills. The results show that the programming ability is the most important for AI software engineers, and the theory and technology of pattern recognition are the most important for algorithm engineers. Product manager positions have strong skills requirements for databases, machine learning and big data technology. The theory and technology of machine learning are the most important to product architects. The results of this research can provide technical support for universities and enterprises to accurately perceive and describe the needs of the AI labor market in real time.