MADDPG算法并行优先经验回放机制

多智能体深度确定性策略梯度(multi-agent deep deterministic policy gradient, MADDPG)算法是深度强化学习方法在多智能体系统(multi-agent system, MAS)领域的重要运用,为提升算法性能,提出基于并行优先经验回放机制的MADDPG算法。分析算法框架及训练方法,针对算法集中式训练、分布式执行的特点,采用并行方法完成经验回放池数据采样,并在采样过程中引入优先回放机制,实现经验数据并行流动、数据处理模型并行工作、经验数据优先回放。分别在OpenAI多智能体对抗、合作两类典型环境中,从训练轮数、训练时间两个维度对改进算法进行了对比验证,结果表明,并行优先经验回放机制的引入使得算法性能提升明显。

Parallel priority experience replay mechanism of MADDPG algorithm

The multi-agent deep deterministic policy gradient(MADDPG)algorithm is an important algorithm for deep reinforcement learning in the field of the multi-agent system(MAS).To improve the performance of the algorithm,the parallel priority experience replay mechanism of the algorithm is proposed.The algorithm framework and training method are analyzed.Aiming at the characteristics of centralized training and distributed execution of the algorithm,the multi-agent experience replay pool data sampling is completed by using the parallel method,and the priority experience replay mechanism is introduced in the sampling process.Thus,the parallel flow of empirical data is realized,the data processing model works in parallel,and the empirical data is prior replayed.Finally,the improved algorithm is compared and verified from the two dimensions of the training episode and the training time respectively in the typical environment of OpenAI multi-agent confrontation and cooperation.The results show that the introduction of the parallel prior experience replay mechanism makes the efficiency of the algorithm being improved obviously.