适用于强化学习惯性环境的分数阶改进OU噪声

本文将DDPG算法中使用的Ornstein-Uhlenbeck (OU)噪声整数阶微分模型推广为分数阶OU噪声模型,使得噪声的产生不仅和前一步的噪声有关而且和前K步产生的噪声都有关联.通过在gym惯性环境下对比基于分数阶OU噪声的DDPG和TD3算法和原始的DDPG和TD3算法,我们发现基于分数阶微积分的OU噪声相比于原始的OU噪声能在更大范围内震荡,使用分数阶OU噪声的算法在惯性环境下具有更好的探索能力,收敛得更快.

An improved Ornstein-Uhlenbeck exploration noise based on fractional order calculus for reinforcement learning environments with momentum

In this paper, the integer-order Ornstein-Uhlenbeck (OU) noise model used in the deep deterministic policy gradient (DDPG) algorithm is extended to the fractional order OU noise model, and the generated noise is not only related to the noise of the previous step but also related to the noise generated in the previous K steps in the proposed model.The DDPG algorithm and twin delayed deep deterministic(TD3) algorithm using the fractional-order OU noise model were compared with the original DDPG algorithm and TD3 algorithm in the gym inertial environment. We found that, compared with the original OU noise, the fractional-order OU noise can oscillate in a wider range, and the algorithm using the fractional-order OU noise had better exploration ability and faster convergence in inertial environment.