基于动态延迟策略更新的TD3 算法

在深度强化学习领域中，为进一步减少双延迟深度确定性策略梯度TD3( Twin Delayed Deep DeterministicPolicy Gradients) 中价值过估计对策略估计的影响，加快模型学习的效率，提出一种基于动态延迟策略更新的双延迟深度确定性策略梯度( DD-TD3: Twin Delayed Deep Deterministic Policy Gradients with Dynamic Delayed PolicyUpdate) 。在DD-TD3 方法中，通过Critic 网络的最新Loss 值与其指数加权移动平均值的动态差异指导Actor 网络的延迟更新步长。实验结果表明，与原始TD3 算法在2 000 步获得较高的奖励值相比，DD-TD3 方法可在约1 000步内学习到最优控制策略，并且获得更高的奖励值，从而提高寻找最优策略的效率。

TD3 Algorithm with Dynamic Delayed Policy Update

In the field of deep reinforcement learning， in order to further reduce the impact of valueoverestimation on policy estimation in TD3 ( Twin Delayed Deep Deterministic Policy Gradients) and acceleratethe efficiency of model learning，a DD-TD3 ( Twin Delayed Deep Deterministic Policy Gradients with DynamicDelayed Policy Update) is proposed． The delay update step size of the actor network is guided by the dynamicdifference between the latest loss of the critic network and its exponential weighted moving average． Experimentalresults show that compared to the original TD3 algorithm that obtain high reward value in the 2 000 steps，theDD-TD3 method can learn the optimal control strategy in about 1 000 steps and obtain a higher reward value，thereby the efficiency of finding the optimal strategy is improved．