A Comparative Study of Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO) on a Toy Environment

Abstract

Proximal Policy Optimization (PPO) and Direct Policy Optimization (DPO) are prominent reinforcement learning algorithms designed to improve policy learning efficiency and stability. This paper presents a comparative study between PPO and DPO applied to a simple multi-armed bandit toy environment. We implement both algorithms with comparable hyperparameters and evaluate their performance over multiple random seeds. Our experiments measure cumulative rewards, convergence speed, and learning stability. Results indicate that DPO achieves higher average rewards and faster convergence than PPO in this setting. The analysis provides insights into the operational differences between these algorithms, contributing a foundational understanding beneficial for future reinforcement learning research and ap- plications.