Difference of convex functions programming for policy optimization in reinforcement learning

Publication Type

Conference Proceeding Article

Publication Date

5-2024

Abstract

We formulate the problem of optimizing an agent's policy within the Markov decision process (MDP) model as a difference-of-convex functions (DC) program. The DC perspective enables optimizing the policy iteratively where each iteration constructs an easier-to-optimize lower bound on the value function using the well known concave-convex procedure. We show that several popular policy gradient based deep RL algorithms (both for discrete and continuous state, action spaces, and stochastic/deterministic policies) such as actor-critic, deterministic policy gradient (DPG), and soft actor critic (SAC) can be derived from the DC perspective. Additionally, the DC formulation enables more sample efficient learning approaches by exploiting the structure of the value function lower bound, and when the policy has a simpler parametric form, allows using efficient nonlinear programming solvers. Furthermore, we show that the DC perspective extends easily to constrained RL and partially observable and multiagent settings. Such connections provide new insight on previous algorithms, and also help develop new algorithms for RL.

Keywords

Agent policy, Reinforcement learning optimization, Difference-of-convex functions, Reinforcement learning algorithm

Discipline

Artificial Intelligence and Robotics

Publication

Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2024) : Auckland, New Zealand, May 6-10

First Page

2339

Last Page

2341

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

City or Country

Richland, SC

This document is currently not available here.

Share

COinS