Negative Update Intervals in Deep Multi-Agent Reinforcement Learning

Palmer, Gregory, Savani, Rahul ORCID: 0000-0003-1262-7831 and Tuyls, Karl (2019) Negative Update Intervals in Deep Multi-Agent Reinforcement Learning .

Access the full-text of this item by clicking on the Open Access link.

Abstract

In Multi-Agent Reinforcement Learning (MA-RL), independent cooperative learners must overcome a number of pathologies to learn optimal joint policies. Addressing one pathology often leaves approaches vulnerable towards others. For instance, hysteretic Q-learning [15] addresses miscoordination while leaving agents vulnerable towards misleading stochastic rewards. Other methods, such as leniency, have proven more robust when dealing with multiple pathologies simultaneously [29]. However, leniency has predominately been studied within the context of strategic form games (bimatrix games) and fully observable Markov games consisting of a small number of probabilistic state transitions. This raises the question of whether these findings scale to more complex domains. For this purpose we implement a temporally extend version of the Climb Game [3], within which agents must overcome multiple pathologies simultaneously, including relative overgeneralisation, stochasticity, the alter-exploration and moving target problems, while learning from a large observation space. We find that existing lenient and hysteretic approaches fail to consistently learn near optimal joint-policies in this environment. To address these pathologies we introduce Negative Update Intervals-DDQN (NUI-DDQN), a Deep MA-RL algorithm which discards episodes yielding cumulative rewards outside the range of expanding intervals. NUI-DDQN consistently gravitates towards optimal joint-policies in our environment, overcoming the outlined pathologies.

Item Type:	Conference Item (Unspecified)
Additional Information:	11 Pages, 6 Figures, AAMAS2019 Conference Proceedings
Uncontrolled Keywords:	Deep Multi-Agent Reinforcement Learning
Depositing User:	Symplectic Admin
Date Deposited:	20 May 2020 09:41
Last Modified:	23 May 2026 01:41
Open Access URL:	http://www.ifaamas.org/Proceedings/aamas2019/pdfs/...
Related Websites:	Author
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3083743
Disclaimer:	The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.