TASK ADAPTATION FROM SKILLS: INFORMATION GEOMETRY, DISENTANGLEMENT, AND NEW OBJECTIVES FOR UNSUPERVISED REINFORCEMENT LEARNING



Yang, Y, Zhou, T, He, Q, Han, L, Pechenizkiy, M and Fang, M ORCID: 0000-0001-6745-286X
(2024) TASK ADAPTATION FROM SKILLS: INFORMATION GEOMETRY, DISENTANGLEMENT, AND NEW OBJECTIVES FOR UNSUPERVISED REINFORCEMENT LEARNING .

[thumbnail of REF_Task_Adaptation_from_Skills__Information_Geometry__Disentanglement__and_New_Objectives_for_Unsupervised_Reinforcement_Learning.pdf] PDF
REF_Task_Adaptation_from_Skills__Information_Geometry__Disentanglement__and_New_Objectives_for_Unsupervised_Reinforcement_Learning.pdf - Submitted version

Download (1MB) | Preview

Abstract

Unsupervised reinforcement learning (URL) aims to learn general skills for unseen downstream tasks. Mutual Information Skill Learning (MISL) addresses URL by maximizing the mutual information between states and skills but lacks sufficient theoretical analysis, e.g., how well its learned skills can initialize a downstream task's policy. Our new theoretical analysis in this paper shows that the diversity and separability of learned skills are fundamentally critical to downstream task adaptation but MISL does not necessarily guarantee these properties. To complement MISL, we propose a novel disentanglement metric LSEPIN. Moreover, we build an information-geometric connection between LSEPIN and downstream task adaptation cost. For better geometric properties, we investigate a new strategy that replaces the KL divergence in information geometry with Wasserstein distance. We extend the geometric analysis to it, which leads to a novel skill-learning objective WSEP. It is theoretically justified to be helpful to downstream task adaptation and it is capable of discovering more initial policies for downstream tasks than MISL. We finally propose another Wasserstein distance-based algorithm PWSEP that can theoretically discover all optimal initial policies.

Item Type: Conference Item (Unspecified)
Divisions: Faculty of Science & Engineering
Faculty of Science & Engineering > School of Computer Science & Informatics
Faculty of Science & Engineering > School of Computer Science & Informatics > Artificial Intelligence
Depositing User: Symplectic Admin
Date Deposited: 19 Dec 2025 08:23
Last Modified: 19 Dec 2025 08:23
URI: https://livrepository.liverpool.ac.uk/id/eprint/3196188
Disclaimer: The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.