Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlation


Van Vo, T, Nhat Vu, M, Huang, B, Nguyen, T, Le, N, Vo, T and Nguyen, A ORCID: 0000-0002-1449-211X (2024) Open-Vocabulary Affordance Detection using Knowledge Distillation and Text-Point Correlation In: 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024-5-13 - 2024-5-17.

[thumbnail of 2023_OpenKD.pdf] Text
2023_OpenKD.pdf - Author Accepted Manuscript
Available under License Creative Commons Attribution.
Download (1MB) | Preview

Abstract

Affordance detection presents intricate challenges and has a wide range of robotic applications. Previous works have faced limitations such as the complexities of 3D object shapes, the wide range of potential affordances on real-world objects, and the lack of open-vocabulary support for affordance understanding. In this paper, we introduce a new open-vocabulary affordance detection method in 3D point clouds, leveraging knowledge distillation and text-point correlation. Our approach employs pre-trained 3D models through knowledge distillation to enhance feature extraction and semantic understanding in 3D point clouds. We further introduce a new text-point correlation method to learn the semantic links between point cloud features and open-vocabulary labels. The intensive experiments show that our approach outperforms previous works and adapts to new affordance labels and unseen objects. Notably, our method achieves the improvement of 7.96% mIOU score compared to the baselines. Furthermore, it offers real-time inference which is well-suitable for robotic manipulation applications.

Item Type: Conference Item (Unspecified)
Uncontrolled Keywords: 4605 Data Management and Data Science, 46 Information and Computing Sciences, Generic health relevance
Divisions: Faculty of Science & Engineering
Faculty of Science & Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User: Symplectic Admin
Date Deposited: 21 Nov 2024 09:56
Last Modified: 24 Jan 2026 04:59
DOI: 10.1109/ICRA57147.2024.10610247
Related Websites:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3188800
Disclaimer: The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.
Altmetric
CORE (COnnecting REpositories)