A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation

Jiang, Jiaqi, Cao, Guanqun, Thanh-Toan, Do and Luo, Shan ORCID: 0000-0003-4760-0372 (2022) A4T: Hierarchical Affordance Detection for Transparent Objects Depth Reconstruction and Manipulation. IEEE ROBOTICS AND AUTOMATION LETTERS, 7 (4). pp. 9826-9833.

Access the full-text of this item by clicking on the Open Access link.

Official URL: http://dx.doi.org/10.1109/lra.2022.3191231

Abstract

Transparent objects are widely used in our daily lives and therefore robots need to be able to handle them. However, transparent objects suffer from light reflection and refraction, which makes it challenging to obtain the accurate depth maps required to perform handling tasks. In this letter, we propose a novel affordance-based framework for depth reconstruction and manipulation of transparent objects, named A4T. A hierarchical AffordanceNet is first used to detect the transparent objects and their associated affordances that encode the relative positions of an object's different parts. Then, given the predicted affordance map, a multi-step depth reconstruction method is used to progressively reconstruct the depth maps of transparent objects. Finally, the reconstructed depth maps are employed for the affordance-based manipulation of transparent objects. To evaluate our proposed method, we construct a real-world dataset TRANS-AFF with affordances and depth maps of transparent objects, which is the first of its kind. Extensive experiments show that our proposed methods can predict accurate affordance maps, and significantly improve the depth reconstruction of transparent objects compared to the state-of-the-art method, with the Root Mean Squared Error in meters significantly decreased from 0.097 to 0.042. Furthermore, we demonstrate the effectiveness of our proposed method with a series of robotic manipulation experiments on transparent objects.

Item Type:	Article
Uncontrolled Keywords:	Computer vision for automation, robotics and automation in life sciences
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	13 Sep 2022 13:11
Last Modified:	18 Jan 2023 20:45
DOI:	10.1109/LRA.2022.3191231
Open Access URL:	https://arxiv.org/pdf/2207.04907.pdf
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3164522