Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation

Cao, Guanqun, Jiang, Jiaqi, Mao, Ningtao, Bollegala, Danushka ORCID: 0000-0003-4476-7003, Li, Min and Luo, Shan (2023) Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023-5-29 - 2023-6-2.

PDF
2301.06826.pdf - Author Accepted Manuscript
Download (5MB) | Preview

Official URL: http://dx.doi.org/10.1109/icra48891.2023.10160373

Abstract

To assist robots in teleoperation tasks, haptic rendering which allows human operators access a virtual touch feeling has been developed in recent years. Most previous haptic rendering methods strongly rely on data collected by tactile sensors. However, tactile data is not widely available for robots due to their limited reachable space and the restrictions of tactile sensors. To eliminate the need for tactile data, in this paper we propose a novel method named as Vis2Hap to generate haptic rendering from visual inputs that can be obtained from a distance without physical interaction. We take the surface texture of objects as key cues to be conveyed to the human operator. To this end, a generative model is designed to simulate the roughness and slipperiness of the object's surface. To embed haptic cues in Vis2Hap, we use height maps from tactile sensors and spectrograms from friction coefficients as the intermediate outputs of the generative model. Once Vis2Hap is trained, it can be used to generate height maps and spectrograms of new surface textures, from which a friction image can be obtained and displayed on a haptic display. The user study demonstrates that our proposed Vis2Hap method enables users to access a realistic haptic feeling similar to that of physical objects. The proposed vision-based haptic rendering has the potential to enhance human operators' perception of the remote environment and facilitate robotic manipulation.

Item Type:	Conference or Workshop Item (Unspecified)
Uncontrolled Keywords:	Eye Disease and Disorders of Vision, Neurosciences
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	05 Sep 2023 15:12
Last Modified:	14 Mar 2024 22:24
DOI:	10.1109/ICRA48891.2023.10160373
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3172558