"Touching to See" and "Seeing to Feel": Robotic Cross-modal Sensory Data Generation for Visual-Tactile Perception

Lee, Jet-Tsyn, Bollegala, Danushka ORCID: 0000-0003-4476-7003 and Luo, Shan (2019) "Touching to See" and "Seeing to Feel": Robotic Cross-modal Sensory Data Generation for Visual-Tactile Perception In: 2019 International Conference on Robotics and Automation (ICRA), 2019-5-20 - 2019-5-24, Montreal, Canada.

Text ICRA 2019_Jet.pdf - Author Accepted Manuscript Download (4MB)

Abstract

The integration of visual-tactile stimulus is common while humans performing daily tasks. In contrast, using unimodal visual or tactile perception limits the perceivable dimensionality of a subject. However, it remains a challenge to integrate the visual and tactile perception to facilitate robotic tasks. In this paper, we propose a novel framework for the cross-modal sensory data generation for visual and tactile perception. Taking texture perception as an example, we apply conditional generative adversarial networks to generate pseudo visual images or tactile outputs from data of the other modality. Extensive experiments on the ViTac dataset of cloth textures show that the proposed method can produce realistic outputs from other sensory inputs. We adopt the structural similarity index to evaluate similarity of the generated output and real data and results show that realistic data have been generated. Classification evaluation has also been performed to show that the inclusion of generated data can improve the perception performance. The proposed framework has potential to expand datasets for classification tasks, generate sensory outputs that are not easy to access, and also advance integrated visual-tactile perception.

Item Type:	Conference Item (Unspecified)
Additional Information:	7 pages, IEEE International Conference on Robotics and Automation 2019
Uncontrolled Keywords:	cs.RO, cs.RO
Depositing User:	Symplectic Admin
Date Deposited:	18 Feb 2019 09:39
Last Modified:	22 May 2026 19:47
DOI:	10.1109/ICRA.2019.8793763
Related Websites:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3032995
Disclaimer:	The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.