Topic driven multimodal similarity learning with multi-view voted convolutional features

Gao, Xinjian, Mu, Tingting, Goulermas, John Y and Wang, Meng (2018) Topic driven multimodal similarity learning with multi-view voted convolutional features. PATTERN RECOGNITION, 75. pp. 223-234.

Text manuscript.pdf - Author Accepted Manuscript Download (4MB)

Abstract

Similarity (and distance metric) learning plays a very important role in many artificial intelligence tasks aiming at quantifying the relevance between objects. We address the challenge of learning complex relation patterns from data objects exhibiting heterogeneous properties, and develop an effective multi-view multimodal similarity learning model with much improved learning performance and model interpretability. The proposed method firstly computes multi-view convolutional features to achieve improved object representation, then analyses the similarities between objects by operating over multiple hidden relation types (modalities), and finally fine-tunes the entire model variables via back-propagating a ranking loss to the convolutional layers. We develop a topic-driven initialization scheme, so that each learned relation type can be interpreted as a representative of semantic topics of the objects. To improve model interpretability and generalization, sparsity is imposed over these hidden relations. The proposed method is evaluated by solving the image retrieval task using challenging image datasets, and is compared with seven state-of-the-art algorithms in the field. Experimental results demonstrate significant performance improvement of the proposed method over the competing ones.

Item Type:	Article
Uncontrolled Keywords:	Convolutional auto-encoder, Representation learning, Multi-view learning, Multimodal similarity learning
Depositing User:	Symplectic Admin
Date Deposited:	25 May 2017 06:51
Last Modified:	19 Jan 2023 07:03
DOI:	10.1016/j.patcog.2017.02.035
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3007635