Compact Trilinear Interaction for Visual Question Answering



Do, Tuong, Tran, Huy, Do, Thanh-Toan, Tjiputra, Erman and Tran, Quang D
(2020) Compact Trilinear Interaction for Visual Question Answering. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019-10-27 - 2019-11-2.

Access the full-text of this item by clicking on the Open Access link.

Abstract

In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear teraction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.

Item Type: Conference or Workshop Item (Unspecified)
Depositing User: Symplectic Admin
Date Deposited: 10 Sep 2020 12:14
Last Modified: 18 Jan 2023 23:34
DOI: 10.1109/ICCV.2019.00048
Open Access URL: https://arxiv.org/pdf/1909.11874.pdf
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3100673