Meta-Embedding as Auxiliary Task Regularization


O'Neill, James and Bollegala, Danushka ORCID: 0000-0003-4476-7003 (2020) Meta-Embedding as Auxiliary Task Regularization. ECAI 2020: 24TH EUROPEAN CONFERENCE ON ARTIFICIAL INTELLIGENCE, 325. pp. 2124-2131.

Access the full-text of this item by clicking on the Open Access link.
[img] Text
1809.05886v1.pdf - Submitted version
Download (409kB)

Abstract

Word embeddings have been shown to benefit from ensambling several word embedding sources, often carried out using straightforward mathematical operations over the set of word vectors. More recently, self-supervised learning has been used to find a lower-dimensional representation, similar in size to the individual word embeddings within the ensemble. However, these methods do not use the available manually labeled datasets that are often used solely for the purpose of evaluation. We propose to reconstruct an ensemble of word embeddings as an auxiliary task that regularises a main task while both tasks share the learned meta-embedding layer. We carry out intrinsic evaluation (6 word similarity datasets and 3 analogy datasets) and extrinsic evaluation (4 downstream tasks). For intrinsic task evaluation, supervision comes from various labeled word similarity datasets. Our experimental results show that the performance is improved for all word similarity datasets when compared to self-supervised learning methods with a mean increase of $11.33$ in Spearman correlation. Specifically, the proposed method shows the best performance in 4 out of 6 of word similarity datasets when using a cosine reconstruction loss and Brier's word similarity loss. Moreover, improvements are also made when performing word meta-embedding reconstruction in sequence tagging and sentence meta-embedding for sentence classification.

Item Type: Article
Uncontrolled Keywords: cs.CL, cs.CL, cs.LG
Depositing User: Symplectic Admin
Date Deposited: 19 Oct 2018 10:10
Last Modified: 19 Jan 2023 01:15
DOI: 10.3233/FAIA200336
Open Access URL: https://ebooks.iospress.nl/publication/55131
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3027043
Dimensions
Altmetric
CORE (COnnecting REpositories)