Neill, James O' and Bollegala, Danushka
(2018)
Angular-Based Word Meta-Embedding Learning.
CoRR, abs/18.
![[img]](https://livrepository.liverpool.ac.uk/style/images/fileicons/text.png) |
Text
1808.04334v1.pdf - Submitted version Download (286kB) |
Abstract
Ensembling word embeddings to improve distributed word representations has shown good success for natural language processing tasks in recent years. These approaches either carry out straightforward mathematical operations over a set of vectors or use unsupervised learning to find a lower-dimensional representation. This work compares meta-embeddings trained for different losses, namely loss functions that account for angular distance between the reconstructed embedding and the target and those that account normalized distances based on the vector length. We argue that meta-embeddings are better to treat the ensemble set equally in unsupervised learning as the respective quality of each embedding is unknown for upstream tasks prior to meta-embedding. We show that normalization methods that account for this such as cosine and KL-divergence objectives outperform meta-embedding trained on standard $\ell_1$ and $\ell_2$ loss on \textit{defacto} word similarity and relatedness datasets and find it outperforms existing meta-learning strategies.
| Item Type: | Article |
|---|---|
| Additional Information: | 5 pages, 2 figures |
| Uncontrolled Keywords: | cs.CL, cs.CL, cs.LG, stat.ML |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 12 Sep 2018 09:26 |
| Last Modified: | 19 Jan 2023 01:25 |
| Related URLs: | |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3025870 |
Share
Share
