Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy



Zhou, Yi ORCID: 0000-0001-7009-8515 and Bollegala, Danushka ORCID: 0000-0003-4476-7003
(2021) Learning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a Proxy. .

[img] Text
Sense_Embedding_PACLIC_2021 (1).pdf - Accepted Version

Download (520kB) | Preview

Abstract

Contextualised word embeddings generated from Neural Language Models (NLMs), such as BERT, represent a word with a vector that considers the semantics of the target word as well its context. On the other hand, static word embeddings such as GloVe represent words by relatively low-dimensional, memory- and compute-efficient vectors but are not sensitive to the different senses of the word. We propose Context Derived Embeddings of Senses (CDES), a method that extracts sense related information from contextualised embeddings and injects it into static embeddings to create sense-specific static embeddings. Experimental results on multiple benchmarks for word sense disambiguation and sense discrimination tasks show that CDES can accurately learn sense-specific static embeddings reporting comparable performance to the current state-of-the-art sense embeddings.

Item Type: Conference or Workshop Item (Unspecified)
Additional Information: Accepted to PACLIC 35
Uncontrolled Keywords: cs.CL, cs.CL
Divisions: Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User: Symplectic Admin
Date Deposited: 08 Oct 2021 15:12
Last Modified: 27 Apr 2022 18:11
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3139728