Using k-Way Co-Occurrences for Learning Word Embeddings

Bollegala, Danushka ORCID: 0000-0003-4476-7003, Yoshida, Yuichi and Kawarabayashi, Ken-ichi (2018) Using k-Way Co-Occurrences for Learning Word Embeddings. In: 32nd AAAI Conference on Artificial Intelligence, 2018-2-2 - 2018-2-7, New Orleans, Louisiana, USA.

	Text context.pdf - Author Accepted Manuscript Download (581kB)
	Text context.pdf - Author Accepted Manuscript Download (581kB)

Abstract

<jats:p> Co-occurrences between two words provide useful insights into the semantics of those words.Consequently, numerous prior work on word embedding learning has used co-occurrences between two wordsas the training signal for learning word embeddings.However, in natural language texts it is common for multiple words to be related and co-occurring in the same context.We extend the notion of co-occurrences to cover k(≥2)-way co-occurrences among a set of k-words.Specifically, we prove a theoretical relationship between the joint probability of k(≥2) words, and the sum of l_2 norms of their embeddings. Next, we propose a learning objective motivated by our theoretical resultthat utilises k-way co-occurrences for learning word embeddings.Our experimental results show that the derived theoretical relationship does indeed hold empirically, anddespite data sparsity, for some smaller k(≤5) values, k-way embeddings perform comparably or better than 2-way embeddings in a range of tasks. </jats:p>

Item Type:	Conference or Workshop Item (Unspecified)
Uncontrolled Keywords:	Clinical Research
Depositing User:	Symplectic Admin
Date Deposited:	18 Apr 2018 14:48
Last Modified:	15 Mar 2024 02:25
DOI:	10.1609/aaai.v32i1.12010
Related URLs:	Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3020270

Available Versions of this Item

• Using k-way Co-occurrences for Learning Word Embeddings. (deposited 09 Jan 2018 10:12)
  • Using k-Way Co-Occurrences for Learning Word Embeddings. (deposited 18 Apr 2018 14:48) [Currently Displayed]