Learning to Evaluate Neural Language Models



O’Neill, James and Bollegala, Danushka ORCID: 0000-0003-4476-7003
(2020) Learning to Evaluate Neural Language Models. In: 16th International Conference of the Pacific Association for Computational Linguistics (PACLING), 2019-10-10 - 2019-10-13, Hanoi, Vietnam.

[img] Text
ONeill_PACLING_2019.pdf - Author Accepted Manuscript

Download (221kB) | Preview

Abstract

Evaluating the performance of neural network-based text generators and density estimators is challenging since no one measure perfectly evaluates language quality. Perplexity has been a mainstay metric for neural language models trained by maximizing the conditional log-likelihood. We argue perplexity alone is a naive measure since it does not explicitly take into account the semantic similarity between generated and target sentences. Instead, it relies on measuring the cross-entropy between the targets and predictions on the word-level, while ignoring alternative incorrect predictions that may be semantically similar and globally coherent, thus ignoring quality of neighbouring tokens that may be good candidates. This is particularly important when learning from smaller corpora where co-occurrences are even more sparse. Thus, this paper proposes the use of a pretrained model-based evaluation that assesses semantic and syntactic similarity between predicted sequences and target sequences. We argue that this is an improvement over perplexity which does not distinguish between incorrect predictions that vary in semantic distance to the target words. We find that models that outperform other models using perplexity as an evaluation metric on Penn-Treebank and WikiText-2, do not necessarily perform better on measures that evaluate using semantic similarity.

Item Type: Conference or Workshop Item (Unspecified)
Depositing User: Symplectic Admin
Date Deposited: 02 Sep 2019 08:30
Last Modified: 19 Jan 2023 00:28
DOI: 10.1007/978-981-15-6168-9_11
Related URLs:
URI: https://livrepository.liverpool.ac.uk/id/eprint/3053008