Ni'Mah, I, Fang, M 
ORCID: 0000-0001-6745-286X, Menkovski, V and Pechenizkiy, M
(2023)
NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist.
In: The 61st Annual Meeting of the Association for Computational Linguistics, 2023-7-9 - ?, Toronto, Canada.
|
Text
ACL_2023_NLG_Evaluation_Metrics_Beyond-camera.pdf - Submitted version Download (560kB) | Preview |
Abstract
In this study, we analyze automatic evaluation metrics for Natural Language Generation (NLG), specifically task-agnostic metrics and human-aligned metrics. Task-agnostic metrics, such as Perplexity, BLEU, BERTScore, are cost-effective and highly adaptable to diverse NLG tasks, yet they have a weak correlation with human. Human-aligned metrics (CTC, CtrlEval, UniEval) improves correlation level by incorporating desirable human-like qualities as training objective. However, their effectiveness at discerning system-level performance and quality of system outputs remain unclear. We present metric preference checklist as a framework to assess the effectiveness of automatic metrics in three NLG tasks: Text Summarization, Dialogue Response Generation, and Controlled Generation. Our proposed framework provides access: (i) for verifying whether automatic metrics are faithful to human preference, regardless of their correlation level to human; and (ii) for inspecting the strengths and limitations of NLG systems via pairwise evaluation. We show that automatic metrics provide a better guidance than human on discriminating system-level performance in Text Summarization and Controlled Generation tasks. We also show that multi-aspect human-aligned metric (UniEval) is not necessarily dominant over single-aspect human-aligned metrics (CTC, CtrlEval) and task-agnostic metrics (BLEU, BERTScore), particularly in Controlled Generation tasks.
| Item Type: | Conference or Workshop Item (Unspecified) |
|---|---|
| Divisions: | Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 24 May 2023 08:06 |
| Last Modified: | 31 Oct 2023 03:11 |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3170615 |
Altmetric
Altmetric
