Evaluating the Evaluation of Diversity in Commonsense Generation

Zhang, T, Peng, B and Bollegala, D ORCID: 0000-0003-4476-7003 (2025) Evaluating the Evaluation of Diversity in Commonsense Generation In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2025-7 - 2025-7, Vienna, Austria.

Text Evaluating_Diversity_Metrics (1).pdf - Author Accepted Manuscript Available under License Creative Commons Attribution. Download (561kB) | Preview

Abstract

In commonsense generation, given a set of input concepts, a model must generate a response that is not only commonsense bearing, but also capturing multiple diverse viewpoints. Numerous evaluation metrics based on form- and content-level overlap have been proposed in prior work for evaluating the diversity of a commonsense generation model. However, it remains unclear as to which metrics are best suited for evaluating the diversity in commonsense generation. To address this gap, we conduct a systematic meta-evaluation of diversity metrics for commonsense generation. We find that form-based diversity metrics tend to consistently overestimate the diversity in sentence sets, where even randomly generated sentences are assigned overly high diversity scores. We then use an Large Language Model (LLM) to create a novel dataset annotated for the diversity of sentences generated for a commonsense generation task, and use it to conduct a meta-evaluation of the existing diversity evaluation metrics. Our experimental results show that content-based diversity evaluation metrics consistently outperform the form-based counterparts, showing high correlations with the LLM-based ratings. We recommend that future work on commonsense generation should use content-based metrics for evaluating the diversity of their outputs.

Item Type:	Conference Item (Unspecified)
Divisions:	Faculty of Science & Engineering Faculty of Science & Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	03 Jun 2025 08:22
Last Modified:	23 May 2026 10:16
DOI:	10.18653/v1/2025.acl-long.1181
Related Websites:	Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3193018
Disclaimer:	The University of Liverpool is not responsible for content contained on other websites from links within repository metadata. Please contact us if you notice anything that appears incorrect or inappropriate.