Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples

Kaneko, M, Bollegala, D ORCID: 0000-0003-4476-7003 and Okazaki, N (2023) Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples. In: European Chapter of the Association for Computational Linguistics (EACL 2023), 2023-5-3 - 2023-5-5, Croatia.

Text EACL2023__Meta_evaluation_for_Social_Bias_Evaluation_Measures.pdf - Author Accepted Manuscript Download (409kB) | Preview

Abstract

Numerous types of social biases have been identified in pre-trained language models (PLMs), and various intrinsic bias evaluation measures have been proposed for quantifying those social biases. Prior works have relied on human annotated examples to compare existing intrinsic bias evaluation measures. However, this approach is not easily adaptable to different languages nor amenable to large scale evaluations due to the costs and difficulties when recruiting human annotators. To overcome this limitation, we propose a method to compare intrinsic gender bias evaluation measures without relying on human-annotated examples. Specifically, we create multiple bias-controlled versions of PLMs using varying amounts of male vs. female gendered sentences, mined automatically from an unannotated corpus using gender-related word lists. Next, each bias-controlled PLM is evaluated using an intrinsic bias evaluation measure, and the rank correlation between the computed bias scores and the gender proportions used to fine-tune the PLMs is computed. Experiments on multiple corpora and PLMs repeatedly show that the correlations reported by our proposed method that does not require human annotated examples are comparable to those computed using human annotated examples in prior work.

Item Type:	Conference or Workshop Item (Unspecified)
Divisions:	Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science
Depositing User:	Symplectic Admin
Date Deposited:	06 Feb 2023 16:15
Last Modified:	23 Nov 2023 22:56
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3168210