Kaneko, M, Bollegala, D 
ORCID: 0000-0003-4476-7003 and Okazaki, N
(2022)
Debiasing isn’t enough! – On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks.
In: 29th International Conference on Computational Linguistics, 2022-10-12 - 2022-10-17, South Korea.
|
Text
COLING2022_downstream_bias (1).pdf - Author Accepted Manuscript Download (454kB) | Preview |
Abstract
We study the relationship between task-agnostic intrinsic and task-specific extrinsic social bias evaluation measures for Masked Language Models (MLMs), and find that there exists only a weak correlation between these two types of evaluation measures. Moreover, we find that MLMs debiased using different methods still re-learn social biases during fine-tuning on downstream tasks. We identify the social biases in both training instances as well as their assigned labels as reasons for the discrepancy between intrinsic and extrinsic bias evaluation measurements. Overall, our findings highlight the limitations of existing MLM bias evaluation measures and raise concerns on the deployment of MLMs in downstream applications using those measures.
| Item Type: | Conference or Workshop Item (Unspecified) |
|---|---|
| Divisions: | Faculty of Science and Engineering > School of Electrical Engineering, Electronics and Computer Science |
| Depositing User: | Symplectic Admin |
| Date Deposited: | 14 Sep 2022 08:10 |
| Last Modified: | 06 Aug 2023 05:22 |
| URI: | https://livrepository.liverpool.ac.uk/id/eprint/3164634 |
Share
Share
