On Rater Reliability and Agreement Based Dynamic Active Learning

Zhang, Yue, Coutinho, Eduardo ORCID: 0000-0001-5234-1497, Schuller, Bjoern, Zhang, Zixing and Adam, Michael (2015) On Rater Reliability and Agreement Based Dynamic Active Learning. In: 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), 2015-9-21 - 2015-9-24.

Text ACII2013_DAL_CameraReady.pdf - Author Accepted Manuscript Download (253kB)

Abstract

In this paper, we propose two novel Dynamic Active Learning (DAL) methods with the aim of ultimately reducing the costly human labelling work for subjective tasks such as speech emotion recognition. Compared to conventional Active Learning (AL) algorithms, the proposed DAL approaches employ a highly efficient adaptive query strategy that minimises the number of annotations through three advancements. First, we shift from the standard majority voting procedure, in which unlabelled instances are annotated by a fixed number of raters, to an agreement-based annotation technique that dynamically determines how many human annotators are required to label a selected instance. Second, we introduce the concept of the order-based DAL algorithm by considering rater reliability and inter-rater agreement. Third, a highly dynamic development trend is successfully implemented by upgrading the agreement levels depending on the prediction uncertainty. In extensive experiments on standardised test-beds, we show that the new dynamic methods significantly improve the efficiency of the existing AL algorithms by reducing human labelling effort up to 85.41%, while achieving the same classification accuracy. Thus, the enhanced DAL derivations opens up high-potential research directions for the utmost exploitation of unlabelled data.

Item Type:	Conference or Workshop Item (Unspecified)
Additional Information:	(acceptance rate oral: 28\,%))
Uncontrolled Keywords:	Active learning algorithms, adaptive query strategies, rater reliability, inter-rater agreement
Depositing User:	Symplectic Admin
Date Deposited:	12 Aug 2016 09:54
Last Modified:	19 Jan 2023 07:32
DOI:	10.1109/acii.2015.7344553
Related URLs:	Author Publisher
URI:	https://livrepository.liverpool.ac.uk/id/eprint/3002899