The University of Southampton
University of Southampton Institutional Repository

Quality assessment in crowdsourced classification tasks

Quality assessment in crowdsourced classification tasks
Quality assessment in crowdsourced classification tasks
Purpose: Ensuring quality is one of the most significant challenges in microtask crowdsourcing. Aggregation of the collected data from the crowd is one of the important steps to infer the correct answer but the existing study seems to be limited to the single-step task. This study looks at multiple-step classification tasks and understands aggregation in such cases, hence is useful for assessing the classification quality.

Design/methodology/approach: We present a model to capture the information of the workflow, questions, and answers for both single-question and multiple-question clas- sification tasks. We propose an adapted approach on top of the classic approach so that our model can handle tasks with several multiple-choice questions in general instead of a specific domain or any specific hierarchical classifications. We evaluate our approach with three representative tasks from existing citizen science projects in which we have the gold standard created by experts.

Findings: The results show our approach can provide significant improvements to the overall classification accuracy. Our analysis also demonstrates that all algorithms can achieve higher accuracy for the volunteer- versus paid-generated datasets for the same task. Furthermore, we observed interesting patterns in the relationship between the performance of different algorithms and workflow specific factors including the number of steps, and the number of available options in each step.

Originality/value: Due to the nature of crowdsourcing, aggregating the collected data is an important process to understand the quality of crowdsourcing results. Different inference algorithms have been studied for simple microtasks consisting of single questions with two or more answers. However, as classification tasks typically contain many questions, our proposed method is able to to be applied to a wide range of tasks including both single-question and multiple-question classification tasks.
crowdsourcing, human computation, quality assessment, classification, aggregation
Bu, Qiong
ce52e778-20d8-466e-afec-fec74620c959
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67
Chapman, Adriane
721b7321-8904-4be2-9b01-876c430743f1
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Bu, Qiong
ce52e778-20d8-466e-afec-fec74620c959
Simperl, Elena
40261ae4-c58c-48e4-b78b-5187b10e4f67
Chapman, Adriane
721b7321-8904-4be2-9b01-876c430743f1
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae

Bu, Qiong, Simperl, Elena, Chapman, Adriane and Maddalena, Eddy (2019) Quality assessment in crowdsourced classification tasks. International Journal of Crowd Science. (doi:10.1108/IJCS-06-2019-0017).

Record type: Article

Abstract

Purpose: Ensuring quality is one of the most significant challenges in microtask crowdsourcing. Aggregation of the collected data from the crowd is one of the important steps to infer the correct answer but the existing study seems to be limited to the single-step task. This study looks at multiple-step classification tasks and understands aggregation in such cases, hence is useful for assessing the classification quality.

Design/methodology/approach: We present a model to capture the information of the workflow, questions, and answers for both single-question and multiple-question clas- sification tasks. We propose an adapted approach on top of the classic approach so that our model can handle tasks with several multiple-choice questions in general instead of a specific domain or any specific hierarchical classifications. We evaluate our approach with three representative tasks from existing citizen science projects in which we have the gold standard created by experts.

Findings: The results show our approach can provide significant improvements to the overall classification accuracy. Our analysis also demonstrates that all algorithms can achieve higher accuracy for the volunteer- versus paid-generated datasets for the same task. Furthermore, we observed interesting patterns in the relationship between the performance of different algorithms and workflow specific factors including the number of steps, and the number of available options in each step.

Originality/value: Due to the nature of crowdsourcing, aggregating the collected data is an important process to understand the quality of crowdsourcing results. Different inference algorithms have been studied for simple microtasks consisting of single questions with two or more answers. However, as classification tasks typically contain many questions, our proposed method is able to to be applied to a wide range of tasks including both single-question and multiple-question classification tasks.

Text
qb1g13_ijcs_accepted - Accepted Manuscript
Download (2MB)
Text
10-1108_IJCS-06-2019-0017 - Version of Record
Available under License Creative Commons Attribution.
Download (2MB)

More information

Accepted/In Press date: 20 August 2019
e-pub ahead of print date: 16 October 2019
Keywords: crowdsourcing, human computation, quality assessment, classification, aggregation

Identifiers

Local EPrints ID: 434006
URI: https://eprints.soton.ac.uk/id/eprint/434006
PURE UUID: f3cc6470-e616-4523-be6c-18755843b7a3
ORCID for Elena Simperl: ORCID iD orcid.org/0000-0003-1722-947X
ORCID for Adriane Chapman: ORCID iD orcid.org/0000-0002-3814-2587

Catalogue record

Date deposited: 10 Sep 2019 16:30
Last modified: 09 Nov 2019 01:32

Export record

Altmetrics

Contributors

Author: Qiong Bu
Author: Elena Simperl ORCID iD
Author: Adriane Chapman ORCID iD
Author: Eddy Maddalena

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×