The University of Southampton
University of Southampton Institutional Repository

All those wasted hours: on task abandonment in crowdsourcing

All those wasted hours: on task abandonment in crowdsourcing
All those wasted hours: on task abandonment in crowdsourcing

Crowdsourcing has become a standard methodology to collect manually annotated data such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or FigureEight, crowd workers select tasks to work on based on different dimensions such as task reward and requester reputation. Requesters then receive the judgments of workers who self-selected into the tasks and completed them successfully. Several crowd workers, however, preview tasks, begin working on them, reaching varying stages of task completion without finally submitting their work. Such behavior results in unrewarded effort which remains invisible to requesters. In this paper, we conduct the first investigation into the phenomenon of task abandonment, the act of workers previewing or beginning a task and deciding not to complete it. We follow a threefold methodology which includes 1) investigating the prevalence and causes of task abandonment by means of a survey over different crowdsourcing platforms, 2) data-driven analyses of logs collected during a large-scale relevance judgment experiment, and 3) controlled experiments measuring the effect of different dimensions on abandonment. Our results show that task abandonment is a widely spread phenomenon. Apart from accounting for a considerable amount of wasted human effort, this bears important implications on the hourly wages of workers as they are not rewarded for tasks that they do not complete. We also show how task abandonment may have strong implications on the use of collected data (for example, on the evaluation of IR systems).

321-329
ACM
Han, Lei
f313fbe7-ba8d-4b92-a074-6789058c0554
Roitero, Kevin
71dbbb60-a1e9-431e-930d-fbd498c6559f
Gadiraju, Ujwal
91e34693-d3ab-4469-a8f5-4c42bda42805
Sarasua, Cristina
8eda1bc4-faa8-46f2-9fab-782bf0f39994
Checco, Alessandro
92073d96-c52f-473b-944f-e468faa443c3
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Demartini, Gianluca
2da91fe3-eac2-42d8-8450-b7d74b1d0209
Han, Lei
f313fbe7-ba8d-4b92-a074-6789058c0554
Roitero, Kevin
71dbbb60-a1e9-431e-930d-fbd498c6559f
Gadiraju, Ujwal
91e34693-d3ab-4469-a8f5-4c42bda42805
Sarasua, Cristina
8eda1bc4-faa8-46f2-9fab-782bf0f39994
Checco, Alessandro
92073d96-c52f-473b-944f-e468faa443c3
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Demartini, Gianluca
2da91fe3-eac2-42d8-8450-b7d74b1d0209

Han, Lei, Roitero, Kevin, Gadiraju, Ujwal, Sarasua, Cristina, Checco, Alessandro, Maddalena, Eddy and Demartini, Gianluca (2019) All those wasted hours: on task abandonment in crowdsourcing. In WSDM 2019 - Proceedings of the 12th ACM International Conference on Web Search and Data Mining. ACM. pp. 321-329 . (doi:10.1145/3289600.3291035).

Record type: Conference or Workshop Item (Paper)

Abstract

Crowdsourcing has become a standard methodology to collect manually annotated data such as relevance judgments at scale. On crowdsourcing platforms like Amazon MTurk or FigureEight, crowd workers select tasks to work on based on different dimensions such as task reward and requester reputation. Requesters then receive the judgments of workers who self-selected into the tasks and completed them successfully. Several crowd workers, however, preview tasks, begin working on them, reaching varying stages of task completion without finally submitting their work. Such behavior results in unrewarded effort which remains invisible to requesters. In this paper, we conduct the first investigation into the phenomenon of task abandonment, the act of workers previewing or beginning a task and deciding not to complete it. We follow a threefold methodology which includes 1) investigating the prevalence and causes of task abandonment by means of a survey over different crowdsourcing platforms, 2) data-driven analyses of logs collected during a large-scale relevance judgment experiment, and 3) controlled experiments measuring the effect of different dimensions on abandonment. Our results show that task abandonment is a widely spread phenomenon. Apart from accounting for a considerable amount of wasted human effort, this bears important implications on the hourly wages of workers as they are not rewarded for tasks that they do not complete. We also show how task abandonment may have strong implications on the use of collected data (for example, on the evaluation of IR systems).

Full text not available from this repository.

More information

Published date: 30 January 2019
Venue - Dates: 12th ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, Australia, 2019-02-11 - 2019-02-15

Identifiers

Local EPrints ID: 428600
URI: https://eprints.soton.ac.uk/id/eprint/428600
PURE UUID: 80695821-c375-45a9-b508-9d40747edb3a

Catalogue record

Date deposited: 04 Mar 2019 17:30
Last modified: 04 Mar 2019 17:30

Export record

Altmetrics

Contributors

Author: Lei Han
Author: Kevin Roitero
Author: Ujwal Gadiraju
Author: Cristina Sarasua
Author: Alessandro Checco
Author: Eddy Maddalena
Author: Gianluca Demartini

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×