The University of Southampton
University of Southampton Institutional Repository

Crowd worker strategies in relevance judgment tasks

Crowd worker strategies in relevance judgment tasks
Crowd worker strategies in relevance judgment tasks
Crowdsourcing is a popular technique to collect large amounts of human-generated labels, such as relevance judgments used to create information retrieval (IR) evaluation collections. Previous research has shown how collecting high quality labels from a crowdsourcing platform can be challenging. Existing quality assurance techniques focus on answer aggregation or on the use of gold questions where ground-truth data allows to check for the quality of the responses.

In this paper, we present qualitative and quantitative results, revealing how different crowd workers adopt different work strategies to complete relevance judgment tasks efficiently and their consequent impact on quality. We delve into the techniques and tools that highly experienced crowd workers use to be more efficient in completing crowdsourcing micro-tasks. To this end, we use both qualitative results from worker interviews and surveys, as well as the results of a data-driven study of behavioral log data (i.e., clicks, keystrokes and keyboard shortcuts) collected from crowd workers performing relevance judgment tasks. Our results highlight the presence of frequently used shortcut patterns that can speed-up task completion, thus increasing the hourly wage of efficient workers. We observe how crowd work experiences result in different types of working strategies, productivity levels, quality and diversity of the crowdsourced judgments.
Crowdsourcing, IR evaluation, Relevance judgment, User behavior
241–249
Association for Computing Machinery
Han, Lei
f313fbe7-ba8d-4b92-a074-6789058c0554
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Checco, Alessandro
92073d96-c52f-473b-944f-e468faa443c3
Sarasua, Cristina
8eda1bc4-faa8-46f2-9fab-782bf0f39994
Gadiraju, Ujwal
91e34693-d3ab-4469-a8f5-4c42bda42805
Roitero, Kevin
71dbbb60-a1e9-431e-930d-fbd498c6559f
Demartini, Gianluca
2da91fe3-eac2-42d8-8450-b7d74b1d0209
Han, Lei
f313fbe7-ba8d-4b92-a074-6789058c0554
Maddalena, Eddy
397dbaba-4363-4c11-8e52-4a7ba4df4bae
Checco, Alessandro
92073d96-c52f-473b-944f-e468faa443c3
Sarasua, Cristina
8eda1bc4-faa8-46f2-9fab-782bf0f39994
Gadiraju, Ujwal
91e34693-d3ab-4469-a8f5-4c42bda42805
Roitero, Kevin
71dbbb60-a1e9-431e-930d-fbd498c6559f
Demartini, Gianluca
2da91fe3-eac2-42d8-8450-b7d74b1d0209

Han, Lei, Maddalena, Eddy, Checco, Alessandro, Sarasua, Cristina, Gadiraju, Ujwal, Roitero, Kevin and Demartini, Gianluca (2020) Crowd worker strategies in relevance judgment tasks. In WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining. Association for Computing Machinery. 241–249 . (doi:10.1145/3336191.3371857).

Record type: Conference or Workshop Item (Paper)

Abstract

Crowdsourcing is a popular technique to collect large amounts of human-generated labels, such as relevance judgments used to create information retrieval (IR) evaluation collections. Previous research has shown how collecting high quality labels from a crowdsourcing platform can be challenging. Existing quality assurance techniques focus on answer aggregation or on the use of gold questions where ground-truth data allows to check for the quality of the responses.

In this paper, we present qualitative and quantitative results, revealing how different crowd workers adopt different work strategies to complete relevance judgment tasks efficiently and their consequent impact on quality. We delve into the techniques and tools that highly experienced crowd workers use to be more efficient in completing crowdsourcing micro-tasks. To this end, we use both qualitative results from worker interviews and surveys, as well as the results of a data-driven study of behavioral log data (i.e., clicks, keystrokes and keyboard shortcuts) collected from crowd workers performing relevance judgment tasks. Our results highlight the presence of frequently used shortcut patterns that can speed-up task completion, thus increasing the hourly wage of efficient workers. We observe how crowd work experiences result in different types of working strategies, productivity levels, quality and diversity of the crowdsourced judgments.

This record has no associated files available for download.

More information

Accepted/In Press date: 10 October 2019
Published date: 20 January 2020
Additional Information: Funding Information: Acknowledgements. This work is supported by ARC Discovery Project (DP190102141) and the Erasmus+ project DISKOW (60171990). Publisher Copyright: © 2020 Association for Computing Machinery.
Venue - Dates: 13th International Conference on Web Search and Data Mining (WSDM2020), Hyatt Regency Houston/Galleria, Houston, United States, 2020-02-03 - 2020-03-07
Keywords: Crowdsourcing, IR evaluation, Relevance judgment, User behavior

Identifiers

Local EPrints ID: 438319
URI: http://eprints.soton.ac.uk/id/eprint/438319
PURE UUID: 48afca40-1fa2-47ba-9d9f-76a0c7b20aa5

Catalogue record

Date deposited: 05 Mar 2020 17:30
Last modified: 16 Mar 2024 06:59

Export record

Altmetrics

Contributors

Author: Lei Han
Author: Eddy Maddalena
Author: Alessandro Checco
Author: Cristina Sarasua
Author: Ujwal Gadiraju
Author: Kevin Roitero
Author: Gianluca Demartini

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×