The University of Southampton
University of Southampton Institutional Repository

Efficient crowdsourcing of unknown experts using multi-armed bandits

Efficient crowdsourcing of unknown experts using multi-armed bandits
Efficient crowdsourcing of unknown experts using multi-armed bandits
We address the expert crowdsourcing problem, in which an employer wishes to assign tasks to a set of available workers with heterogeneous working costs. Critically, as workers produce results of varying quality, the utility of each assigned task is unknown and can vary both between workers and individual tasks. Furthermore, in realistic settings, workers are likely to have limits on the number of tasks they can perform and the employer will have a fixed budget to spend on hiring workers. Given these constraints, the objective of the employer is to assign tasks to workers in order to maximise the overall utility achieved. To achieve this, we introduce a novel multi–armed bandit (MAB) model, the bounded MAB, that naturally captures the problem of expert crowdsourcing. We also propose an algorithm to solve it efficiently, called bounded ?–first, which uses the first ?B of its total budget B to derive estimates of the workers’ quality characteristics (exploration), while the remaining (1 ? ?) B is used to maximise the total utility based on those estimates (exploitation). We show that using this technique allows us to derive an O(B2/3) upper bound on our algorithm’s performance regret (i.e. the expected difference in utility between the optimal and our algorithm). In addition, we demonstrate that our algorithm outperforms existing crowdsourcing methods by up to 155% in experiments based on real–world data from a prominent crowdsourcing site, while achieving up to 75% of a hypothetical optimal with full information.
768-773
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Stein, Sebastian
cb2325e7-5e63-475e-8a69-9db2dfbdb00b
Rogers, Alex
f9130bc6-da32-474e-9fab-6c6cb8077fdc
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Stein, Sebastian
cb2325e7-5e63-475e-8a69-9db2dfbdb00b
Rogers, Alex
f9130bc6-da32-474e-9fab-6c6cb8077fdc
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30

Tran-Thanh, Long, Stein, Sebastian, Rogers, Alex and Jennings, Nicholas R. (2012) Efficient crowdsourcing of unknown experts using multi-armed bandits. 20th European Conference on Artificial Intelligence (ECAI 2012), Montpellier, France. 27 - 31 Aug 2012. pp. 768-773 . (doi:10.3233/978-1-61499-098-7-768).

Record type: Conference or Workshop Item (Paper)

Abstract

We address the expert crowdsourcing problem, in which an employer wishes to assign tasks to a set of available workers with heterogeneous working costs. Critically, as workers produce results of varying quality, the utility of each assigned task is unknown and can vary both between workers and individual tasks. Furthermore, in realistic settings, workers are likely to have limits on the number of tasks they can perform and the employer will have a fixed budget to spend on hiring workers. Given these constraints, the objective of the employer is to assign tasks to workers in order to maximise the overall utility achieved. To achieve this, we introduce a novel multi–armed bandit (MAB) model, the bounded MAB, that naturally captures the problem of expert crowdsourcing. We also propose an algorithm to solve it efficiently, called bounded ?–first, which uses the first ?B of its total budget B to derive estimates of the workers’ quality characteristics (exploration), while the remaining (1 ? ?) B is used to maximise the total utility based on those estimates (exploitation). We show that using this technique allows us to derive an O(B2/3) upper bound on our algorithm’s performance regret (i.e. the expected difference in utility between the optimal and our algorithm). In addition, we demonstrate that our algorithm outperforms existing crowdsourcing methods by up to 155% in experiments based on real–world data from a prominent crowdsourcing site, while achieving up to 75% of a hypothetical optimal with full information.

Text
mab_crowdsourcing_ecai2012_final.pdf - Author's Original
Download (124kB)

More information

e-pub ahead of print date: 21 May 2012
Published date: 27 August 2012
Venue - Dates: 20th European Conference on Artificial Intelligence (ECAI 2012), Montpellier, France, 2012-08-27 - 2012-08-31
Organisations: Agents, Interactions & Complexity

Identifiers

Local EPrints ID: 339244
URI: http://eprints.soton.ac.uk/id/eprint/339244
PURE UUID: 7ff396f4-d6ac-4c0f-9a0a-7c2b90a76694
ORCID for Long Tran-Thanh: ORCID iD orcid.org/0000-0003-1617-8316
ORCID for Sebastian Stein: ORCID iD orcid.org/0000-0003-2858-8857

Catalogue record

Date deposited: 28 May 2012 15:02
Last modified: 15 Mar 2024 03:30

Export record

Altmetrics

Contributors

Author: Long Tran-Thanh ORCID iD
Author: Sebastian Stein ORCID iD
Author: Alex Rogers
Author: Nicholas R. Jennings

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×