BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees
BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees
Crowdsourcing is a multi-agent task allocation paradigm that involves up to millions of workers, of varying reliability and availability, performing large numbers of micro-tasks. A key challenge is to crowdsource, at minimal cost and with predictable accuracy, complex tasks that involve different types of interdependent microtasks structured into complex workflows. In this paper, we propose the first crowdsourcing algorithm that solves this problem. Our algorithm, called BudgetFix, determines the number of interdependent micro-tasks and the price to pay for each task given budget constraints. Moreover, BudgetFix provides quality guarantees on the accuracy of the output of each phase of a given workflow. BudgetFix is empirically evaluated on a well-known crowdsourcingbased text correction workflow using Amazon Mechanical Turk, and is shown that BudgetFix can provide similar accuracy, compared to the state-of-the-art algorithm for this workflow, but is on average 32% cheaper.
978-1-4503-1993-5
477-484
Association for Computing Machinery
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, A
d2209641-6339-434c-b578-74bf364d098b
Ramchurn, Sarvapali
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
2014
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, A
d2209641-6339-434c-b578-74bf364d098b
Ramchurn, Sarvapali
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
Tran-Thanh, Long, Huynh, Trung Dong, Rosenfeld, A, Ramchurn, Sarvapali and Jennings, Nicholas R.
(2014)
BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees.
In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems: AAMAS '14.
Association for Computing Machinery.
.
Record type:
Conference or Workshop Item
(Paper)
Abstract
Crowdsourcing is a multi-agent task allocation paradigm that involves up to millions of workers, of varying reliability and availability, performing large numbers of micro-tasks. A key challenge is to crowdsource, at minimal cost and with predictable accuracy, complex tasks that involve different types of interdependent microtasks structured into complex workflows. In this paper, we propose the first crowdsourcing algorithm that solves this problem. Our algorithm, called BudgetFix, determines the number of interdependent micro-tasks and the price to pay for each task given budget constraints. Moreover, BudgetFix provides quality guarantees on the accuracy of the output of each phase of a given workflow. BudgetFix is empirically evaluated on a well-known crowdsourcingbased text correction workflow using Amazon Mechanical Turk, and is shown that BudgetFix can provide similar accuracy, compared to the state-of-the-art algorithm for this workflow, but is on average 32% cheaper.
Text
fp531_Tran-Thanh.pdf
- Version of Record
More information
Accepted/In Press date: 20 December 2013
e-pub ahead of print date: 5 May 2014
Published date: 2014
Venue - Dates:
13th International Conference on Autonomous Agents and Multi-Agent Systems, , Paris, France, 2014-05-05 - 2014-05-09
Organisations:
Agents, Interactions & Complexity
Identifiers
Local EPrints ID: 362321
URI: http://eprints.soton.ac.uk/id/eprint/362321
ISBN: 978-1-4503-1993-5
PURE UUID: e5f91c2e-f166-4613-82b0-f9ca19da8914
Catalogue record
Date deposited: 20 Feb 2014 11:45
Last modified: 16 Mar 2024 03:44
Export record
Contributors
Author:
Long Tran-Thanh
Author:
Trung Dong Huynh
Author:
A Rosenfeld
Author:
Sarvapali Ramchurn
Author:
Nicholas R. Jennings
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics