The University of Southampton
University of Southampton Institutional Repository

BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees

BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees
BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees
Crowdsourcing is a multi-agent task allocation paradigm that involves up to millions of workers, of varying reliability and availability, performing large numbers of micro-tasks. A key challenge is to crowdsource, at minimal cost and with predictable accuracy, complex tasks that involve different types of interdependent microtasks structured into complex workflows. In this paper, we propose the first crowdsourcing algorithm that solves this problem. Our algorithm, called BudgetFix, determines the number of interdependent micro-tasks and the price to pay for each task given budget constraints. Moreover, BudgetFix provides quality guarantees on the accuracy of the output of each phase of a given workflow. BudgetFix is empirically evaluated on a well-known crowdsourcingbased text correction workflow using Amazon Mechanical Turk, and is shown that BudgetFix can provide similar accuracy, compared to the state-of-the-art algorithm for this workflow, but is on average 32% cheaper.
978-1-4503-1993-5
477-484
ACM
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, A
d2209641-6339-434c-b578-74bf364d098b
Ramchurn, Sarvapali
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, A
d2209641-6339-434c-b578-74bf364d098b
Ramchurn, Sarvapali
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30

Tran-Thanh, Long, Huynh, Trung Dong, Rosenfeld, A, Ramchurn, Sarvapali and Jennings, Nicholas R. (2014) BudgetFix: budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems: AAMAS '14. ACM. pp. 477-484.

Record type: Conference or Workshop Item (Paper)

Abstract

Crowdsourcing is a multi-agent task allocation paradigm that involves up to millions of workers, of varying reliability and availability, performing large numbers of micro-tasks. A key challenge is to crowdsource, at minimal cost and with predictable accuracy, complex tasks that involve different types of interdependent microtasks structured into complex workflows. In this paper, we propose the first crowdsourcing algorithm that solves this problem. Our algorithm, called BudgetFix, determines the number of interdependent micro-tasks and the price to pay for each task given budget constraints. Moreover, BudgetFix provides quality guarantees on the accuracy of the output of each phase of a given workflow. BudgetFix is empirically evaluated on a well-known crowdsourcingbased text correction workflow using Amazon Mechanical Turk, and is shown that BudgetFix can provide similar accuracy, compared to the state-of-the-art algorithm for this workflow, but is on average 32% cheaper.

Text fp531_Tran-Thanh.pdf - Version of Record
Download (963kB)

More information

Accepted/In Press date: 20 December 2013
e-pub ahead of print date: 5 May 2014
Published date: 2014
Venue - Dates: 13th International Conference on Autonomous Agents and Multi-Agent Systems, France, 2014-05-05 - 2014-05-09
Organisations: Agents, Interactions & Complexity

Identifiers

Local EPrints ID: 362321
URI: https://eprints.soton.ac.uk/id/eprint/362321
ISBN: 978-1-4503-1993-5
PURE UUID: e5f91c2e-f166-4613-82b0-f9ca19da8914
ORCID for Long Tran-Thanh: ORCID iD orcid.org/0000-0003-1617-8316
ORCID for Trung Dong Huynh: ORCID iD orcid.org/0000-0003-4937-2473
ORCID for Sarvapali Ramchurn: ORCID iD orcid.org/0000-0001-9686-4302

Catalogue record

Date deposited: 20 Feb 2014 11:45
Last modified: 05 Jul 2018 00:33

Export record

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×