The University of Southampton
University of Southampton Institutional Repository

Crowdsourcing complex workflows under budget constraints

Crowdsourcing complex workflows under budget constraints
Crowdsourcing complex workflows under budget constraints
We consider the problem of task allocation in crowdsourcing systems with multiple complex workflows, each of which consists of a set of interdependent micro-tasks. We propose Budgeteer, an algorithm to solve this problem under a budget constraint. In particular, our algorithm first calculates an efficient way to allocate budget to each workflow. It then determines the number of inter-dependent micro-tasks and the price to pay for each task within each workflow, given the corresponding budget constraints. We empirically evaluate it on a well-known crowdsourcing-based text correction workflow using Amazon Mechanical Turk, and show that Budgeteer can achieve similar levels of accuracy to current benchmarks, but is on average 45% cheaper.
crowdsourcing, find-fix-verify, budgeted task allocation, performance guarantees
1298-1304
ACM
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, Avi
4a81409b-7d71-4a4a-bc86-9869a0305b01
Ramchurn, Sarvapali D.
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Huynh, Trung Dong
ddea6cf3-5a82-4c99-8883-7c31cf22dd36
Rosenfeld, Avi
4a81409b-7d71-4a4a-bc86-9869a0305b01
Ramchurn, Sarvapali D.
1d62ae2a-a498-444e-912d-a6082d3aaea3
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30

Tran-Thanh, Long, Huynh, Trung Dong, Rosenfeld, Avi, Ramchurn, Sarvapali D. and Jennings, Nicholas R. (2015) Crowdsourcing complex workflows under budget constraints. In AAAI'15 Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. ACM. pp. 1298-1304 .

Record type: Conference or Workshop Item (Paper)

Abstract

We consider the problem of task allocation in crowdsourcing systems with multiple complex workflows, each of which consists of a set of interdependent micro-tasks. We propose Budgeteer, an algorithm to solve this problem under a budget constraint. In particular, our algorithm first calculates an efficient way to allocate budget to each workflow. It then determines the number of inter-dependent micro-tasks and the price to pay for each task within each workflow, given the corresponding budget constraints. We empirically evaluate it on a well-known crowdsourcing-based text correction workflow using Amazon Mechanical Turk, and show that Budgeteer can achieve similar levels of accuracy to current benchmarks, but is on average 45% cheaper.

Text
aaai15-budgetfix.pdf - Accepted Manuscript
Download (1MB)

More information

Published date: January 2015
Venue - Dates: Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI-15), Austin, United States, 2015-01-25 - 2015-01-30
Keywords: crowdsourcing, find-fix-verify, budgeted task allocation, performance guarantees
Organisations: Web & Internet Science, Agents, Interactions & Complexity

Identifiers

Local EPrints ID: 372107
URI: https://eprints.soton.ac.uk/id/eprint/372107
PURE UUID: 2acbf2ac-700c-4225-81df-608a2939af21
ORCID for Long Tran-Thanh: ORCID iD orcid.org/0000-0003-1617-8316
ORCID for Trung Dong Huynh: ORCID iD orcid.org/0000-0003-4937-2473
ORCID for Sarvapali D. Ramchurn: ORCID iD orcid.org/0000-0001-9686-4302

Catalogue record

Date deposited: 25 Nov 2014 12:05
Last modified: 30 Nov 2019 01:37

Export record

Contributors

Author: Long Tran-Thanh ORCID iD
Author: Trung Dong Huynh ORCID iD
Author: Avi Rosenfeld
Author: Sarvapali D. Ramchurn ORCID iD
Author: Nicholas R. Jennings

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×