The University of Southampton
University of Southampton Institutional Repository

Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks

Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks
Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks
In this paper we address the problem of budget allocation for redundantly crowdsourcing a set of classification tasks where a key challenge is to find a trade–off between the total cost and the accuracy of estimation. We propose CrowdBudget, an agent–based budget allocation algorithm, that efficiently divides a given budget among different tasks in order to achieve low estimation error. In particular, we prove that CrowdBudget can achieve at most max { 0, K/2 - O (√B) } estimation error with high probability, where K is the number of tasks and B is the budget size. This result significantly outperforms the current best theoretical guarantee from Karger et al. In addition, we demonstrate that our algorithm outperforms existing methods by up to 40% in experiments based on real–world data from a prominent database of crowdsourced classification responses.
978-1-4503-1993-5
901-908
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Venanzi, Matteo
ba24a77f-31a6-4c05-a647-babf8f660440
Rogers, Alex
f9130bc6-da32-474e-9fab-6c6cb8077fdc
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30
Tran-Thanh, Long
e0666669-d34b-460e-950d-e8b139fab16c
Venanzi, Matteo
ba24a77f-31a6-4c05-a647-babf8f660440
Rogers, Alex
f9130bc6-da32-474e-9fab-6c6cb8077fdc
Jennings, Nicholas R.
ab3d94cc-247c-4545-9d1e-65873d6cdb30

Tran-Thanh, Long, Venanzi, Matteo, Rogers, Alex and Jennings, Nicholas R. (2013) Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks. AAMAS '13 Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems. pp. 901-908 .

Record type: Conference or Workshop Item (Paper)

Abstract

In this paper we address the problem of budget allocation for redundantly crowdsourcing a set of classification tasks where a key challenge is to find a trade–off between the total cost and the accuracy of estimation. We propose CrowdBudget, an agent–based budget allocation algorithm, that efficiently divides a given budget among different tasks in order to achieve low estimation error. In particular, we prove that CrowdBudget can achieve at most max { 0, K/2 - O (√B) } estimation error with high probability, where K is the number of tasks and B is the budget size. This result significantly outperforms the current best theoretical guarantee from Karger et al. In addition, we demonstrate that our algorithm outperforms existing methods by up to 40% in experiments based on real–world data from a prominent database of crowdsourced classification responses.

Text
crowdverify_final.pdf - Other
Download (364kB)

More information

Published date: May 2013
Venue - Dates: AAMAS '13 Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, 2013-05-01
Organisations: Agents, Interactions & Complexity

Identifiers

Local EPrints ID: 346675
URI: http://eprints.soton.ac.uk/id/eprint/346675
ISBN: 978-1-4503-1993-5
PURE UUID: 08645daf-fb1a-4c27-8598-5ddc4e1595b5
ORCID for Long Tran-Thanh: ORCID iD orcid.org/0000-0003-1617-8316

Catalogue record

Date deposited: 05 Jan 2013 19:50
Last modified: 14 Mar 2024 12:40

Export record

Contributors

Author: Long Tran-Thanh ORCID iD
Author: Matteo Venanzi
Author: Alex Rogers
Author: Nicholas R. Jennings

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×