Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks

Tran-Thanh, Long, Venanzi, Matteo, Rogers, Alex and Jennings, Nicholas R. (2013) Efficient Budget Allocation with Accuracy Guarantees for Crowdsourcing Classification Tasks. AAMAS '13 Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems. pp. 901-908 .

Record type: Conference or Workshop Item (Paper)

Abstract

In this paper we address the problem of budget allocation for redundantly crowdsourcing a set of classification tasks where a key challenge is to find a trade–off between the total cost and the accuracy of estimation. We propose CrowdBudget, an agent–based budget allocation algorithm, that efficiently divides a given budget among different tasks in order to achieve low estimation error. In particular, we prove that CrowdBudget can achieve at most max { 0, K/2 - O (√B) } estimation error with high probability, where K is the number of tasks and B is the budget size. This result significantly outperforms the current best theoretical guarantee from Karger et al. In addition, we demonstrate that our algorithm outperforms existing methods by up to 40% in experiments based on real–world data from a prominent database of crowdsourced classification responses.

Text

crowdverify_final.pdf - Other

Download (364kB)