Aggregating crowdsourced quantitative claims: Additive and multiplicative models

Ouyang, Robin Wentao, Kaplan, Lance M., Toniolo, Alice, Srivastava, Mani and Norman, Timothy J. (2016) Aggregating crowdsourced quantitative claims: Additive and multiplicative models. IEEE Transactions on Knowledge and Data Engineering, 28 (7), 1621-1634, [7422085]. (doi:10.1109/TKDE.2016.2535383).

Abstract

Truth discovery is an important technique for enabling reliable crowdsourcing applications. It aims to automatically discover the truths from possibly conflicting crowdsourced claims. Most existing truth discovery approaches focus on categorical applications, such as image classification. They use the accuracy, i.e., rate of exactly correct claims, to capture the reliability of participants. As a consequence, they are not effective for truth discovery in quantitative applications, such as percentage annotation and object counting, where similarity rather than exact matching between crowdsourced claims and latent truths should be considered. In this paper, we propose two unsupervised Quantitative Truth Finders (QTFs) for truth discovery in quantitative crowdsourcing applications. One QTF explores an additive model and the other explores a multiplicative model to capture different relationships between crowdsourced claims and latent truths in different classes of quantitative tasks. These QTFs naturally incorporate the similarity between variables. Moreover, they use the bias and the confidence instead of the accuracy to capture participants' abilities in quantity estimation. These QTFs are thus capable of accurately discovering quantitative truths in particular domains. Through extensive experiments, we demonstrate that these QTFs outperform other state-of-The-Art approaches for truth discovery in quantitative crowdsourcing applications and they are also quite efficient.

This record has no associated files available for download.

Contributors

Author: Timothy J. Norman

Download statistics