The University of Southampton
University of Southampton Institutional Repository

Mechanism design for the truthful elicitation of costly probabilistic estimates in distributed information systems

Record type: Article

This paper reports on the design of a novel two-stage mechanism, based on strictly proper scoring rules, that allows a centre to acquire a costly forecast of a future event (such as a meteorological phenomenon or a probabilistic estimate of a specific parameter such as the quality of an expected service), with a specified minimum precision, from one or more agents. In particular, this is the first mechanism that can be applied in a setting where the centre has no knowledge about the actual costs involved in the generation of the agents' estimates and also has no means of evaluating the quality and accuracy of the estimates it receives. En route to this mechanism, we first consider a setting in which any single agent can provide an estimate of the required precision, and the centre can evaluate this estimate by comparing it with the outcome which is observed at a later stage. This mechanism is then extended, so that it can be applied in a setting where the agents' different capabilities are reflected in the maximum precision of the estimates that they can provide, and hence the centre may need to select multiple agents and combine their individual results in order to obtain an estimate of the required precision. For all three mechanisms, we prove their economic properties (i.e. incentive compatibility and individual rationality) and then present specific empirical results. For the single agent mechanism we compare the quadratic, spherical and logarithmic scoring rules with a parametric family of scoring rules. We show that although the logarithmic scoring rule minimises both the mean and variance of the centre's total payments, using this rule means that an agent may face an unbounded penalty if it provides an estimate of extremely poor quality. We show that this is not the case for the parametric family, and thus, we suggest that the parametric scoring rule is the best candidate in our setting. Furthermore, we show that the 'multiple agent' extension describes a family of possible approaches to select agents in the first stage of our mechanism, and we show empirically and prove analytically that there is one approach that dominates all others. Finally, we compare our novel contribution and with the peer prediction mechanism introduced by Miller et al. (2007) [29] and show that the centre's total expected payment is the same in both mechanisms (and is equal to total expected payment in the case that the estimates can be compared to the actual outcome), while the variance in these payments is significantly reduced within our mechanism.

PDF 4120.pdf - Other
Download (270kB)


Papakonstantinou, Athanasios, Rogers, Alex, Gerding, Enrico H. and Jennings, Nicholas R. (2011) Mechanism design for the truthful elicitation of costly probabilistic estimates in distributed information systems Artificial Intelligence, 175, (2), pp. 648-672. (doi:10.1016/j.artint.2010.10.007).

More information

e-pub ahead of print date: 22 October 2010
Published date: February 2011
Keywords: multiagent systems, mechanism design, scoring rules, auction theory
Organisations: Agents, Interactions & Complexity


Local EPrints ID: 271316
PURE UUID: 60ee1fef-cc49-412d-a1f2-17c119c188d9

Catalogue record

Date deposited: 29 Jun 2010 10:51
Last modified: 18 Jul 2017 06:44

Export record



Author: Athanasios Papakonstantinou
Author: Alex Rogers
Author: Enrico H. Gerding
Author: Nicholas R. Jennings

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton:

ePrints Soton supports OAI 2.0 with a base URL of

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.