The University of Southampton
University of Southampton Institutional Repository

Testing whether a learning procedure is calibrated

Testing whether a learning procedure is calibrated
Testing whether a learning procedure is calibrated
A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.
2331-8422
Cockayne, Jonathan
da87c8b2-fafb-4856-938d-50be8f0e4a5b
Graham, Matthew M.
2f815357-15ee-470e-a2a6-071844d35e75
Oates, Chris J.
faae6d14-7a66-4ca3-a6ba-daaf1938e164
Sullivan, T.J.
1ef5be06-ad9c-44df-afdd-7b2294eb1e6b
Teymur, Onur
fe9cd697-4c7b-4346-896f-2a9eeee8996f
Cockayne, Jonathan
da87c8b2-fafb-4856-938d-50be8f0e4a5b
Graham, Matthew M.
2f815357-15ee-470e-a2a6-071844d35e75
Oates, Chris J.
faae6d14-7a66-4ca3-a6ba-daaf1938e164
Sullivan, T.J.
1ef5be06-ad9c-44df-afdd-7b2294eb1e6b
Teymur, Onur
fe9cd697-4c7b-4346-896f-2a9eeee8996f

Cockayne, Jonathan, Graham, Matthew M., Oates, Chris J., Sullivan, T.J. and Teymur, Onur (2020) Testing whether a learning procedure is calibrated. arXiv.

Record type: Article

Abstract

A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

Text
Testing whether a Learning Procedure is Calibrated - Accepted Manuscript
Restricted to Repository staff only
Request a copy

More information

Accepted/In Press date: 23 September 2020
e-pub ahead of print date: 23 December 2020

Identifiers

Local EPrints ID: 451490
URI: http://eprints.soton.ac.uk/id/eprint/451490
ISSN: 2331-8422
PURE UUID: 9d18c760-73bd-4da7-8a85-595a4ed5bca1
ORCID for Jonathan Cockayne: ORCID iD orcid.org/0000-0002-3287-199X

Catalogue record

Date deposited: 01 Oct 2021 16:38
Last modified: 13 Aug 2022 02:08

Export record

Contributors

Author: Matthew M. Graham
Author: Chris J. Oates
Author: T.J. Sullivan
Author: Onur Teymur

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×