The University of Southampton
University of Southampton Institutional Repository

# Testing whether a learning procedure is calibrated

Cockayne, Jonathan, Graham, Matthew M., Oates, Chris J., Sullivan, T.J. and Teymur, Onur (2020) Testing whether a learning procedure is calibrated. arXiv.

Record type: Article

## Abstract

A learning procedure takes as input a dataset and performs inference for the parameters $\theta$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $\theta$ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

Text
Testing whether a Learning Procedure is Calibrated - Accepted Manuscript
Restricted to Repository staff only
Request a copy

Accepted/In Press date: 23 September 2020
e-pub ahead of print date: 23 December 2020

## Identifiers

Local EPrints ID: 451490
URI: http://eprints.soton.ac.uk/id/eprint/451490
ISSN: 2331-8422
PURE UUID: 9d18c760-73bd-4da7-8a85-595a4ed5bca1
ORCID for Jonathan Cockayne: orcid.org/0000-0002-3287-199X

## Catalogue record

Date deposited: 01 Oct 2021 16:38

## Contributors

Author: Matthew M. Graham
Author: Chris J. Oates
Author: T.J. Sullivan
Author: Onur Teymur