Testing whether a learning procedure is calibrated
Testing whether a learning procedure is calibrated
A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or
under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.
1-36
Cockayne, Jon
da87c8b2-fafb-4856-938d-50be8f0e4a5b
Graham, Matthew M.
2f815357-15ee-470e-a2a6-071844d35e75
Oates, Chris J.
faae6d14-7a66-4ca3-a6ba-daaf1938e164
Sullivan, T.J.
1ef5be06-ad9c-44df-afdd-7b2294eb1e6b
Teymur, Onur
fe9cd697-4c7b-4346-896f-2a9eeee8996f
1 May 2022
Cockayne, Jon
da87c8b2-fafb-4856-938d-50be8f0e4a5b
Graham, Matthew M.
2f815357-15ee-470e-a2a6-071844d35e75
Oates, Chris J.
faae6d14-7a66-4ca3-a6ba-daaf1938e164
Sullivan, T.J.
1ef5be06-ad9c-44df-afdd-7b2294eb1e6b
Teymur, Onur
fe9cd697-4c7b-4346-896f-2a9eeee8996f
Cockayne, Jon, Graham, Matthew M., Oates, Chris J., Sullivan, T.J. and Teymur, Onur
(2022)
Testing whether a learning procedure is calibrated.
Journal of Machine Learning Research, 23, .
Abstract
A learning procedure takes as input a dataset and performs inference for the parameters θ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about θ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or
under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.
Text
Testing whether a Learning Procedure is Calibrated
- Author's Original
Available under License Other.
Text
1-s2.0-S2211379723003194-main
- Version of Record
More information
Accepted/In Press date: 1 March 2022
Published date: 1 May 2022
Identifiers
Local EPrints ID: 451490
URI: http://eprints.soton.ac.uk/id/eprint/451490
PURE UUID: 9d18c760-73bd-4da7-8a85-595a4ed5bca1
Catalogue record
Date deposited: 01 Oct 2021 16:38
Last modified: 17 Mar 2024 04:08
Export record
Contributors
Author:
Matthew M. Graham
Author:
Chris J. Oates
Author:
T.J. Sullivan
Author:
Onur Teymur
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics