The University of Southampton
University of Southampton Institutional Repository

Fieller Stability Measure: a novel model-dependent backtesting approach

Fieller Stability Measure: a novel model-dependent backtesting approach
Fieller Stability Measure: a novel model-dependent backtesting approach
Dataset shift is present in almost all real-world applications, since most of them are constantly dealing with changing environments. Detecting fractures in datasets on time allows recalibrating the models before a significant decrease in the model’s performance is observed. Since small changes are normal in most applications and do not justify the efforts that a model recalibration requires, we are only interested in identifying those changes that are critical for the correct functioning of the model. In this work we propose a model-dependent backtesting strategy designed to identify significant changes in the covariates, relating a confidence zone of the change to a maximal deviance measure obtained from the coefficients of the model. Using logistic regression as a predictive approach, we performed experiments on simulated data, and on a real-world credit scoring dataset. The results show that the proposed method has better performance than traditional approaches, consistently identifying major changes in variables while taking into account important characteristics of the problem, such as sample sizes and variances, and uncertainty in the coefficients
0160-5682
1895-1905
Bravo, Cristian
b22c4145-644e-40ee-85d8-431c59c3c71b
Maldonado, Sebastián
9e5fb121-d905-4337-beb3-bba6f7da9ae2
Bravo, Cristian
b22c4145-644e-40ee-85d8-431c59c3c71b
Maldonado, Sebastián
9e5fb121-d905-4337-beb3-bba6f7da9ae2

Bravo, Cristian and Maldonado, Sebastián (2015) Fieller Stability Measure: a novel model-dependent backtesting approach. Journal of the Operational Research Society, 66 (11), 1895-1905. (doi:10.1057/jors.2015.18).

Record type: Article

Abstract

Dataset shift is present in almost all real-world applications, since most of them are constantly dealing with changing environments. Detecting fractures in datasets on time allows recalibrating the models before a significant decrease in the model’s performance is observed. Since small changes are normal in most applications and do not justify the efforts that a model recalibration requires, we are only interested in identifying those changes that are critical for the correct functioning of the model. In this work we propose a model-dependent backtesting strategy designed to identify significant changes in the covariates, relating a confidence zone of the change to a maximal deviance measure obtained from the coefficients of the model. Using logistic regression as a predictive approach, we performed experiments on simulated data, and on a real-world credit scoring dataset. The results show that the proposed method has better performance than traditional approaches, consistently identifying major changes in variables while taking into account important characteristics of the problem, such as sample sizes and variances, and uncertainty in the coefficients

This record has no associated files available for download.

More information

Published date: 15 April 2015
Organisations: Southampton Business School

Identifiers

Local EPrints ID: 396648
URI: http://eprints.soton.ac.uk/id/eprint/396648
ISSN: 0160-5682
PURE UUID: 0e24f3c3-04f6-4d87-b3d7-7ff2a8debe96
ORCID for Cristian Bravo: ORCID iD orcid.org/0000-0003-1579-1565

Catalogue record

Date deposited: 09 Jun 2016 13:16
Last modified: 15 Mar 2024 03:33

Export record

Altmetrics

Contributors

Author: Cristian Bravo ORCID iD
Author: Sebastián Maldonado

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×