The University of Southampton
University of Southampton Institutional Repository

A robust profit measure for binary classification model evaluation

A robust profit measure for binary classification model evaluation
A robust profit measure for binary classification model evaluation
Using profit-based evaluation measures is a necessity in business-oriented contexts, as they aid companies in making cost-optimal decisions. Among the measures that effectively include the true nature of costs and benefits in binary classification, the expected maximum profit (EMP) has been used successfully for churn prediction and credit scoring, and defined in general for binary classification problems. However, despite its competitive results against the most frequently used measures, the EMP relies on a fixed probability distribution of costs and benefits, the range of which in real applications is not entirely known. In this paper, we propose to extend this measure by adding random shocks to these distributions. We call this new measure the R-EMP, following the convention of the analogous EMP measure. Our metric adds a stochastic component to each point of the cost-benefit distributions, assuming that costs and benefits have a fixed probability, but its distribution range is subject to an external shock, which can be different for each cost or benefit. The experimental set-up is focused on a credit scoring application using a dataset of a Chilean financial institution, with the attribute selection for a logistic regression being accomplished using the AUC, EMP, H-measure, and R-EMP as the selection criteria. The results indicate that the R-EMP measure is the most robust metric for achieving the greatest profit for the company under uncertain external conditions.
Supervised Binary Classification, Business analytics, Performance measures, Profit-driven analytics
0957-4174
154-160
Garrido, Franco
4f863fe0-af98-4550-96ca-405d3c5827e9
Verbeke, Wouter
6805dfd7-ecc9-4746-8e64-dd7eee6e3e3e
Bravo, Cristian
b22c4145-644e-40ee-85d8-431c59c3c71b
Garrido, Franco
4f863fe0-af98-4550-96ca-405d3c5827e9
Verbeke, Wouter
6805dfd7-ecc9-4746-8e64-dd7eee6e3e3e
Bravo, Cristian
b22c4145-644e-40ee-85d8-431c59c3c71b

Garrido, Franco, Verbeke, Wouter and Bravo, Cristian (2018) A robust profit measure for binary classification model evaluation Expert Systems with Applications, 92, pp. 154-160. (doi:10.1016/j.eswa.2017.09.045).

Record type: Article

Abstract

Using profit-based evaluation measures is a necessity in business-oriented contexts, as they aid companies in making cost-optimal decisions. Among the measures that effectively include the true nature of costs and benefits in binary classification, the expected maximum profit (EMP) has been used successfully for churn prediction and credit scoring, and defined in general for binary classification problems. However, despite its competitive results against the most frequently used measures, the EMP relies on a fixed probability distribution of costs and benefits, the range of which in real applications is not entirely known. In this paper, we propose to extend this measure by adding random shocks to these distributions. We call this new measure the R-EMP, following the convention of the analogous EMP measure. Our metric adds a stochastic component to each point of the cost-benefit distributions, assuming that costs and benefits have a fixed probability, but its distribution range is subject to an external shock, which can be different for each cost or benefit. The experimental set-up is focused on a credit scoring application using a dataset of a Chilean financial institution, with the attribute selection for a logistic regression being accomplished using the AUC, EMP, H-measure, and R-EMP as the selection criteria. The results indicate that the R-EMP measure is the most robust metric for achieving the greatest profit for the company under uncertain external conditions.

Text REMP_Self_Archive - Accepted Manuscript
Restricted to Repository staff only until 19 September 2018.

More information

Accepted/In Press date: 17 September 2017
e-pub ahead of print date: 19 September 2017
Published date: February 2018
Keywords: Supervised Binary Classification, Business analytics, Performance measures, Profit-driven analytics

Identifiers

Local EPrints ID: 414261
URI: http://eprints.soton.ac.uk/id/eprint/414261
ISSN: 0957-4174
PURE UUID: 3065eba8-e14a-4992-a9a3-87e527d1d09f
ORCID for Cristian Bravo: ORCID iD orcid.org/0000-0003-1579-1565

Catalogue record

Date deposited: 21 Sep 2017 16:31
Last modified: 23 Oct 2017 16:31

Export record

Altmetrics

Contributors

Author: Franco Garrido
Author: Wouter Verbeke
Author: Cristian Bravo ORCID iD

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×