Probabilistic forecasting with discrete choice models: evaluating predictions with pseudo-coefficients of determination
Probabilistic forecasting with discrete choice models: evaluating predictions with pseudo-coefficients of determination
Probabilistic forecasts from discrete choice models, which are widely used in marketing science and competitive event forecasting, are often best evaluated out-of-sample using pseudo-coefficients of determination, or pseudo-R2s. How- ever, there is a danger of misjudging the accuracy of forecast probabilities of event outcomes based on observed frequencies, because of issues related to pseudo-R2s. First, we show that McFadden’s pseudo-R2 varies predictably with the number of alternatives in the choice set. Then we evaluate the relative merits of two methods (bootstrap and asymptotic) for estimating the variance of pseudo-R2s so that their values can be appropriately compared across non-nested models. Finally, in the context of competitive event forecasting, where the accuracy of forecasts has direct economic consequence, we derive new R2 measures that can be used to assess the economic value of forecasts. Throughout, we illustrate using data drawn from UK and Ireland horse race betting markets.
forecasting, decision analysis, finance, discrete choice models, horseracing
1021-1030
Sung, Ming-Chien
2114f823-bc7f-4306-a775-67aee413aa03
McDonald, David
fbe07af1-0891-4db7-858c-793633d98f3f
Johnson, Johnnie
6d9f1a51-38a8-4011-a792-bfc82040fac4
1 February 2016
Sung, Ming-Chien
2114f823-bc7f-4306-a775-67aee413aa03
McDonald, David
fbe07af1-0891-4db7-858c-793633d98f3f
Johnson, Johnnie
6d9f1a51-38a8-4011-a792-bfc82040fac4
Sung, Ming-Chien, McDonald, David and Johnson, Johnnie
(2016)
Probabilistic forecasting with discrete choice models: evaluating predictions with pseudo-coefficients of determination.
European Journal of Operational Research, 248 (3), .
(doi:10.1016/j.ejor.2015.08.068).
Abstract
Probabilistic forecasts from discrete choice models, which are widely used in marketing science and competitive event forecasting, are often best evaluated out-of-sample using pseudo-coefficients of determination, or pseudo-R2s. How- ever, there is a danger of misjudging the accuracy of forecast probabilities of event outcomes based on observed frequencies, because of issues related to pseudo-R2s. First, we show that McFadden’s pseudo-R2 varies predictably with the number of alternatives in the choice set. Then we evaluate the relative merits of two methods (bootstrap and asymptotic) for estimating the variance of pseudo-R2s so that their values can be appropriately compared across non-nested models. Finally, in the context of competitive event forecasting, where the accuracy of forecasts has direct economic consequence, we derive new R2 measures that can be used to assess the economic value of forecasts. Throughout, we illustrate using data drawn from UK and Ireland horse race betting markets.
Text
PseudoR2_Final20150827.pdf
- Accepted Manuscript
More information
Accepted/In Press date: 28 August 2015
e-pub ahead of print date: 10 September 2015
Published date: 1 February 2016
Keywords:
forecasting, decision analysis, finance, discrete choice models, horseracing
Organisations:
Centre of Excellence in Decision, Analytics & Risk Research
Identifiers
Local EPrints ID: 381025
URI: http://eprints.soton.ac.uk/id/eprint/381025
ISSN: 0377-2217
PURE UUID: 7fbe2f4e-4863-4906-98a8-ed04df63faec
Catalogue record
Date deposited: 03 Sep 2015 09:19
Last modified: 15 Mar 2024 05:20
Export record
Altmetrics
Contributors
Author:
David McDonald
Author:
Johnnie Johnson
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
