The importance of error measures for machine learning regression to approximate the ground truth

Abstract

As machine learning technology improves, it is increasingly relied upon when making significant decisions which require a high level of trust. Accuracy and interpretability is paramount for trust in regression methods, which comprise a large portion of the field. To apply these methods with confidence there needs to be a certainty that they have modelled the ground truth of a dataset— the correct input-output relationships. Conventional regression error measures, however, do not ensure that the correct relationships are modelled, as they only require accurate point predictions to assign low error to a method. A case study of power prediction for merchant vessels is used to illustrate the problem, where accurate prediction and correct input-output relationship modelling is required, although there is limited understanding of these input-output relationships. For this problem neural networks can produce predictions with a 2% Mean Absolute Relative Error, which is low enough for use in fuel saving devices on-board vessels in operation. The methods developed in this thesis have been deployed on over a dozen merchant vessels operated by Shell Shipping and Maritime, saving over 1/4 million tonnes of CO₂ emissions in 2020. However, the predictions are not interpretable, as the input-output relationships modelled are not consistent or correct. A new error measure, the Mean Fit to Median Error, is investigated which ensures networks approximate the conditional averages and is applicable to any dataset. This is verified on 36 artificial datasets, where the ground truth is known, and is shown to correlate to the ground truth on average 60% higher than traditional error measures correlate to the ground truth. The Mean Fit to Median Error is then applied to the ship powering example and shows a shift in the approximated relationships for the same Mean Absolute Relative Error values, showing an improvement in determining the ground truth. Networks reporting low Mean Fit to Median errors model more consistent and correct input-output relationships and are robust to areas of sparse data.

More information

Identifiers

ORCID for Dominic Hudson: orcid.org/0000-0002-2012-6255

Catalogue record

Export record