Robust Automatic Methods for Outlier and Error Detection

Chambers, Ray, Hentges, Adão and Zhao, Xinqiang (2003) Robust Automatic Methods for Outlier and Error Detection (S3RI Methodology Working Papers, M03/17) Southampton, UK. Southampton Statistical Sciences Research Institute, University of Southampton 29pp.

Record type: Monograph (Project Report)

Abstract

Editing in surveys of economic populations is often complicated by the fact that outliers due to errors in the data are mixed in with correct, but extreme, data values. In this paper we describe and evaluate two automatic techniques for error identification in such long tailed data distributions. The first is a forward search procedure based on finding a sequence of error-free subsets of the error contaminated data and then using regression modelling within these subsets to identify errors. The second uses a robust regression tree modelling procedure to identify errors. Both approaches can be implemented on a univariate basis or on a multivariate basis. An application to a business survey data set that contains a mix of extreme errors and true outliers is
described.

Text

8167-01.pdf - Other

Available under License University of Southampton Accepted Manuscript Licence.

Download (3MB)