Outlier detection and robust covariance estimation using mathematical programming
Nguyen, TriDzung and Welsch, Roy E. (2010) Outlier detection and robust covariance estimation using mathematical programming. Advances in Data Analysis and Classification, Special Issue on Robust Methods for Classification and Data Analysis, 4, (4), 301334. (doi:10.1007/s1163401000707).
Download
Full text not available from this repository.
Description/Abstract
The outlier detection problem and the robust covariance estimation problem are often interchangeable. Without outliers, the classical method of maximum likelihood estimation (MLE) can be used to estimate parameters of a known distribution from observational data. When outliers are present, they dominate the log likelihood function causing the MLE estimators to be pulled toward them. Many robust statistical methods have been developed to detect outliers and to produce estimators that are robust against deviation from model assumptions. However, the existing methods suffer either from computational complexity when problem size increases or from giving up desirable properties, such as affine equivariance. An alternative approach is to design a special mathematical programming model to find the optimal weights for all the observations, such that at the optimal solution, outliers are given smaller weights and can be detected. This method produces a covariance estimator that has the following properties: First, it is affine equivariant. Second, it is computationally efficient even for large problem sizes. Third, it easy to incorporate prior beliefs into the estimator by using semidefinite programming. The accuracy of this method is tested for different contamination models, including recently proposed ones. The method is not only faster than the FastMCD method for high dimensional data but also has reasonable accuracy for the tested cases.
Item Type:  Article  

Digital Object Identifier (DOI):  doi:10.1007/s1163401000707  
ISSNs:  18625347 (print) 18625355 (electronic) 

Keywords:  covariance matrix estimation, robust statistics, outlier detection, optimization, semidefinite programming, newton–raphson method  
Subjects:  H Social Sciences > HA Statistics Q Science > QA Mathematics 

Divisions :  University Structure  Pre August 2011 > School of Management University Structure  Pre August 2011 > School of Mathematics > Operational Research University Structure  Pre August 2011 > School of Mathematics > Statistics 

ePrint ID:  181475  
Accepted Date and Publication Date: 


Date Deposited:  18 Apr 2011 13:35  
Last Modified:  31 Mar 2016 13:36  
URI:  http://eprints.soton.ac.uk/id/eprint/181475 
Actions (login required)
View Item 