The University of Southampton
University of Southampton Institutional Repository

On the equivalence between Total Least Squares and Maximum Likelihood PCA

On the equivalence between Total Least Squares and Maximum Likelihood PCA
On the equivalence between Total Least Squares and Maximum Likelihood PCA
The maximum likelihood PCA (MLPCA) method has been devised in chemometrics as a generalization of the well-known PCA method in order to derive consistent estimators in the presence of errors with known error distribution. For similar reasons, the total least squares (TLS) method has been generalized in the field of computational mathematics and engineering to maintain consistency of the parameter estimates in linear models with measurement errors of known distribution. The basic motivation for TLS is the following. Let a set of multidimensional data points (vectors) be given. How can one obtain a linear model that explains these data? The idea is to modify all data points in such a way that some norm of the modification is minimized subject to the constraint that the modified vectors satisfy a linear relation. Although the name “total least squares” appeared in the literature only 25 years ago, this method of fitting is certainly not new and has a long history in the statistical literature, where the method is known as “orthogonal regression”, “errors-in-variables regression” or “measurement error modeling”. The purpose of this paper is to explore the tight equivalences between MLPCA and element-wise weighted TLS (EW-TLS). Despite their seemingly different problem formulation, it is shown that both methods can be reduced to the same mathematical kernel problem, i.e. finding the closest (in a certain sense) weighted low rank matrix approximation where the weight is derived from the distribution of the errors in the data. Different solution approaches, as used in MLPCA and EW-TLS, are discussed. In particular, we will discuss the weighted low rank approximation (WLRA), the MLPCA, the EW-TLS and the generalized TLS (GTLS) problems. These four approaches tackle an equivalent weighted low rank approximation problem, but different algorithms are used to come up with the best approximation matrix. We will compare their computation times on chemical data and discuss their convergence behavior.
TLS, MLPCA, Rank reduction, Measurement errors
0003-2670
254-267
Schuermans, M.
ebcbfd85-0c9d-4932-af28-5ab81d12a93d
Markovsky, I.
3e68743b-f22e-4b2b-b1a8-2ba4eb036a69
Wentzell, P.
5fd93c63-032c-4bad-9eae-1a90cf0faf79
Van Huffel, S.
e64be3d0-00e1-4900-ab8e-74aed4792678
Schuermans, M.
ebcbfd85-0c9d-4932-af28-5ab81d12a93d
Markovsky, I.
3e68743b-f22e-4b2b-b1a8-2ba4eb036a69
Wentzell, P.
5fd93c63-032c-4bad-9eae-1a90cf0faf79
Van Huffel, S.
e64be3d0-00e1-4900-ab8e-74aed4792678

Schuermans, M., Markovsky, I., Wentzell, P. and Van Huffel, S. (2005) On the equivalence between Total Least Squares and Maximum Likelihood PCA. Analytica Chimica Acta, 544 (1-2), 254-267.

Record type: Article

Abstract

The maximum likelihood PCA (MLPCA) method has been devised in chemometrics as a generalization of the well-known PCA method in order to derive consistent estimators in the presence of errors with known error distribution. For similar reasons, the total least squares (TLS) method has been generalized in the field of computational mathematics and engineering to maintain consistency of the parameter estimates in linear models with measurement errors of known distribution. The basic motivation for TLS is the following. Let a set of multidimensional data points (vectors) be given. How can one obtain a linear model that explains these data? The idea is to modify all data points in such a way that some norm of the modification is minimized subject to the constraint that the modified vectors satisfy a linear relation. Although the name “total least squares” appeared in the literature only 25 years ago, this method of fitting is certainly not new and has a long history in the statistical literature, where the method is known as “orthogonal regression”, “errors-in-variables regression” or “measurement error modeling”. The purpose of this paper is to explore the tight equivalences between MLPCA and element-wise weighted TLS (EW-TLS). Despite their seemingly different problem formulation, it is shown that both methods can be reduced to the same mathematical kernel problem, i.e. finding the closest (in a certain sense) weighted low rank matrix approximation where the weight is derived from the distribution of the errors in the data. Different solution approaches, as used in MLPCA and EW-TLS, are discussed. In particular, we will discuss the weighted low rank approximation (WLRA), the MLPCA, the EW-TLS and the generalized TLS (GTLS) problems. These four approaches tackle an equivalent weighted low rank approximation problem, but different algorithms are used to come up with the best approximation matrix. We will compare their computation times on chemical data and discuss their convergence behavior.

Text
mieke.pdf - Other
Download (313kB)

More information

Published date: 2005
Keywords: TLS, MLPCA, Rank reduction, Measurement errors
Organisations: Southampton Wireless Group

Identifiers

Local EPrints ID: 263302
URI: http://eprints.soton.ac.uk/id/eprint/263302
ISSN: 0003-2670
PURE UUID: e61f88fb-368f-4c9b-ac7e-6af3e4750fb0

Catalogue record

Date deposited: 06 Jan 2007
Last modified: 14 Mar 2024 07:29

Export record

Contributors

Author: M. Schuermans
Author: I. Markovsky
Author: P. Wentzell
Author: S. Van Huffel

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×