Classical Multidimensional Scaling: New Perspectives from EDM Optimization

Abstract

The classical Multi-Dimensional Scaling (cMDS) has become a cornerstone for analyzing metric dissimilarity data due to its simplicity in derivation, low computational complexity and its nice interpretation via the principle component analysis. It originated from the classical Euclidean geometry and has found a large number of applications in various disciplines both in sciences and social sciences. Its popularity has significantly extended to machine learning community due to its modernized version ISOMAP. The purpose of this thesis is to study why cMDS works from an optimization point of view and in particular, how cMDS works hand in hand with modern optimization under the framework of Euclidean Distance Matrix (EDM) Optimization.
The EDM optimization from cMDS has three difficulties to solve. One is from the requirement of low-dimensional embedding, which often results in a low-rank constraint in EDM optimization. The second is from the Euclidean distance constraints, which often can be reformulated as a connic constraint. The third difficulty is caused by many of the constraints enforced on certain distances such as lower and upper bounds constraints. Modern matrix optimization can efficiently handle those difficulties arose from the standard MDS problems. However, our target is not any of the standard MDS problems. It is related to outlier detection among given noisy distances.
Despite its wide and successful use in various disciplines, the view on the capability of cMDS of denoising and outlier detection is largely negative. However, its reason has never been seriously investigated. Our new interpretation shows that cMDS always overly
denoises a sparsely perturbed data by subtracting a fully dense denoising matrix in a subspace from the given data matrix. This leads us to consider two types of sparsitydriven models: Subspace sparse MDS and Full-space sparse MDS, which respectively uses the `1 and `1􀀀2 regularization to induce sparsity. For the subspace sparse MDS, we developed a proximal alternating direction method of multipliers (ADMM) and established convergence properties, its numerical performance is not as good as we expected. It can be used to solve some less challenging problems.
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.
its convergence. Interestingly, although ADMM has all the promised (and nice)
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish
their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial
and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.

More information

Identifiers

ORCID for Hou-Duo Qi: orcid.org/0000-0003-3481-4814

Catalogue record

Export record