The University of Southampton
University of Southampton Institutional Repository
Warning ePrints Soton is experiencing an issue with some file downloads not being available. We are working hard to fix this. Please bear with us.

Classical Multidimensional Scaling: New Perspectives from EDM Optimization

Classical Multidimensional Scaling: New Perspectives from EDM Optimization
Classical Multidimensional Scaling: New Perspectives from EDM Optimization
The classical Multi-Dimensional Scaling (cMDS) has become a cornerstone for analyzing metric dissimilarity data due to its simplicity in derivation, low computational complexity and its nice interpretation via the principle component analysis. It originated from the classical Euclidean geometry and has found a large number of applications in various disciplines both in sciences and social sciences. Its popularity has significantly extended to machine learning community due to its modernized version ISOMAP. The purpose of this thesis is to study why cMDS works from an optimization point of view and in particular, how cMDS works hand in hand with modern optimization under the framework of Euclidean Distance Matrix (EDM) Optimization.
The EDM optimization from cMDS has three difficulties to solve. One is from the requirement of low-dimensional embedding, which often results in a low-rank constraint in EDM optimization. The second is from the Euclidean distance constraints, which often can be reformulated as a connic constraint. The third difficulty is caused by many of the constraints enforced on certain distances such as lower and upper bounds constraints. Modern matrix optimization can efficiently handle those difficulties arose from the standard MDS problems. However, our target is not any of the standard MDS problems. It is related to outlier detection among given noisy distances.
Despite its wide and successful use in various disciplines, the view on the capability of cMDS of denoising and outlier detection is largely negative. However, its reason has never been seriously investigated. Our new interpretation shows that cMDS always overly
denoises a sparsely perturbed data by subtracting a fully dense denoising matrix in a subspace from the given data matrix. This leads us to consider two types of sparsitydriven models: Subspace sparse MDS and Full-space sparse MDS, which respectively uses the `1 and `1􀀀2 regularization to induce sparsity. For the subspace sparse MDS, we developed a proximal alternating direction method of multipliers (ADMM) and established convergence properties, its numerical performance is not as good as we expected. It can be used to solve some less challenging problems.
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.
its convergence. Interestingly, although ADMM has all the promised (and nice)
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish
their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial
and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.
University of Southampton
Qi, Chuanqi
3137b347-91a7-47f6-907d-ae1e6ae0177a
Qi, Chuanqi
3137b347-91a7-47f6-907d-ae1e6ae0177a
Qi, Hou-Duo
e9789eb9-c2bc-4b63-9acb-c7e753cc9a85

Qi, Chuanqi (2020) Classical Multidimensional Scaling: New Perspectives from EDM Optimization. University of Southampton, Doctoral Thesis, 119pp.

Record type: Thesis (Doctoral)

Abstract

The classical Multi-Dimensional Scaling (cMDS) has become a cornerstone for analyzing metric dissimilarity data due to its simplicity in derivation, low computational complexity and its nice interpretation via the principle component analysis. It originated from the classical Euclidean geometry and has found a large number of applications in various disciplines both in sciences and social sciences. Its popularity has significantly extended to machine learning community due to its modernized version ISOMAP. The purpose of this thesis is to study why cMDS works from an optimization point of view and in particular, how cMDS works hand in hand with modern optimization under the framework of Euclidean Distance Matrix (EDM) Optimization.
The EDM optimization from cMDS has three difficulties to solve. One is from the requirement of low-dimensional embedding, which often results in a low-rank constraint in EDM optimization. The second is from the Euclidean distance constraints, which often can be reformulated as a connic constraint. The third difficulty is caused by many of the constraints enforced on certain distances such as lower and upper bounds constraints. Modern matrix optimization can efficiently handle those difficulties arose from the standard MDS problems. However, our target is not any of the standard MDS problems. It is related to outlier detection among given noisy distances.
Despite its wide and successful use in various disciplines, the view on the capability of cMDS of denoising and outlier detection is largely negative. However, its reason has never been seriously investigated. Our new interpretation shows that cMDS always overly
denoises a sparsely perturbed data by subtracting a fully dense denoising matrix in a subspace from the given data matrix. This leads us to consider two types of sparsitydriven models: Subspace sparse MDS and Full-space sparse MDS, which respectively uses the `1 and `1􀀀2 regularization to induce sparsity. For the subspace sparse MDS, we developed a proximal alternating direction method of multipliers (ADMM) and established convergence properties, its numerical performance is not as good as we expected. It can be used to solve some less challenging problems.
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.
its convergence. Interestingly, although ADMM has all the promised (and nice)
Driven by the numerical weakness of the proximal ADMM, we then develop fast majorization algorithms for both the subspace model and the full space model. We establish
their convergence to a stationary point, which is the best outcome in general expected of an optimization algorithm. The majorization has been based on a penalty method, which penalizes the difficult low rank constraint to the objective. Moreover, we are able to control the sparsity level at every iterate provided that the sparsity control parameter is above a computable threshold. This is a desirable property that has not been enjoyed by any of existing sparse MDS methods. Our numerical experiments on both artificial
and real data demonstrates that cMDS with appropriate regularization can perform the tasks of denoising and outlier detection, and inherits the efficiency of cMDS in comparison with several state-of-the-art sparsity-driven MDS methods.

Text
Thesis_final - Version of Record
Available under License University of Southampton Thesis Licence.
Download (1MB)
Text
Permission to deposit thesis - form - Version of Record
Restricted to Repository staff only

More information

Published date: November 2020

Identifiers

Local EPrints ID: 451412
URI: http://eprints.soton.ac.uk/id/eprint/451412
PURE UUID: ce9d15f7-4fdd-4b8a-b596-ae556152f3f2
ORCID for Hou-Duo Qi: ORCID iD orcid.org/0000-0003-3481-4814

Catalogue record

Date deposited: 24 Sep 2021 16:35
Last modified: 25 Sep 2021 01:39

Export record

Contributors

Author: Chuanqi Qi
Thesis advisor: Hou-Duo Qi ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×