The University of Southampton
University of Southampton Institutional Repository

Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method

Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method
Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method

Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems, with each one for each fold in T-fold cross-validation. The resulting bilevel optimization model is then converted to a mathematical program with equilibrium constraints (MPEC). To solve this MPEC, we propose a global relaxation cross-validation algorithm (GR–CV) based on the well-know Sholtes-type global relaxation method (GRM). It is proven to converge to a C-stationary point. Moreover, we prove that the MPEC-tailored version of the Mangasarian–Fromovitz constraint qualification (MFCQ), which is a key property to guarantee the convergence of the GRM, automatically holds at each feasible point of this MPEC. Extensive numerical results verify the efficiency of the proposed approach. In particular, compared with other methods, our algorithm enjoys superior generalization performance over almost all the data sets used in this paper.

Bilevel optimization, C-stationarity, Hyperparameter selection, Mathematical program with equilibrium constraints, Support vector classification
1432-2994
315-350
Li, Qingna
a189d836-f8f0-407b-9983-0a73bf8a214a
Li, Zhen
12cdf2e2-b3a2-48ad-8201-dbdc8ceed67e
Zemkoho, Alain
30c79e30-9879-48bd-8d0b-e2fbbc01269e
Li, Qingna
a189d836-f8f0-407b-9983-0a73bf8a214a
Li, Zhen
12cdf2e2-b3a2-48ad-8201-dbdc8ceed67e
Zemkoho, Alain
30c79e30-9879-48bd-8d0b-e2fbbc01269e

Li, Qingna, Li, Zhen and Zemkoho, Alain (2022) Bilevel hyperparameter optimization for support vector classification: theoretical analysis and a solution method. Mathematical Methods of Operations Research, 96 (3), 315-350. (doi:10.1007/s00186-022-00798-6).

Record type: Article

Abstract

Support vector classification (SVC) is a classical and well-performed learning method for classification problems. A regularization parameter, which significantly affects the classification performance, has to be chosen and this is usually done by the cross-validation procedure. In this paper, we reformulate the hyperparameter selection problem for support vector classification as a bilevel optimization problem in which the upper-level problem minimizes the average number of misclassified data points over all the cross-validation folds, and the lower-level problems are the l1-loss SVC problems, with each one for each fold in T-fold cross-validation. The resulting bilevel optimization model is then converted to a mathematical program with equilibrium constraints (MPEC). To solve this MPEC, we propose a global relaxation cross-validation algorithm (GR–CV) based on the well-know Sholtes-type global relaxation method (GRM). It is proven to converge to a C-stationary point. Moreover, we prove that the MPEC-tailored version of the Mangasarian–Fromovitz constraint qualification (MFCQ), which is a key property to guarantee the convergence of the GRM, automatically holds at each feasible point of this MPEC. Extensive numerical results verify the efficiency of the proposed approach. In particular, compared with other methods, our algorithm enjoys superior generalization performance over almost all the data sets used in this paper.

Text
s00186-022-00798-6 - Version of Record
Available under License Creative Commons Attribution.
Download (1MB)

More information

Accepted/In Press date: 16 August 2022
e-pub ahead of print date: 26 August 2022
Published date: December 2022
Additional Information: Funding Information: A.Zemkoho: The work of this author is supported by the EPSRC grant EP/V049038/1 and the Alan Turing Institute under the EPSRC grant EP/N510129/1. Funding Information: Q.Li: This author’s research is supported by the National Science Foundation of China (NSFC) 12071032. Publisher Copyright: © 2022, The Author(s).
Keywords: Bilevel optimization, C-stationarity, Hyperparameter selection, Mathematical program with equilibrium constraints, Support vector classification

Identifiers

Local EPrints ID: 473404
URI: http://eprints.soton.ac.uk/id/eprint/473404
ISSN: 1432-2994
PURE UUID: af5b5244-40cd-446d-98b5-99e9bbc5f68e
ORCID for Alain Zemkoho: ORCID iD orcid.org/0000-0003-1265-4178

Catalogue record

Date deposited: 17 Jan 2023 17:47
Last modified: 06 Jun 2024 01:53

Export record

Altmetrics

Contributors

Author: Qingna Li
Author: Zhen Li
Author: Alain Zemkoho ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×