Gaussian oblique decision tree technique for classification and regression

Abstract

Decision tree techniques remain a cornerstone of interpretable machine learning techniques. Specifically, axis-parallel decision tree techniques are very popular in real-world applications. However, traditional axis-parallel decision tree techniques that perform orthogonal splits construct staircase decision boundaries, which would limit the predictive power when presented with data that has strong feature interactions. Hence, we propose a novel oblique decision tree technique that constructs a multivariate decision tree, called Gaussian mixture model Oblique Decision Trees (GODT) for classification and regression. We also propose a novel quadratic decision tree technique that constructs quadratic decision boundaries at each non-terminal node to split the data, called Gaussian mixture model Quadratic Decision Tree (GQDT) for classification. In these proposed techniques, the Gaussian mixture model is efficiently incorporated into the architecture of a binary tree-building process. We propose a novel and computationally cheaper way to construct hyperplanes for the oblique tree and hypersurface for the quadratic decision tree techniques. This is because, most of the current techniques have a high computational cost associated to them when trying to achieve a good prediction accuracy. Experiments using the GODT for classification and regression tasks, and GQDT on classification tasks on open-source datasets illustrate that both GODT and GQDT performs better in most cases when compared against existing established techniques both in test accuracy and computational times. Furthermore, as single tree-based techniques are known to have high-variance and sensitivity to training data, we extend our work to build an ensemble of GODT and GQDT techniques. Finally, we run extensive experiments to test the performance of the techniques that we propose on open-source datasets and we compare the performance against established techniques like the Classification and Regression Trees (CART), Support Vector Machines (SVM), Standard Random Forest (RF), Linear Discriminant Analysis (LDA), and Quadratic Discriminant Analysis (QDA).

More information

Identifiers

ORCID for Aditya Sivanand: orcid.org/0009-0004-8152-8380

ORCID for Joerg Fliege: orcid.org/0000-0002-4459-5419

ORCID for Erengul Dodd: orcid.org/0000-0001-6658-0990

Catalogue record

Export record