An empirical review of uncertainty estimation for quality control in CAD model segmentation

Abstract

Deep neural networks are able to achieve high accuracy in semantic segmentation of geometries used in computational engineering. Being able to recognise abstract and sometimes hard to describe geometric features has applications for automated simulation, model simplification, structural failure analysis, meshing, and additive manufacturing. However, for these systems to be integrated into engineering workflows, they must provide some measures of predictive uncertainty such that engineers can reason about and trust their outputs. This work presents an empirical study of practical uncertainty estimation techniques that can be used with pre-trained neural networks for the task of boundary representation model segmentation. A point-based graph neural network is used as a base. Monte-Carlo Dropout (MCD), Deep Ensembles, test time input augmentation, and post-processing calibration are evaluated for segmentation quality control. The Deep Ensemble technique is found to be top performing and the error of a human-in-the-loop system across a dataset can be reduced from 3.8% to 0.7% for MFCAD++ and from16% to 11% for Fusion360 Gallery when 10% of the most uncertain predictions are flagged for manual correction. Models trained on only 5%of the MFCAD++ dataset were also tested, with the uncertainty estimation technique reducing the error from 9.4% to 4.3% with 10% of predictions flagged. Additionally, a point-based input augmentation is presented; which, when combined with MCD, is competitive with the Deep Ensemble while having lower computational requirements.

More information

Identifiers

ORCID for David Toal: orcid.org/0000-0002-2203-0302

ORCID for Andy Keane: orcid.org/0000-0001-7993-1569

ORCID for Xu Zhang: orcid.org/0000-0002-6918-1861

Catalogue record

Export record

Altmetrics