Arguments for and against the use of multiple comparison control in stochastic simulation studies

Abstract

Pick up any of the standard discrete-event simulation textbooks and you will find that the output analysis
section includes a note on multiple comparison control (MCC). These procedures aim to mitigate the
problem of inflating the probability of making a single type I error when comparing many simulated
scenarios simultaneously. We consider the use of MCC in stochastic simulation studies and present an
argument discouraging its use in the classical sense. In particular, we focus on the impracticality of
procedures, the benefits of common random numbers and that simulation is very different from empirical
studies where MCC has its roots. We then consider in what instances would abandoning MCC altogether
be problematic and what alternatives are available. We present an argument for medium to large
exploratory studies to move their attention away from classical Type I errors and instead control a
subtlety different quantity: the rate of false positives amongst all ‘discoveries’.

More information

Identifiers

ORCID for Thomas Monks: orcid.org/0000-0003-2631-4481

ORCID for Christine Currie: orcid.org/0000-0002-7016-3652

Catalogue record

Export record