The University of Southampton
University of Southampton Institutional Repository

The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

The statistical interpretation of pilot trials: should significance thresholds be reconsidered?
The statistical interpretation of pilot trials: should significance thresholds be reconsidered?
In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit. We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised. We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology. We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.
1471-2288
1-8
Lee, C. Ellen
2b147d7c-1e36-4c75-86c4-4049d8498dd2
Whitehead, Amy
7bd4e1d1-078b-4f2b-bfc9-ed44ba0a195a
Jacques, Richard
5fd134d2-4cb8-44e2-aff0-0711ac6bec54
Julious, Steven A.
e70c3d71-b62d-41a8-8e98-df954f127935
Lee, C. Ellen
2b147d7c-1e36-4c75-86c4-4049d8498dd2
Whitehead, Amy
7bd4e1d1-078b-4f2b-bfc9-ed44ba0a195a
Jacques, Richard
5fd134d2-4cb8-44e2-aff0-0711ac6bec54
Julious, Steven A.
e70c3d71-b62d-41a8-8e98-df954f127935

Lee, C. Ellen, Whitehead, Amy, Jacques, Richard and Julious, Steven A. (2014) The statistical interpretation of pilot trials: should significance thresholds be reconsidered? BMC Medical Research Methodology, 14 (41), 1-8. (doi:10.1186/1471-2288-14-41).

Record type: Article

Abstract

In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit. We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised. We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology. We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

Text
1471-2288-14-41 - Version of Record
Available under License Creative Commons Attribution.
Download (477kB)

More information

e-pub ahead of print date: 20 March 2014
Published date: December 2014

Identifiers

Local EPrints ID: 421837
URI: https://eprints.soton.ac.uk/id/eprint/421837
ISSN: 1471-2288
PURE UUID: 5d4beb8d-cc92-4f16-8f5a-9da704b4dc64

Catalogue record

Date deposited: 29 Jun 2018 16:30
Last modified: 19 Jul 2019 17:11

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×