The University of Southampton
University of Southampton Institutional Repository

New treatments compared to established treatments in randomized trials (Review)

New treatments compared to established treatments in randomized trials (Review)
New treatments compared to established treatments in randomized trials (Review)
Background
The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments.

Objectives
Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results.

Search methods
We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010).

Selection criteria
Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans.

Data collection and analysis
RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival.

Main results
The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I2 = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy).

Authors' conclusions
Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.
1465-1858
Djulbegovic, B.
9cbfa39c-8e1a-4b02-8dee-e4d83f520aad
Kumar, A.
d5d62c56-87f6-4ae1-ac67-2fb4bf8db422
Glasziou, P.P.
581b90a5-025c-48a6-a59a-cc4f3c603fbf
Perera, R.
d0b24f7a-0794-4563-92b9-3e624d725378
Reljic, T.
b263b4b1-c108-4713-8b17-2676cc9571ca
Dent, Louise
8b827763-d839-4b4b-bbf2-358a84110294
Raftery, J.
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Johansen, M.
f775a23d-2304-4f77-84ea-30671572f9a3
Di Tanna, G.L.
e28735e5-a37c-4cf3-9985-d392ee922983
Miladinovic, B.
dcd70465-92b9-4646-a5d8-5004b116c91d
Soares, H.P.
d4a3b0cf-21c2-4aa6-8116-77c3b7469afe
Vist, G.E.
ea3be7ec-c425-46ad-97ce-a9514fad8b95
Chalmers, I.
fe8a4da7-35a6-4352-9fdf-5edbc08ef94f
Djulbegovic, B.
9cbfa39c-8e1a-4b02-8dee-e4d83f520aad
Kumar, A.
d5d62c56-87f6-4ae1-ac67-2fb4bf8db422
Glasziou, P.P.
581b90a5-025c-48a6-a59a-cc4f3c603fbf
Perera, R.
d0b24f7a-0794-4563-92b9-3e624d725378
Reljic, T.
b263b4b1-c108-4713-8b17-2676cc9571ca
Dent, Louise
8b827763-d839-4b4b-bbf2-358a84110294
Raftery, J.
27c2661d-6c4f-448a-bf36-9a89ec72bd6b
Johansen, M.
f775a23d-2304-4f77-84ea-30671572f9a3
Di Tanna, G.L.
e28735e5-a37c-4cf3-9985-d392ee922983
Miladinovic, B.
dcd70465-92b9-4646-a5d8-5004b116c91d
Soares, H.P.
d4a3b0cf-21c2-4aa6-8116-77c3b7469afe
Vist, G.E.
ea3be7ec-c425-46ad-97ce-a9514fad8b95
Chalmers, I.
fe8a4da7-35a6-4352-9fdf-5edbc08ef94f

Djulbegovic, B., Kumar, A., Glasziou, P.P., Perera, R., Reljic, T., Dent, Louise, Raftery, J., Johansen, M., Di Tanna, G.L., Miladinovic, B., Soares, H.P., Vist, G.E. and Chalmers, I. (2012) New treatments compared to established treatments in randomized trials (Review). The Cochrane Library, (10). (doi:10.1002/14651858).

Record type: Article

Abstract

Background
The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments.

Objectives
Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results.

Search methods
We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010).

Selection criteria
Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans.

Data collection and analysis
RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival.

Main results
The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I2 = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy).

Authors' conclusions
Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

Text
downloadstats - Version of Record
Restricted to Repository staff only
Request a copy
Text
tables - Version of Record
Restricted to Repository staff only
Request a copy
Text
full - Version of Record
Restricted to Repository staff only
Request a copy

More information

Published date: 17 October 2012
Additional Information: Random allocation to different groups to compare the effects of treatments is used in fair tests to find out which among the treatment options is preferable. Random allocation is only ethical, however, if there is genuine uncertainty about which of the treatment options is preferable. If a patient or their healthcare provider is certain which of the treatments being compared is preferable they should not agree to random allocation, because this would involve the risk that they would be assigned to a treatment they believed to be inferior. Decisions about whether to participate in randomized trials are made more difficult because of the widespread belief that new treatments must inevitably be superior to existing (standard) treatments. Indeed, it is understandable that people hope that this will be the case. If this was actually so, however, the ethical precondition of uncertainty would often not apply. This Cochrane methodology review addresses this important question: "What is the likelihood that new treatments being compared to established treatments in randomized trials will be shown to be superior?" Four cohorts of consecutive, publicly funded, randomized trials, which altogether included 743 trials that enrolled 297,744 patients, met our inclusion criteria for this review. We found that, on average, new treatments were very slightly more likely to have favorable results than established treatments, both in terms of the primary outcomes targeted and overall survival. In other words, when new treatments are compared with established treatments in randomized trials we can expect slightly more than half will prove to be better, and slightly less than half will prove to be worse than established treatments. This conclusion applies to publicly funded trials as we did not include studies funded by commercial sponsors in our analysis.The results are consistent with the ethical preconditions for random allocation – when people are enrolled in randomized trials, the results cannot be predicted in advance as there is genuine uncertainty about which of the treatments being compared in randomized trials will prove to be superior.
Organisations: Faculty of Medicine

Identifiers

Local EPrints ID: 354837
URI: http://eprints.soton.ac.uk/id/eprint/354837
ISSN: 1465-1858
PURE UUID: 0bf2ddc4-0ebb-41d9-933a-82994fa9e747
ORCID for Louise Dent: ORCID iD orcid.org/0000-0001-8181-840X

Catalogue record

Date deposited: 31 Jul 2013 15:41
Last modified: 17 Dec 2019 01:44

Export record

Altmetrics

Contributors

Author: B. Djulbegovic
Author: A. Kumar
Author: P.P. Glasziou
Author: R. Perera
Author: T. Reljic
Author: Louise Dent ORCID iD
Author: J. Raftery
Author: M. Johansen
Author: G.L. Di Tanna
Author: B. Miladinovic
Author: H.P. Soares
Author: G.E. Vist
Author: I. Chalmers

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×