The University of Southampton
University of Southampton Institutional Repository
Warning ePrints Soton is experiencing an issue with some file downloads not being available. We are working hard to fix this. Please bear with us.

Correction for bias in meta-analysis of little-replicated studies

Correction for bias in meta-analysis of little-replicated studies
Correction for bias in meta-analysis of little-replicated studies
1. Meta-analyses conventionally weight study estimates on the inverse of their error variance, in order to maximize precision. Unbiased variability in the estimates of these study-level error variances increases with the inverse of study-level replication. Here, we demonstrate how this variability accumulates asymmetrically across studies in precision-weighted meta-analysis, to cause undervaluation of the meta-level effect size or its error variance (the meta-effect and meta-variance).

2. Small samples, typical of the ecological literature, induce big sampling errors in variance estimation, which substantially bias precision-weighted meta-analysis. Simulations revealed that biases differed little between random- and fixed-effects tests. Meta-estimation of a one-sample mean from 20 studies, with sample sizes of 3–20 observations, undervalued the meta-variance by c. 20%. Meta-analysis of two-sample designs from 20 studies, with sample sizes of 3–10 observations, undervalued the meta-variance by 15%–20% for the log response ratio (lnR); it undervalued the meta-effect by c. 10% for the standardized mean difference (SMD).

3. For all estimators, biases were eliminated or reduced by a simple adjustment to the weighting on study precision. The study-specific component of error variance prone to sampling error and not parametrically attributable to study-specific replication was replaced by its cross-study mean, on the assumptions of random sampling from the same population variance for all studies, and sufficient studies for averaging. Weighting each study by the inverse of this mean-adjusted error variance universally improved accuracy in estimation of both the meta-effect and its significance, regardless of number of studies. For comparison, weighting only on sample size gave the same improvement in accuracy, but could not sensibly estimate significance.

4. For the one-sample mean and two-sample lnR, adjusted weighting also improved estimation of between-study variance by DerSimonian-Laird and REML methods. For random-effects meta-analysis of SMD from little-replicated studies, the most accurate meta-estimates obtained from adjusted weights following conventionally weighted estimation of between-study variance.
We recommend adoption of weighting by inverse adjusted-variance for meta-analyses of well- and little-replicated studies, because it improves accuracy and significance of meta-estimates, and it can extend the scope of the meta-analysis to include some studies without variance estimates.
fixed effect; Hedges’ d; Hedges’ g; inverse-variance weighting; ln R; random effect; small sample
2041-210X
634-644
Doncaster, C. Patrick
0eff2f42-fa0a-4e35-b6ac-475ad3482047
Spake, Rebecca
1cda8ad0-2ab2-45d9-a844-ec3d8be2786a
Doncaster, C. Patrick
0eff2f42-fa0a-4e35-b6ac-475ad3482047
Spake, Rebecca
1cda8ad0-2ab2-45d9-a844-ec3d8be2786a

Doncaster, C. Patrick and Spake, Rebecca (2018) Correction for bias in meta-analysis of little-replicated studies. Methods in Ecology and Evolution, 9 (3), 634-644. (doi:10.1111/2041-210X.12927).

Record type: Article

Abstract

1. Meta-analyses conventionally weight study estimates on the inverse of their error variance, in order to maximize precision. Unbiased variability in the estimates of these study-level error variances increases with the inverse of study-level replication. Here, we demonstrate how this variability accumulates asymmetrically across studies in precision-weighted meta-analysis, to cause undervaluation of the meta-level effect size or its error variance (the meta-effect and meta-variance).

2. Small samples, typical of the ecological literature, induce big sampling errors in variance estimation, which substantially bias precision-weighted meta-analysis. Simulations revealed that biases differed little between random- and fixed-effects tests. Meta-estimation of a one-sample mean from 20 studies, with sample sizes of 3–20 observations, undervalued the meta-variance by c. 20%. Meta-analysis of two-sample designs from 20 studies, with sample sizes of 3–10 observations, undervalued the meta-variance by 15%–20% for the log response ratio (lnR); it undervalued the meta-effect by c. 10% for the standardized mean difference (SMD).

3. For all estimators, biases were eliminated or reduced by a simple adjustment to the weighting on study precision. The study-specific component of error variance prone to sampling error and not parametrically attributable to study-specific replication was replaced by its cross-study mean, on the assumptions of random sampling from the same population variance for all studies, and sufficient studies for averaging. Weighting each study by the inverse of this mean-adjusted error variance universally improved accuracy in estimation of both the meta-effect and its significance, regardless of number of studies. For comparison, weighting only on sample size gave the same improvement in accuracy, but could not sensibly estimate significance.

4. For the one-sample mean and two-sample lnR, adjusted weighting also improved estimation of between-study variance by DerSimonian-Laird and REML methods. For random-effects meta-analysis of SMD from little-replicated studies, the most accurate meta-estimates obtained from adjusted weights following conventionally weighted estimation of between-study variance.
We recommend adoption of weighting by inverse adjusted-variance for meta-analyses of well- and little-replicated studies, because it improves accuracy and significance of meta-estimates, and it can extend the scope of the meta-analysis to include some studies without variance estimates.

Text
Doncaster_et_al-2017-Methods_in_Ecology_and_Evolution - Version of Record
Available under License Creative Commons Attribution.
Download (1MB)

More information

Accepted/In Press date: 19 October 2017
e-pub ahead of print date: 21 November 2017
Published date: March 2018
Keywords: fixed effect; Hedges’ d; Hedges’ g; inverse-variance weighting; ln R; random effect; small sample

Identifiers

Local EPrints ID: 416153
URI: http://eprints.soton.ac.uk/id/eprint/416153
ISSN: 2041-210X
PURE UUID: 2fadce8a-f14b-49d2-aef1-dc8ceaaec3da
ORCID for C. Patrick Doncaster: ORCID iD orcid.org/0000-0001-9406-0693

Catalogue record

Date deposited: 06 Dec 2017 17:30
Last modified: 22 Nov 2021 02:39

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×