The University of Southampton
University of Southampton Institutional Repository

Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis

Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis
Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis
We assess how survey outcome distributions change over repeated calls made to addresses in face-to- face household interview surveys. We consider this question for 559 survey variables, drawn from six major face-to-face UK surveys which have different sample designs, cover different topic areas, and achieve response rates between 54% and 76%. Using a multi-level meta-analytic framework, we estimate for each survey variable, the expected difference between the point estimate for a proportion at call n and for the full achieved sample. We find that most variables are surprisingly close to the final achieved sample distribution after only one or two call attempts and before any post- stratification weighting has been applied; the mean expected difference from the final sample proportion across all 559 variables after 1 call is 1.6%, dropping to 0.7% after 3 calls, and to 0.4% after 5 calls. These estimates vary only marginally across the six surveys and the different types of questions examined. Our findings add further weight to the body of evidence which questions the strength of the relationship between response rate and nonresponse bias. In practical terms, our results suggest that making large numbers of calls at sampled addresses and converting ‘soft’ refusals into interviews are not cost-effective means of minimizing survey error.
0033-362X
523-542
Sturgis, Patrick
b9f6b40c-50d2-4117-805a-577b501d0b3c
Williams, Joel
c8d7987b-d6cf-4cb1-86d3-d97fdf05f4da
Brunton-Smith, Ian
fdb27626-ba05-4d54-b00a-12a8c0a82db3
Moore, Jamie
5f015c47-3165-4f64-8561-7c047a9d2186
Sturgis, Patrick
b9f6b40c-50d2-4117-805a-577b501d0b3c
Williams, Joel
c8d7987b-d6cf-4cb1-86d3-d97fdf05f4da
Brunton-Smith, Ian
fdb27626-ba05-4d54-b00a-12a8c0a82db3
Moore, Jamie
5f015c47-3165-4f64-8561-7c047a9d2186

Sturgis, Patrick, Williams, Joel, Brunton-Smith, Ian and Moore, Jamie (2016) Fieldwork effort, response rate, and the distribution of survey outcomes: a multi-level meta-analysis. Public Opinion Quarterly, 81 (2), 523-542. (doi:10.1093/poq/nfw055).

Record type: Article

Abstract

We assess how survey outcome distributions change over repeated calls made to addresses in face-to- face household interview surveys. We consider this question for 559 survey variables, drawn from six major face-to-face UK surveys which have different sample designs, cover different topic areas, and achieve response rates between 54% and 76%. Using a multi-level meta-analytic framework, we estimate for each survey variable, the expected difference between the point estimate for a proportion at call n and for the full achieved sample. We find that most variables are surprisingly close to the final achieved sample distribution after only one or two call attempts and before any post- stratification weighting has been applied; the mean expected difference from the final sample proportion across all 559 variables after 1 call is 1.6%, dropping to 0.7% after 3 calls, and to 0.4% after 5 calls. These estimates vary only marginally across the six surveys and the different types of questions examined. Our findings add further weight to the body of evidence which questions the strength of the relationship between response rate and nonresponse bias. In practical terms, our results suggest that making large numbers of calls at sampled addresses and converting ‘soft’ refusals into interviews are not cost-effective means of minimizing survey error.

Text
Paper_Draft_final_RR_April16_submitted.pdf - Accepted Manuscript
Download (2MB)

More information

Accepted/In Press date: 19 July 2016
e-pub ahead of print date: 13 December 2016
Organisations: Social Statistics & Demography

Identifiers

Local EPrints ID: 398569
URI: https://eprints.soton.ac.uk/id/eprint/398569
ISSN: 0033-362X
PURE UUID: f7e17691-a433-4085-80ee-1267f879938f
ORCID for Patrick Sturgis: ORCID iD orcid.org/0000-0003-1180-3493

Catalogue record

Date deposited: 28 Jul 2016 10:36
Last modified: 15 Jan 2019 01:33

Export record

Altmetrics

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×