The efficacy of propensity score matching for separating selection and measurement effects across different survey modes
The efficacy of propensity score matching for separating selection and measurement effects across different survey modes
Effective evaluation of data quality between data collected in different modes is complicated by the confounding of selection and measurement effects. This study evaluates the utility of propensity score matching (PSM) as a method that has been proposed as a means of removing selection effects across surveys conducted in different modes. Our results show large differences in estimates for the same variables between parallel face-to-face and online surveys, even after matching on standard demographic variables. Moreover, discrepancies in estimates are still present after matching between surveys conducted in the same (online) mode, where differences in measurement properties can be ruled out a priori. Our findings suggest that PSM has substantial limitations as a method for separating measurement and selection differences across modes and should be used only with caution.
1-26
Kibuchi, Eliud
a8e48182-8b0a-48f9-8c53-f610726b0974
Sturgis, Patrick
b9f6b40c-50d2-4117-805a-577b501d0b3c
Durrant, Gabriele B.
14fcc787-2666-46f2-a097-e4b98a210610
Maslovskaya, Olga
9c979052-e9d7-4400-a657-38f1f9cd74d0
17 April 2024
Kibuchi, Eliud
a8e48182-8b0a-48f9-8c53-f610726b0974
Sturgis, Patrick
b9f6b40c-50d2-4117-805a-577b501d0b3c
Durrant, Gabriele B.
14fcc787-2666-46f2-a097-e4b98a210610
Maslovskaya, Olga
9c979052-e9d7-4400-a657-38f1f9cd74d0
Kibuchi, Eliud, Sturgis, Patrick, Durrant, Gabriele B. and Maslovskaya, Olga
(2024)
The efficacy of propensity score matching for separating selection and measurement effects across different survey modes.
Journal of Survey Statistics and Methodology, .
(doi:10.1093/jssam/smae017).
Abstract
Effective evaluation of data quality between data collected in different modes is complicated by the confounding of selection and measurement effects. This study evaluates the utility of propensity score matching (PSM) as a method that has been proposed as a means of removing selection effects across surveys conducted in different modes. Our results show large differences in estimates for the same variables between parallel face-to-face and online surveys, even after matching on standard demographic variables. Moreover, discrepancies in estimates are still present after matching between surveys conducted in the same (online) mode, where differences in measurement properties can be ruled out a priori. Our findings suggest that PSM has substantial limitations as a method for separating measurement and selection differences across modes and should be used only with caution.
Text
smae017
- Version of Record
More information
Published date: 17 April 2024
Identifiers
Local EPrints ID: 489433
URI: http://eprints.soton.ac.uk/id/eprint/489433
ISSN: 2325-0984
PURE UUID: 395e8547-be1d-4e74-8aaa-20ffd4c5c2f8
Catalogue record
Date deposited: 24 Apr 2024 16:33
Last modified: 18 May 2024 01:40
Export record
Altmetrics
Contributors
Author:
Eliud Kibuchi
Author:
Patrick Sturgis
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics