Investigating the impact of technologies on the quality of data collected through surveys

Abstract

Social surveys continue to play an important role in social science and policy making. In addition to providing information about attitudes and behaviours, they act as a vehicle for the collection of many different types such as biomeasures, geographical data, administrative records, social media posts and so on. The resilience and adaptability of the social survey owes much to the way that they have adapted to the enormous changes in the technological environment. Radical changes in telephony, personal computing, the internet, and mobile devices have transformed many aspects of the research process. While these changes have brought many benefits, the application of each new technology in survey data collection needs careful consideration in terms of ethics, burden, cost, and implementation. Moreover, they may introduce representation errors and measurement errors that must be accounted for. In this context, this thesis considers the effects of new technologies on data quality and measurement error. It presents three examples of the use of technologies in social surveys and examines an aspect of data quality in relation to each. The thesis makes specific recommendations and encourages further methodological research in the use of technology in survey data collection.
The focus of the first study is on three biomeasures which are frequently collected in health or multidisciplinary surveys but may be recorded using different equipment. A randomised cross-over trial of 118 healthy adults aged 45-74 years was conducted using two sphygmomanometers to measure blood pressure, four handgrip dynamometers to measure grip strength, and two spirometers to measure lung function. For each of these three measures, multiple readings from each device were combined with information about the individual, drawn from a self-completion questionnaire, to build a pseudo-anonymised analytical dataset. Evidence was found of differences in measurements when assessed using alternative devices. For blood pressure, there is a difference, on average, of 3.85 mm Hg for Systolic Blood Pressure and 1.35 mm Hg for Diastolic Blood Pressure. For grip strength, two electronic dynamometers record measurements
on average 4-5kg higher than either a hydraulic or a spring-gauge dynamometer. For lung function, a difference of 0.47 litres, on average, was found for measures of Forced Vital Capacity, but no difference was found in measures of Forced Expiratory Volume. The primary analysis was conducted using Bland and Altman plots. Sensitivity analyses tested different definitions of each measure and used multilevel regression modelling as an alternative way of estimating device effects. The findings have implications for analysts who may want to test the sensitivity of their findings to the average differences observed with these combinations of devices and may help investigators who are selecting equipment for new studies or changing equipment for longitudinal studies. Further trials are needed to replicate the comparison of these devices and to test different device combinations, both in stand-alone studies and within larger observational surveys. Future analysts may wish to consider using multilevel modelling to assess device effects.
The second paper also considers device effects, this time, exploring whether the device used to complete an online survey (that is a PC, smartphone, or tablet) affects data quality. The study is based on the Wellcome Trust Science Education Tracker, a mobile-optimised, online survey of over 4,000 pupils aged 14-18. It uses the Wellcome Science Education Tracker 2016 dataset, available through the UK Data Archive, with additional survey process variables obtained with the agreement of the Wellcome Trust. The study uses propensity scores (more specifically, Inverse Probability Treatment Weights) to balance the samples, to reduce the possibility that measurement effects are confounded by selection. The analysis draws on linked geographical, administrative and survey process data which provides an opportunity to assess the use of exogenous confounder variables in the matching process. The large sample size makes it possible to test the sensitivity of the finding to the inclusion or exclusion of tablet users. Overall, the study identifies few consistent device effects, and those that are observed are small, providing reassurance for survey practitioners and analysts. After controlling for selection, those who use a mobile device are seen to have higher levels of “don’t know” responses and are more likely to have interruptions during survey completion. Contrary to the findings of some earlier studies, smartphone responders complete the survey more quickly than PC responders. The results for straightlining are mixed and no clear pattern between mobile and PC could be found. The findings encourage the inclusion of a wide range of covariates when controlling for selection, beyond basic demographics, ideally including exogenous variables, and including those which capture topic salience.
The third research study addresses the potential for app-based research. It is an exploratory study which assesses the quality of data collected using an app-based expenditure diary over a one-month period. A total of 268 members of the Understanding Society Innovation Panel agreed to take part. The analysis uses a combination of two datasets from Understanding Society: Spending Study 1 (2016-2017) and Wave 9 of the Innovation Panel (2016), both of which are available from
the UK Data Archive. Other analyses have explored initial response rates to this study, noting that just 16.5% of the invited sample completed the registration process and fewer still downloaded the app. In this study, the investigation of data quality involved defining and examining four measures of adherence to protocol, and the extent to which these aspects of adherence were sustained over the duration of the study period. The research identifies a reasonable level of engagement from those who agreed to participate in the app study. For example, the mean number of app use days in the one-month period was 21.7 and the mean number of spending events reported was 27.6. Almost all participants (96.6%) reported at least one spending event and of those, most (95%) used a combination of photographing receipts and making direct entries, or only photographed receipts, with 61% of all spending events reported by photographing receipts. Almost all of those (94.9%) who photographed one or more receipts which had relevant date information did so within, on average, 24 hours of the time of the spending event. Although adherence based on all four measures clearly declines across the study month, it remains reasonably high. This study provides encouragement for further development of the app, and further methodological research and experimentation to increase full and sustained adherence to protocol. If a spending study app is to be embedded successfully in a large-scale study such as Understanding Society, future efforts will inevitably focus on ways to raise initial participation rates, but it would be unfortunate if the particular benefits of app-based research, such as capturing detailed spending data from receipts using photographs, were entirely let go in favour of achieving higher initial response rates.

More information

Identifiers

ORCID for Gabriele Durrant: orcid.org/0009-0001-3436-1512

Catalogue record

Export record

Altmetrics