The University of Southampton
University of Southampton Institutional Repository

Review of options for the National Survey for Wales

Review of options for the National Survey for Wales
Review of options for the National Survey for Wales
Until March 2020, the National Survey for Wales was a face-to-face (CAPI) survey carried out in people’s homes. However, the Covid-19 pandemic meant that alternative National Survey designs have now been trialled. The Welsh Government are considering what design should be deployed for the National Survey in the next contracting phase. This report summarises the different design options that could be used for the National Survey in the future. All the survey design options have their own strengths and weaknesses. This report summarises the various trade-offs the Welsh Government need to consider when deciding on which design to procure.A set of survey design parameters was provided by the Welsh Government. These included criteria such as random probability sampling, minimum sample sizes, questionnaire length, reporting frequency, accessibility, being able to run in under lock-down conditions, the desire to reduce CO2emissions (compared to previous years of data collection), and the facilitation of recontact research. Requirements on practicability were also included, such whether contractors would be able to deliver the design, the complexity of analyses, and what can be achieved in the existing budget envelope. In terms of sample frame, access panels and random digit dialling were deemed unsuitable for the National Survey as they would not provide results based on a random sample. Therefore address-based sampling remains the preferred sampling method. The Postcode Address File (PAF) and AddressBase Plus are currently the only practical options for this. AddressBase Plus may have some advantages over the PAF in that it is updated more regularly, and that it provides better characterisation of property type (for example whether an address is a business address). Neither PAF nor AddressBase Plus include information on the name, telephone numbers or email addresses of people who could be living in the selected address. This means there are only two ways contact can be made with people at randomly selected addresses; mailcontact and/or face-to-face contact. Using a mail-only contact strategy has some advantages. The cost of the survey is reduced, CO2 emissions are reduced, and (in terms of the sampling design) there is no need for clustering in order to offset travel costs. A mail-only contact survey can be administered flexibly, which means that the sample can easily be distributed across the year, and the quarterly design can be nationally representative. Finally, a mail-only contact method could 6allow for more frequent reporting. However, the major downside to mail-only contact is a decreased response rate, and thus an increased risk of not achieving a representative sample. Having face-to-face contact will increase response rates and thus reduce the risk of nonresponse bias. If face-to-face contact methods are used there are various possible designs to consider. These are: Knock-to-nudge designs, where survey representatives visit addresses to encourage response but do not carry out in-home interviewing; Sequential CAPI, where face-to-face interviews are conducted with people who have not responded to an earlier mail invitation; and  CAPI-first designs, where all selected households are visited by an interviewer, and all are offered a face-to-face interview as the default mode of data collection. Knock-to-nudge designs offer some benefits over mail-only contact surveys, most notably the improved response rates and the increased likelihood of achieving a representative sample. However, it is unclear to what extent this method will remain in use post-pandemic now that in-home visits are permitted. There is a strong argument that once a survey representative has made contact on the doorstep the ideal option is to conduct a CAPI interview straight away, rather than risk drop-out by referring householders to a different mode of data collection.Sequential CAPI modes offer most of the benefits of CAP- only modes, with the added advantage they could offer some reductions in both cost and CO2 emissions. However, if multiple modes of contact and/or data collection are used, additional resource will be required from both the Welsh Government and contractors in terms of survey management.There will be more process documents to produce and sign off (procedures, mailings, interviewer instructions etc). Questionnaires will need to be checked and signed off for each mode of data collection. There will also be greater effort required in fieldwork monitoring, data processing and weighting. The more complicated the design the greater the resource required, from both Welsh Government and their contractors, and the higher the risk that errors could occur in practice. The mixed mode design will add some complexity to analysis.Finally, the CAPI-only mode (as used in the National Survey pre-pandemic albeit with a paper self-completion module) was considered the ‘gold-standard’ until recently due to its high response rates. However, the cost of conducting CAPI fieldwork is high compared to 7other modes of data collection. Retaining the CAPI only design would mean that reductions in interviewer mileage will not occur. If mileage reductions are not possible then CO2emissions will be more difficult to reduce. The CAPI only mode is not ‘future-proofed’ in the event of further lockdowns or if cost savings are required. In terms of data collection, the alternative modes to face-to-face interviewing under consideration are web questionnaires or telephone interviews (CATI). Both modes would be suitable for National Survey data collection. Developing a mobile-friendly National Survey web questionnaire would be a more effortful process than developing a CATI questionnaire, and in web data collection randomised within household selection methods are problematic. That said a sequential web-CAPI design would offer marginal cost savings compared to sequential CATI-CAPI design, and web designs may offer some improvement in data quality for sensitive questions compared to CATI designs. The report concludes with someindicative costs for a web-CAPI sequential design and a CATI-CAPI sequential design, for the Welsh Government to consider.
Welsh Government
Cornick, Peter
0d97dfd3-7b65-49ba-8eae-af6b0cd2dc13
d'Ardenne, Jo
79f0ea86-e46d-4cd4-9aa1-932dbf3f5f1e
Maslovskaya, Olga
9c979052-e9d7-4400-a657-38f1f9cd74d0
Mesplie-Cowan, Sierra
11a747c3-3000-486d-ae89-528d6d1500af
Nicolaas, Gerry
7bd9364f-ff09-4ebb-8d67-4cb5f3248ce0
Smith, Paul A.
a2548525-4f99-4baf-a4d0-2b216cce059c
Cornick, Peter
0d97dfd3-7b65-49ba-8eae-af6b0cd2dc13
d'Ardenne, Jo
79f0ea86-e46d-4cd4-9aa1-932dbf3f5f1e
Maslovskaya, Olga
9c979052-e9d7-4400-a657-38f1f9cd74d0
Mesplie-Cowan, Sierra
11a747c3-3000-486d-ae89-528d6d1500af
Nicolaas, Gerry
7bd9364f-ff09-4ebb-8d67-4cb5f3248ce0
Smith, Paul A.
a2548525-4f99-4baf-a4d0-2b216cce059c

Cornick, Peter, d'Ardenne, Jo, Maslovskaya, Olga, Mesplie-Cowan, Sierra, Nicolaas, Gerry and Smith, Paul A. (2022) Review of options for the National Survey for Wales Welsh Government 74pp.

Record type: Monograph (Project Report)

Abstract

Until March 2020, the National Survey for Wales was a face-to-face (CAPI) survey carried out in people’s homes. However, the Covid-19 pandemic meant that alternative National Survey designs have now been trialled. The Welsh Government are considering what design should be deployed for the National Survey in the next contracting phase. This report summarises the different design options that could be used for the National Survey in the future. All the survey design options have their own strengths and weaknesses. This report summarises the various trade-offs the Welsh Government need to consider when deciding on which design to procure.A set of survey design parameters was provided by the Welsh Government. These included criteria such as random probability sampling, minimum sample sizes, questionnaire length, reporting frequency, accessibility, being able to run in under lock-down conditions, the desire to reduce CO2emissions (compared to previous years of data collection), and the facilitation of recontact research. Requirements on practicability were also included, such whether contractors would be able to deliver the design, the complexity of analyses, and what can be achieved in the existing budget envelope. In terms of sample frame, access panels and random digit dialling were deemed unsuitable for the National Survey as they would not provide results based on a random sample. Therefore address-based sampling remains the preferred sampling method. The Postcode Address File (PAF) and AddressBase Plus are currently the only practical options for this. AddressBase Plus may have some advantages over the PAF in that it is updated more regularly, and that it provides better characterisation of property type (for example whether an address is a business address). Neither PAF nor AddressBase Plus include information on the name, telephone numbers or email addresses of people who could be living in the selected address. This means there are only two ways contact can be made with people at randomly selected addresses; mailcontact and/or face-to-face contact. Using a mail-only contact strategy has some advantages. The cost of the survey is reduced, CO2 emissions are reduced, and (in terms of the sampling design) there is no need for clustering in order to offset travel costs. A mail-only contact survey can be administered flexibly, which means that the sample can easily be distributed across the year, and the quarterly design can be nationally representative. Finally, a mail-only contact method could 6allow for more frequent reporting. However, the major downside to mail-only contact is a decreased response rate, and thus an increased risk of not achieving a representative sample. Having face-to-face contact will increase response rates and thus reduce the risk of nonresponse bias. If face-to-face contact methods are used there are various possible designs to consider. These are: Knock-to-nudge designs, where survey representatives visit addresses to encourage response but do not carry out in-home interviewing; Sequential CAPI, where face-to-face interviews are conducted with people who have not responded to an earlier mail invitation; and  CAPI-first designs, where all selected households are visited by an interviewer, and all are offered a face-to-face interview as the default mode of data collection. Knock-to-nudge designs offer some benefits over mail-only contact surveys, most notably the improved response rates and the increased likelihood of achieving a representative sample. However, it is unclear to what extent this method will remain in use post-pandemic now that in-home visits are permitted. There is a strong argument that once a survey representative has made contact on the doorstep the ideal option is to conduct a CAPI interview straight away, rather than risk drop-out by referring householders to a different mode of data collection.Sequential CAPI modes offer most of the benefits of CAP- only modes, with the added advantage they could offer some reductions in both cost and CO2 emissions. However, if multiple modes of contact and/or data collection are used, additional resource will be required from both the Welsh Government and contractors in terms of survey management.There will be more process documents to produce and sign off (procedures, mailings, interviewer instructions etc). Questionnaires will need to be checked and signed off for each mode of data collection. There will also be greater effort required in fieldwork monitoring, data processing and weighting. The more complicated the design the greater the resource required, from both Welsh Government and their contractors, and the higher the risk that errors could occur in practice. The mixed mode design will add some complexity to analysis.Finally, the CAPI-only mode (as used in the National Survey pre-pandemic albeit with a paper self-completion module) was considered the ‘gold-standard’ until recently due to its high response rates. However, the cost of conducting CAPI fieldwork is high compared to 7other modes of data collection. Retaining the CAPI only design would mean that reductions in interviewer mileage will not occur. If mileage reductions are not possible then CO2emissions will be more difficult to reduce. The CAPI only mode is not ‘future-proofed’ in the event of further lockdowns or if cost savings are required. In terms of data collection, the alternative modes to face-to-face interviewing under consideration are web questionnaires or telephone interviews (CATI). Both modes would be suitable for National Survey data collection. Developing a mobile-friendly National Survey web questionnaire would be a more effortful process than developing a CATI questionnaire, and in web data collection randomised within household selection methods are problematic. That said a sequential web-CAPI design would offer marginal cost savings compared to sequential CATI-CAPI design, and web designs may offer some improvement in data quality for sensitive questions compared to CATI designs. The report concludes with someindicative costs for a web-CAPI sequential design and a CATI-CAPI sequential design, for the Welsh Government to consider.

Text
review-of-options-for-the-national-survey-for-wales - Version of Record
Available under License Other.
Download (1MB)

More information

Published date: 7 April 2022

Identifiers

Local EPrints ID: 456765
URI: http://eprints.soton.ac.uk/id/eprint/456765
PURE UUID: 036899db-c2ec-4ad0-9814-fa66870af171
ORCID for Olga Maslovskaya: ORCID iD orcid.org/0000-0003-3814-810X
ORCID for Paul A. Smith: ORCID iD orcid.org/0000-0001-5337-2746

Catalogue record

Date deposited: 10 May 2022 17:08
Last modified: 11 May 2022 01:46

Export record

Contributors

Author: Peter Cornick
Author: Jo d'Ardenne
Author: Sierra Mesplie-Cowan
Author: Gerry Nicolaas
Author: Paul A. Smith ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×