Review of options for the National Survey for Wales

Abstract

Until March 2020, the National Survey for Wales was a face-to-face (CAPI) survey carried out in people’s homes. However, the Covid-19 pandemic meant that alternative National Survey designs have now been trialled. The Welsh Government are considering what design should be deployed for the National Survey in the next contracting phase. This report summarises the different design options that could be used for the National Survey in the future. All the survey design options have their own strengths and weaknesses. This report summarises the various trade-offs the Welsh Government need to consider when deciding on which design to procure.A set of survey design parameters was provided by the Welsh Government. These included criteria such as random probability sampling, minimum sample sizes, questionnaire length, reporting frequency, accessibility, being able to run in under lock-down conditions, the desire to reduce CO2emissions (compared to previous years of data collection), and the facilitation of recontact research. Requirements on practicability were also included, such whether contractors would be able to deliver the design, the complexity of analyses, and what can be achieved in the existing budget envelope. In terms of sample frame, access panels and random digit dialling were deemed unsuitable for the National Survey as they would not provide results based on a random sample. Therefore address-based sampling remains the preferred sampling method. The Postcode Address File (PAF) and AddressBase Plus are currently the only practical options for this. AddressBase Plus may have some advantages over the PAF in that it is updated more regularly, and that it provides better characterisation of property type (for example whether an address is a business address). Neither PAF nor AddressBase Plus include information on the name, telephone numbers or email addresses of people who could be living in the selected address. This means there are only two ways contact can be made with people at randomly selected addresses; mailcontact and/or face-to-face contact. Using a mail-only contact strategy has some advantages. The cost of the survey is reduced, CO2 emissions are reduced, and (in terms of the sampling design) there is no need for clustering in order to offset travel costs. A mail-only contact survey can be administered flexibly, which means that the sample can easily be distributed across the year, and the quarterly design can be nationally representative. Finally, a mail-only contact method could 6allow for more frequent reporting. However, the major downside to mail-only contact is a decreased response rate, and thus an increased risk of not achieving a representative sample. Having face-to-face contact will increase response rates and thus reduce the risk of nonresponse bias. If face-to-face contact methods are used there are various possible designs to consider. These are: Knock-to-nudge designs, where survey representatives visit addresses to encourage response but do not carry out in-home interviewing; Sequential CAPI, where face-to-face interviews are conducted with people who have not responded to an earlier mail invitation; and  CAPI-first designs, where all selected households are visited by an interviewer, and all are offered a face-to-face interview as the default mode of data collection. Knock-to-nudge designs offer some benefits over mail-only contact surveys, most notably the improved response rates and the increased likelihood of achieving a representative sample. However, it is unclear to what extent this method will remain in use post-pandemic now that in-home visits are permitted. There is a strong argument that once a survey representative has made contact on the doorstep the ideal option is to conduct a CAPI interview straight away, rather than risk drop-out by referring householders to a different mode of data collection.Sequential CAPI modes offer most of the benefits of CAP- only modes, with the added advantage they could offer some reductions in both cost and CO2 emissions. However, if multiple modes of contact and/or data collection are used, additional resource will be required from both the Welsh Government and contractors in terms of survey management.There will be more process documents to produce and sign off (procedures, mailings, interviewer instructions etc). Questionnaires will need to be checked and signed off for each mode of data collection. There will also be greater effort required in fieldwork monitoring, data processing and weighting. The more complicated the design the greater the resource required, from both Welsh Government and their contractors, and the higher the risk that errors could occur in practice. The mixed mode design will add some complexity to analysis.Finally, the CAPI-only mode (as used in the National Survey pre-pandemic albeit with a paper self-completion module) was considered the ‘gold-standard’ until recently due to its high response rates. H...

More information

Identifiers

ORCID for Olga Maslovskaya: orcid.org/0000-0003-3814-810X

ORCID for Paul A. Smith: orcid.org/0000-0001-5337-2746

Catalogue record

Export record