The University of Southampton
University of Southampton Institutional Repository

Impact of question order on prioritisation of outcomes in the development of a core outcome set: A randomised controlled trial

Impact of question order on prioritisation of outcomes in the development of a core outcome set: A randomised controlled trial
Impact of question order on prioritisation of outcomes in the development of a core outcome set: A randomised controlled trial

Background: Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery. Methods: A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey. Results: In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]). Conclusions: In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias. Trial registration: The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.

Context effects, Core outcome set, Delphi, Question order
1745-6215
Brookes, Sara T.
ffd7dc65-ac9a-456e-bc9c-dfee06239835
Chalmers, Katy A.
1976d813-3842-410e-9621-f104cf62efd3
Avery, Kerry N.L.
052f5698-3c59-492c-8e80-aa0132bce5ff
Coulman, Karen
b59a07f7-cfd1-43b3-8e52-0e8fd97cdf89
Blazeby, Jane M.
689d490e-fca3-4430-88de-f19ec6cebf58
Alderson, Derek
5e5c1e02-05d6-4634-aa0b-0c03c13e58b3
Whiting, John
678c4dd4-51c3-41eb-ad9e-ad26e6554930
Alkhaffaf, Bilal
2065b48d-d01a-4b62-b98b-e5351cb41664
Allum, William
19a9b8a7-e082-4450-9703-ca80b5456087
Attwood, Stephen
5ea8ff54-04c5-4cbf-a718-90742922b665
Barr, Hugh
d9616fd6-936c-40c5-a4b8-ec37ae3d3bde
Hewin, David
1f215154-722b-4d7f-b1aa-30c3fb578eff
Higgs, Simon
70a12c6f-ec8d-40ae-840d-10ca6c58c1e5
Vipond, Mark
7c16748e-c6f8-49e3-92c1-d9c448d37f3e
Batiwalla, Issy
d2207c20-583e-4939-8d53-c02a196c0ab7
Blackshaw, Guy
fffd09e7-4381-4092-8a69-6ad3ae70e310
Bolter, Marilyn
4046d07c-a6f4-43a3-b033-ffb801a4fc87
Goulding, Claire
4ce6e454-938c-4f42-9a9b-408511bc6e2c
Berrisford, Richard
b160bfa3-4287-4aed-9384-1b2a404bf9cd
Sanders, Grant
39fea4d6-1cad-4e2f-a2a7-d5bf8c162319
Wheatley, Tim
1684b26d-3c10-4ecf-9f37-903c3106c7ae
Botha, Abrie
054267a5-3dfd-4c63-acdc-501c1f2b694b
Mason, Robert
a3bbcfc3-f586-416a-a571-f14b97475608
Byrne, Jim
fcde4078-a0e2-4980-8894-494b99f672d0
Kelly, Jamie
c8b841fe-7134-4332-b453-5dc8c76058a5
Callan, Joanne
53c42f2d-8d70-4985-bc3e-fb29beb89715
Couper, Graeme
dd579800-af18-4f8b-8d3d-7a2b7c566853
Deans, Chris
8d0a48be-1f9b-4962-bf86-a76a5ecd8c8e
Paterson-Brown, Simon
24f09586-6849-4f98-b145-a44cf1fde822
Dawas, Khaled
5b801d69-8f81-4b6e-8ca9-cc45d9a038ba
Mughal, Muntzer
f42b5a05-8a38-4e4a-a26e-d0b707978ca3
Galloway, Simon
49ca3331-c275-4b98-815e-754bb77c1a11
George, Michelle
eb16aaec-0dd2-4063-ba87-ece86ba82887
Gokhale, Jay
a78cfeb0-1a19-4739-a969-00ef86ef3623
Goodman, Mike
983ceeef-e532-4d07-a5be-ddefd7789838
Hardwick, Richard
f145c324-1a1e-468b-a4b7-78be6c27d648
Hassn, Ahmed
9f32324c-8792-473c-b237-26434ba197cb
Henwood, Mark
5b7d392d-13c1-4c2b-ba15-1f8e355d6549
Kryzstopik, Richard
cd4d2a56-4860-4180-9c0e-016ce435b8e2
Lewis, Michael
f046ffc3-6d03-4a51-9c3d-fa8fa6a01473
Tallett, Jane
b1dacbad-e3ee-4e5f-afa0-d0710f3d5bbb
MacKay, Colin
26429e87-8092-41e7-9a64-84878e40701f
Manson, James
993b8267-60b6-453d-ad53-a51433601dab
Moxon, Ruth
5c43b93f-7fa2-4f52-a069-cb05b72528e4
Osborne, Sue
7dabccef-9d9f-4483-9d39-b8f2ecc04fe1
Page, Richard
481c2cb1-6c0e-43b4-8866-a505e60a94ca
Shackcloth, Mike
63b9ca8b-e788-479a-8667-7943a9e00aa7
Parameswaran, Raj
e3f26db0-720b-419a-9d3a-785114d0933d
Parsons, Simon
979d203f-915a-42f6-a6d7-3a5edaf834c6
Underwood, Tim
8e81bf60-edd2-4b0e-8324-3068c95ea1c6
on behalf of the ROMIO study group
Brookes, Sara T.
ffd7dc65-ac9a-456e-bc9c-dfee06239835
Chalmers, Katy A.
1976d813-3842-410e-9621-f104cf62efd3
Avery, Kerry N.L.
052f5698-3c59-492c-8e80-aa0132bce5ff
Coulman, Karen
b59a07f7-cfd1-43b3-8e52-0e8fd97cdf89
Blazeby, Jane M.
689d490e-fca3-4430-88de-f19ec6cebf58
Alderson, Derek
5e5c1e02-05d6-4634-aa0b-0c03c13e58b3
Whiting, John
678c4dd4-51c3-41eb-ad9e-ad26e6554930
Alkhaffaf, Bilal
2065b48d-d01a-4b62-b98b-e5351cb41664
Allum, William
19a9b8a7-e082-4450-9703-ca80b5456087
Attwood, Stephen
5ea8ff54-04c5-4cbf-a718-90742922b665
Barr, Hugh
d9616fd6-936c-40c5-a4b8-ec37ae3d3bde
Hewin, David
1f215154-722b-4d7f-b1aa-30c3fb578eff
Higgs, Simon
70a12c6f-ec8d-40ae-840d-10ca6c58c1e5
Vipond, Mark
7c16748e-c6f8-49e3-92c1-d9c448d37f3e
Batiwalla, Issy
d2207c20-583e-4939-8d53-c02a196c0ab7
Blackshaw, Guy
fffd09e7-4381-4092-8a69-6ad3ae70e310
Bolter, Marilyn
4046d07c-a6f4-43a3-b033-ffb801a4fc87
Goulding, Claire
4ce6e454-938c-4f42-9a9b-408511bc6e2c
Berrisford, Richard
b160bfa3-4287-4aed-9384-1b2a404bf9cd
Sanders, Grant
39fea4d6-1cad-4e2f-a2a7-d5bf8c162319
Wheatley, Tim
1684b26d-3c10-4ecf-9f37-903c3106c7ae
Botha, Abrie
054267a5-3dfd-4c63-acdc-501c1f2b694b
Mason, Robert
a3bbcfc3-f586-416a-a571-f14b97475608
Byrne, Jim
fcde4078-a0e2-4980-8894-494b99f672d0
Kelly, Jamie
c8b841fe-7134-4332-b453-5dc8c76058a5
Callan, Joanne
53c42f2d-8d70-4985-bc3e-fb29beb89715
Couper, Graeme
dd579800-af18-4f8b-8d3d-7a2b7c566853
Deans, Chris
8d0a48be-1f9b-4962-bf86-a76a5ecd8c8e
Paterson-Brown, Simon
24f09586-6849-4f98-b145-a44cf1fde822
Dawas, Khaled
5b801d69-8f81-4b6e-8ca9-cc45d9a038ba
Mughal, Muntzer
f42b5a05-8a38-4e4a-a26e-d0b707978ca3
Galloway, Simon
49ca3331-c275-4b98-815e-754bb77c1a11
George, Michelle
eb16aaec-0dd2-4063-ba87-ece86ba82887
Gokhale, Jay
a78cfeb0-1a19-4739-a969-00ef86ef3623
Goodman, Mike
983ceeef-e532-4d07-a5be-ddefd7789838
Hardwick, Richard
f145c324-1a1e-468b-a4b7-78be6c27d648
Hassn, Ahmed
9f32324c-8792-473c-b237-26434ba197cb
Henwood, Mark
5b7d392d-13c1-4c2b-ba15-1f8e355d6549
Kryzstopik, Richard
cd4d2a56-4860-4180-9c0e-016ce435b8e2
Lewis, Michael
f046ffc3-6d03-4a51-9c3d-fa8fa6a01473
Tallett, Jane
b1dacbad-e3ee-4e5f-afa0-d0710f3d5bbb
MacKay, Colin
26429e87-8092-41e7-9a64-84878e40701f
Manson, James
993b8267-60b6-453d-ad53-a51433601dab
Moxon, Ruth
5c43b93f-7fa2-4f52-a069-cb05b72528e4
Osborne, Sue
7dabccef-9d9f-4483-9d39-b8f2ecc04fe1
Page, Richard
481c2cb1-6c0e-43b4-8866-a505e60a94ca
Shackcloth, Mike
63b9ca8b-e788-479a-8667-7943a9e00aa7
Parameswaran, Raj
e3f26db0-720b-419a-9d3a-785114d0933d
Parsons, Simon
979d203f-915a-42f6-a6d7-3a5edaf834c6
Underwood, Tim
8e81bf60-edd2-4b0e-8324-3068c95ea1c6

Brookes, Sara T., Chalmers, Katy A., Avery, Kerry N.L., Coulman, Karen, Blazeby, Jane M., Alderson, Derek, Whiting, John, Alkhaffaf, Bilal, Allum, William, Attwood, Stephen, Barr, Hugh, Hewin, David, Higgs, Simon, Vipond, Mark, Batiwalla, Issy, Blackshaw, Guy, Bolter, Marilyn, Goulding, Claire, Berrisford, Richard, Sanders, Grant, Wheatley, Tim, Botha, Abrie, Mason, Robert, Byrne, Jim, Kelly, Jamie, Callan, Joanne, Couper, Graeme, Deans, Chris, Paterson-Brown, Simon, Dawas, Khaled, Mughal, Muntzer, Galloway, Simon, George, Michelle, Gokhale, Jay, Goodman, Mike, Hardwick, Richard, Hassn, Ahmed, Henwood, Mark, Kryzstopik, Richard, Lewis, Michael, Tallett, Jane, MacKay, Colin, Manson, James, Moxon, Ruth, Osborne, Sue, Page, Richard, Shackcloth, Mike, Parameswaran, Raj, Parsons, Simon and Underwood, Tim , on behalf of the ROMIO study group (2018) Impact of question order on prioritisation of outcomes in the development of a core outcome set: A randomised controlled trial. Trials, 19 (66). (doi:10.1186/s13063-017-2405-6).

Record type: Article

Abstract

Background: Core outcome set (COS) developers increasingly employ Delphi surveys to elicit stakeholders' opinions of which outcomes to measure and report in trials of a particular condition or intervention. Research outside of Delphi surveys and COS development demonstrates that question order can affect response rates and lead to 'context effects', where prior questions determine an item's meaning and influence responses. This study examined the impact of question order within a Delphi survey for a COS for oesophageal cancer surgery. Methods: A randomised controlled trial was nested within the Delphi survey. Patients and health professionals were randomised to receive a survey including clinical and patient-reported outcomes (PROs), where the PRO section appeared first or last. Participants rated (1-9) the importance of 68 items for inclusion in a COS (ratings 7-9 considered 'essential'). Analyses considered the impact of question order on: (1) survey response rates; (2) participants' responses; and (3) items retained at end of the survey. Results: In total, 116 patients and 71 professionals returned completed surveys. Question order did not affect response rates among patients, but fewer professionals responded when clinical items appeared first (difference = 31.3%, 95% confidence interval [CI] = 13.6-48.9%, P = 0.001). Question order led to different context effects within patients and professionals. While patients rated clinical items highly, irrespective of question order, more PROs were rated essential when appearing last rather than first (difference = 23.7%, 95% CI = 10.5-40.8%). Among professionals, the greatest impact was on clinical items; a higher percentage rated essential when appearing last (difference = 11.6%, 95% CI = 0.0-23.3%). An interaction between question order and the percentage of PRO/clinical items rated essential was observed for patients (P = 0.025) but not professionals (P = 0.357). Items retained for further consideration at the end of the survey were dependent on question order, with discordant items (retained by one question order group only) observed in patients (18/68 [26%]) and professionals (20/68 [29%]). Conclusions: In the development of a COS, participants' ratings of potential outcomes within a Delphi survey depend on the context (order) in which the outcomes are asked, consequently impacting on the final COS. Initial piloting is recommended with consideration of the randomisation of items in the survey to reduce potential bias. Trial registration: The randomised controlled trial reported within this paper was nested within the development of a core outcome set to investigate processes in core outcome set development. Outcomes were not health-related and trial registration was not therefore applicable.

Text
document-SB - Version of Record
Available under License Creative Commons Attribution.
Download (715kB)

More information

Accepted/In Press date: 13 December 2017
e-pub ahead of print date: 25 January 2018
Published date: 25 January 2018
Keywords: Context effects, Core outcome set, Delphi, Question order

Identifiers

Local EPrints ID: 417979
URI: https://eprints.soton.ac.uk/id/eprint/417979
ISSN: 1745-6215
PURE UUID: 996d668d-abcd-48cb-b19b-8493dc5e153b
ORCID for Tim Underwood: ORCID iD orcid.org/0000-0001-9455-2188

Catalogue record

Date deposited: 20 Feb 2018 17:30
Last modified: 14 Mar 2019 01:44

Export record

Altmetrics

Contributors

Author: Sara T. Brookes
Author: Katy A. Chalmers
Author: Kerry N.L. Avery
Author: Karen Coulman
Author: Jane M. Blazeby
Author: Derek Alderson
Author: John Whiting
Author: Bilal Alkhaffaf
Author: William Allum
Author: Stephen Attwood
Author: Hugh Barr
Author: David Hewin
Author: Simon Higgs
Author: Mark Vipond
Author: Issy Batiwalla
Author: Guy Blackshaw
Author: Marilyn Bolter
Author: Claire Goulding
Author: Richard Berrisford
Author: Grant Sanders
Author: Tim Wheatley
Author: Abrie Botha
Author: Robert Mason
Author: Jim Byrne
Author: Jamie Kelly
Author: Joanne Callan
Author: Graeme Couper
Author: Chris Deans
Author: Simon Paterson-Brown
Author: Khaled Dawas
Author: Muntzer Mughal
Author: Simon Galloway
Author: Michelle George
Author: Jay Gokhale
Author: Mike Goodman
Author: Richard Hardwick
Author: Ahmed Hassn
Author: Mark Henwood
Author: Richard Kryzstopik
Author: Michael Lewis
Author: Jane Tallett
Author: Colin MacKay
Author: James Manson
Author: Ruth Moxon
Author: Sue Osborne
Author: Richard Page
Author: Mike Shackcloth
Author: Raj Parameswaran
Author: Simon Parsons
Author: Tim Underwood ORCID iD

University divisions

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×