Inter-rater reliability and content validity of network analysis as a method for measuring distributed situation awareness
Inter-rater reliability and content validity of network analysis as a method for measuring distributed situation awareness
Robust measures support the development of theory and advancement of literature and insights into the reliability and validity of methods form a crucial part of methods selection for practitioners and researchers. In order to support the selection of data collection technique to measure distributed situation awareness (DSA), the reliability and validity associated with them must be explored. This paper presents an empirical study in which the validity of two data collection techniques which are used to collect DSA data was considered. The data collection techniques are, in turn, analysed using a network analysis method and the paper, further, considered the reliability of the network analysis method. It was shown that the network analysis method provides a reliable means of assessing DSA. While low levels of hit rates were observed for both communication logs and the CDM, higher hit rates were observed for the communication logs indicating that this data collection technique is associated with slightly higher levels of validity. These findings support the human factors community in the assessment and representation of DSA in teams.
distributed situation awareness, inter-rater reliability, criterion-referenced validity, network analysis, communication, critical decision method
42-63
Sorensen, L.J.
6dcab66b-19bb-447c-a312-01a3e8a85eea
Stanton, N.A.
351a44ab-09a0-422a-a738-01df1fe0fadd
2016
Sorensen, L.J.
6dcab66b-19bb-447c-a312-01a3e8a85eea
Stanton, N.A.
351a44ab-09a0-422a-a738-01df1fe0fadd
Sorensen, L.J. and Stanton, N.A.
(2016)
Inter-rater reliability and content validity of network analysis as a method for measuring distributed situation awareness.
[in special issue: Distributed Situation Awareness]
Theoretical Issues in Ergonomics Science, 17 (1), .
(doi:10.1080/1463922X.2015.1106619).
Abstract
Robust measures support the development of theory and advancement of literature and insights into the reliability and validity of methods form a crucial part of methods selection for practitioners and researchers. In order to support the selection of data collection technique to measure distributed situation awareness (DSA), the reliability and validity associated with them must be explored. This paper presents an empirical study in which the validity of two data collection techniques which are used to collect DSA data was considered. The data collection techniques are, in turn, analysed using a network analysis method and the paper, further, considered the reliability of the network analysis method. It was shown that the network analysis method provides a reliable means of assessing DSA. While low levels of hit rates were observed for both communication logs and the CDM, higher hit rates were observed for the communication logs indicating that this data collection technique is associated with slightly higher levels of validity. These findings support the human factors community in the assessment and representation of DSA in teams.
This record has no associated files available for download.
More information
Accepted/In Press date: 7 October 2015
e-pub ahead of print date: 24 November 2015
Published date: 2016
Keywords:
distributed situation awareness, inter-rater reliability, criterion-referenced validity, network analysis, communication, critical decision method
Organisations:
Transportation Group
Identifiers
Local EPrints ID: 388108
URI: http://eprints.soton.ac.uk/id/eprint/388108
ISSN: 1464-536X
PURE UUID: ce89e466-f0fb-4caf-a413-f22635543cb8
Catalogue record
Date deposited: 19 Feb 2016 13:50
Last modified: 15 Mar 2024 03:33
Export record
Altmetrics
Contributors
Author:
L.J. Sorensen
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
