Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation
Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation
In this paper we illustrate how novel AI methods can improve the performance of intelligence analysts. These analysts aim to make sense of — often conflicting or incomplete — information, weighing up competing hypotheses which serve to explain an observed situation. Analysts have access to numerous visual analytic tools which support the temporal and/or conceptual structuring of information and collection, and support the evaluation of alternative hypotheses. We believe, however, that there are currently no tools or methods which allow analysts to combine the recording and interpretation of information, and that there is little understanding about how software tools can facilitate the hypothesis formation process. Following the identification of these requirements, we developed the CISpaces (Collaborative Intelligence Spaces) decision support tool in collaboration with professional intelligence analysts. CISpaces combines multiple AI-based methods including argumentation theory, crowdsourced Bayesian analysis, and provenance recording. We show that CISpaces is able to provide support to analysts by facilitating the interpretation of different types of evidence through argumentation-based reasoning, provenance analysis and crowdsourcing. We undertook an experimental analysis with intelligence analysts which highlights three key points. (1) The novel, principled AI methods implemented in CISpaces advance performance in intelligence analysis. (2) While designed as a research prototype, analysts benchmarked it against their existing software tools, and we provide results suggesting intention to adopt CISpaces in analysts’ daily activities. (3) Finally, the evaluation highlights some drawbacks in CISpaces. However, these are not due to the technologies underpinning the tool, but rather in its lack of integration with existing organisational standards regarding input and output formats. Our evaluation with intelligence analysts therefore demonstrates the potential impact that an integrated tool building on state-of-the-art AI techniques can have on the process of understanding complex situations, and on how such a tool can help focus human effort on identifying more credible interpretations of evidence.
Argumentation, Provenance, Crowd-sourcing, Intelligence analysis, human machine interface
Toniolo, Alice
e54ad578-9232-471a-a5d7-cd3a7bc70872
Cerutti, Federico
fec75499-632a-460f-987a-1a09420d8cb1
Norman, Timothy
663e522f-807c-4569-9201-dc141c8eb50d
Oren, Nir
00646ccd-977b-4442-88c7-d18089b26670
Allen, John A.
15552530-a660-4577-9a3a-f2f6857f447e
Srivastava, Mani
77c0ad90-6073-4dd8-908f-05087d856cd6
Sullivan, Paul
b406885e-cc88-4b09-85bd-d29bd41d3da7
February 2023
Toniolo, Alice
e54ad578-9232-471a-a5d7-cd3a7bc70872
Cerutti, Federico
fec75499-632a-460f-987a-1a09420d8cb1
Norman, Timothy
663e522f-807c-4569-9201-dc141c8eb50d
Oren, Nir
00646ccd-977b-4442-88c7-d18089b26670
Allen, John A.
15552530-a660-4577-9a3a-f2f6857f447e
Srivastava, Mani
77c0ad90-6073-4dd8-908f-05087d856cd6
Sullivan, Paul
b406885e-cc88-4b09-85bd-d29bd41d3da7
Toniolo, Alice, Cerutti, Federico, Norman, Timothy, Oren, Nir, Allen, John A., Srivastava, Mani and Sullivan, Paul
(2023)
Human-Machine Collaboration in Intelligence Analysis: An Expert Evaluation.
Intelligent Systems with Applications, 17, [200151].
(doi:10.1016/j.iswa.2022.200151).
Abstract
In this paper we illustrate how novel AI methods can improve the performance of intelligence analysts. These analysts aim to make sense of — often conflicting or incomplete — information, weighing up competing hypotheses which serve to explain an observed situation. Analysts have access to numerous visual analytic tools which support the temporal and/or conceptual structuring of information and collection, and support the evaluation of alternative hypotheses. We believe, however, that there are currently no tools or methods which allow analysts to combine the recording and interpretation of information, and that there is little understanding about how software tools can facilitate the hypothesis formation process. Following the identification of these requirements, we developed the CISpaces (Collaborative Intelligence Spaces) decision support tool in collaboration with professional intelligence analysts. CISpaces combines multiple AI-based methods including argumentation theory, crowdsourced Bayesian analysis, and provenance recording. We show that CISpaces is able to provide support to analysts by facilitating the interpretation of different types of evidence through argumentation-based reasoning, provenance analysis and crowdsourcing. We undertook an experimental analysis with intelligence analysts which highlights three key points. (1) The novel, principled AI methods implemented in CISpaces advance performance in intelligence analysis. (2) While designed as a research prototype, analysts benchmarked it against their existing software tools, and we provide results suggesting intention to adopt CISpaces in analysts’ daily activities. (3) Finally, the evaluation highlights some drawbacks in CISpaces. However, these are not due to the technologies underpinning the tool, but rather in its lack of integration with existing organisational standards regarding input and output formats. Our evaluation with intelligence analysts therefore demonstrates the potential impact that an integrated tool building on state-of-the-art AI techniques can have on the process of understanding complex situations, and on how such a tool can help focus human effort on identifying more credible interpretations of evidence.
Text
Human-MachineCollaborationInIntelligenceAnalysis
- Accepted Manuscript
Text
1-s2.0-S2667305322000886-main
- Version of Record
More information
e-pub ahead of print date: 15 November 2022
Published date: February 2023
Additional Information:
Funding Information:
We would particularly like to acknowledge the contribution made by the late Paul Sullivan to this work. Without his expertise, this research would not have been possible. We would like to thank the professional analysts from the UK, US and international agencies for their support in developing this research. This research was sponsored by the U.S. Army Research Laboratory and the U.K. Ministry of Defence and was accomplished under Agreement Number W911NF-06-3-0001. The views and conclusions contained in this document are those of the author(s) and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Army Research Laboratory, the U.S. Government, the U.K. Ministry of Defence or the U.K. Government. The U.S. and U.K. Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.
Funding Information:
This research was sponsored by the U.S. Army Research Laboratory and the U.K. Ministry of Defence and was accomplished under Agreement Number W911NF-06-3-0001. The views and conclusions contained in this document are those of the author(s) and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Army Research Laboratory, the U.S. Government, the U.K. Ministry of Defence or the U.K. Government. The U.S. and U.K. Governments are authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation hereon.
Publisher Copyright:
© 2022
Keywords:
Argumentation, Provenance, Crowd-sourcing, Intelligence analysis, human machine interface
Identifiers
Local EPrints ID: 472703
URI: http://eprints.soton.ac.uk/id/eprint/472703
ISSN: 2667-3053
PURE UUID: 4e7593e0-1aa7-428f-8ecf-febc17fdb4ee
Catalogue record
Date deposited: 15 Dec 2022 17:31
Last modified: 17 Mar 2024 03:41
Export record
Altmetrics
Contributors
Author:
Alice Toniolo
Author:
Federico Cerutti
Author:
Nir Oren
Author:
John A. Allen
Author:
Mani Srivastava
Author:
Paul Sullivan
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics
