"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system
"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system
Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users’ complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems.
Association for Computing Machinery
Liu, Haiming
3ed791e3-9f1e-417e-a531-7faf19cca547
Wang, Silang
33b8214f-45b4-4b86-a367-7c77844d2c52
Kim, Hyeongcheol
dfaa7f3b-1294-45b7-be8e-ff9cc7e8d75b
Janaka, Nuwan
0e961a8a-a2f0-46af-a70e-4a9102b1c3b8
Yue, Kun
44e73de5-2a7d-4cbe-91f4-0da9dddca7ac
Nguyen, Hoang-Long
2d1c93ec-5ddc-4486-9d77-6033adfb0c68
Zhao, Shengdong
38dab0d8-7f35-4f6a-9563-71f8b00c2f03
Le, Khanh-Duy
56638272-3858-4480-9804-9387f2111fbf
21 September 2024
Liu, Haiming
3ed791e3-9f1e-417e-a531-7faf19cca547
Wang, Silang
33b8214f-45b4-4b86-a367-7c77844d2c52
Kim, Hyeongcheol
dfaa7f3b-1294-45b7-be8e-ff9cc7e8d75b
Janaka, Nuwan
0e961a8a-a2f0-46af-a70e-4a9102b1c3b8
Yue, Kun
44e73de5-2a7d-4cbe-91f4-0da9dddca7ac
Nguyen, Hoang-Long
2d1c93ec-5ddc-4486-9d77-6033adfb0c68
Zhao, Shengdong
38dab0d8-7f35-4f6a-9563-71f8b00c2f03
Le, Khanh-Duy
56638272-3858-4480-9804-9387f2111fbf
Liu, Haiming, Wang, Silang, Kim, Hyeongcheol, Janaka, Nuwan, Yue, Kun, Nguyen, Hoang-Long, Zhao, Shengdong and Le, Khanh-Duy
(2024)
"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system.
Dingler, Tilman and Salim, Flora
(eds.)
In MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction.
Association for Computing Machinery.
7 pp
.
(doi:10.1145/3640471.368023).
Record type:
Conference or Workshop Item
(Paper)
Abstract
Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users’ complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems.
This record has no associated files available for download.
More information
Published date: 21 September 2024
Identifiers
Local EPrints ID: 503504
URI: http://eprints.soton.ac.uk/id/eprint/503504
PURE UUID: b6be3ad7-89a1-468e-8022-8ba01aadbdea
Catalogue record
Date deposited: 04 Aug 2025 16:47
Last modified: 05 Aug 2025 02:04
Export record
Altmetrics
Contributors
Author:
Haiming Liu
Author:
Silang Wang
Author:
Hyeongcheol Kim
Author:
Nuwan Janaka
Author:
Kun Yue
Author:
Hoang-Long Nguyen
Author:
Shengdong Zhao
Author:
Khanh-Duy Le
Editor:
Tilman Dingler
Editor:
Flora Salim
Download statistics
Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.
View more statistics