The University of Southampton
University of Southampton Institutional Repository

"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system

"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system
"What’s this?": understanding user interaction behaviour with multimodal input information retrieval system
Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users’ complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems.
Association for Computing Machinery
Liu, Haiming
3ed791e3-9f1e-417e-a531-7faf19cca547
Wang, Silang
33b8214f-45b4-4b86-a367-7c77844d2c52
Kim, Hyeongcheol
dfaa7f3b-1294-45b7-be8e-ff9cc7e8d75b
Janaka, Nuwan
0e961a8a-a2f0-46af-a70e-4a9102b1c3b8
Yue, Kun
44e73de5-2a7d-4cbe-91f4-0da9dddca7ac
Nguyen, Hoang-Long
2d1c93ec-5ddc-4486-9d77-6033adfb0c68
Zhao, Shengdong
38dab0d8-7f35-4f6a-9563-71f8b00c2f03
Le, Khanh-Duy
56638272-3858-4480-9804-9387f2111fbf
Dingler, Tilman
Salim, Flora
Liu, Haiming
3ed791e3-9f1e-417e-a531-7faf19cca547
Wang, Silang
33b8214f-45b4-4b86-a367-7c77844d2c52
Kim, Hyeongcheol
dfaa7f3b-1294-45b7-be8e-ff9cc7e8d75b
Janaka, Nuwan
0e961a8a-a2f0-46af-a70e-4a9102b1c3b8
Yue, Kun
44e73de5-2a7d-4cbe-91f4-0da9dddca7ac
Nguyen, Hoang-Long
2d1c93ec-5ddc-4486-9d77-6033adfb0c68
Zhao, Shengdong
38dab0d8-7f35-4f6a-9563-71f8b00c2f03
Le, Khanh-Duy
56638272-3858-4480-9804-9387f2111fbf
Dingler, Tilman
Salim, Flora

Liu, Haiming, Wang, Silang, Kim, Hyeongcheol, Janaka, Nuwan, Yue, Kun, Nguyen, Hoang-Long, Zhao, Shengdong and Le, Khanh-Duy (2024) "What’s this?": understanding user interaction behaviour with multimodal input information retrieval system. Dingler, Tilman and Salim, Flora (eds.) In MobileHCI '24 Adjunct: Adjunct Proceedings of the 26th International Conference on Mobile Human-Computer Interaction. Association for Computing Machinery. 7 pp . (doi:10.1145/3640471.368023).

Record type: Conference or Workshop Item (Paper)

Abstract

Human communication relies on integrated multimodal channels to facilitate rich information exchange. Building on this foundation, researchers have long speculated about the potential benefits of incorporating multimodal input channels into conventional information retrieval (IR) systems to support users’ complex daily IR tasks more effectively. However, the true benefits of such integration remain uncertain. This paper presents a series of exploratory pilot tests comparing Multimodal Input IR (MIIR) with Unimodal Input IR (UIIR) across various IR scenarios, concluding that MIIR offers distinct advantages over UIIR in terms of user experiences. Our preliminary results suggest that MIIR could reduce the cognitive load associated with IR query formulation by allowing users to formulate different query-component in a unified manner across different input modalities, particularly when conducting complex exploratory search tasks in unfamiliar, in-situ contexts. The discussions stemming from this finding draw scholarly attention and suggest new angles for designing and developing MIIR systems.

This record has no associated files available for download.

More information

Published date: 21 September 2024

Identifiers

Local EPrints ID: 503504
URI: http://eprints.soton.ac.uk/id/eprint/503504
PURE UUID: b6be3ad7-89a1-468e-8022-8ba01aadbdea
ORCID for Haiming Liu: ORCID iD orcid.org/0000-0002-0390-3657

Catalogue record

Date deposited: 04 Aug 2025 16:47
Last modified: 05 Aug 2025 02:04

Export record

Altmetrics

Contributors

Author: Haiming Liu ORCID iD
Author: Silang Wang
Author: Hyeongcheol Kim
Author: Nuwan Janaka
Author: Kun Yue
Author: Hoang-Long Nguyen
Author: Shengdong Zhao
Author: Khanh-Duy Le
Editor: Tilman Dingler
Editor: Flora Salim

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of http://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×