SYNERGY: an LLM-based system for smart home device control and user social interaction

Abstract

Socially assistive robots and smart home devices are increasingly integrated into daily life, offering emotional and social support in a modern society where many individuals live alone. Inspired by this context, this paper presents the SYNERGY framework for managing smart home environments comprising multiple smart devices and assistive robots. SYNERGY is designed to handle a broad spectrum of user requests, from simple queries to complex tasks requiring multi-step reasoning, and assigns them to appropriate agents for optimal execution. We conducted a series of experiments under various configurations to evaluate the framework's performance in processing user queries across socially relevant contexts and its effectiveness in task allocation. Experimental results demonstrate that incorporating contextual retrieval into our designed decision-making module significantly improves the system’s understanding of user intent and is crucial in handling complex, multi-step tasks. Additionally, the designed task allocation module proves its effectiveness in optimizing assignments using cost matrices, enabling flexible and efficient multi-agent coordination.

More information

Identifiers

ORCID for Nguyen Tan Viet Tuyen: orcid.org/0000-0001-8000-6485

Catalogue record

Export record