The University of Southampton
University of Southampton Institutional Repository

MotionDuet: Dual-conditioned 3D human motion generation with video-regularized text learning

MotionDuet: Dual-conditioned 3D human motion generation with video-regularized text learning
MotionDuet: Dual-conditioned 3D human motion generation with video-regularized text learning
3D Human motion generation is pivotal across film, animation, gaming, and embodied intelligence. Traditional 3D motion synthesis relies on costly motion capture, while recent work shows that 2D videos provide rich, temporally coherent observations of human behavior. Existing approaches, however, either map high-level text descriptions to motion or rely solely on video conditioning, leaving a gap between generated dynamics and real-world motion statistics. We introduce MotionDuet, a multimodal framework that aligns motion generation with the distribution of video-derived representations. In this dual-conditioning paradigm, video cues extracted from a pretrained model (e.g., VideoMAE) ground low-level motion dynamics, while textual prompts provide semantic intent. To bridge the distribution gap across modalities, we propose Dual-stream Unified Encoding and Transformation (DUET) and a Distribution-Aware Structural Harmonization (DASH) loss. DUET fuses video-informed cues into the motion latent space via unified encoding and dynamic attention, while DASH aligns motion trajectories with both distributional and structural statistics of video features. An auto-guidance mechanism further balances textual and visual signals by leveraging a weakened copy of the model, enhancing controllability without sacrificing diversity. Extensive experiments demonstrate that MotionDuet generates realistic and controllable human motions, surpassing strong state-of-the-art baselines.
cs.GR
arXiv
Zhang, Yi-Yang
08fa6f7d-6b6a-4f4a-b49b-97f3c5dff989
Sun, Tengjiao
c5e1adca-e857-41af-939b-03a12bc57a9b
Fang, Pengcheng
7f3b5cc1-6fd3-4e94-8338-0820f3fbd189
Wang, Deng-Bao
896ffc1e-420f-46e2-ab1b-dee908aa4fa8
Cai, Xiaohao
de483445-45e9-4b21-a4e8-b0427fc72cee
Zhang, Min-Ling
dfc78617-c6bf-40d9-a643-427ba8f2c24f
Kim, Hansung
2c7c135c-f00b-4409-acb2-85b3a9e8225f
Zhang, Yi-Yang
08fa6f7d-6b6a-4f4a-b49b-97f3c5dff989
Sun, Tengjiao
c5e1adca-e857-41af-939b-03a12bc57a9b
Fang, Pengcheng
7f3b5cc1-6fd3-4e94-8338-0820f3fbd189
Wang, Deng-Bao
896ffc1e-420f-46e2-ab1b-dee908aa4fa8
Cai, Xiaohao
de483445-45e9-4b21-a4e8-b0427fc72cee
Zhang, Min-Ling
dfc78617-c6bf-40d9-a643-427ba8f2c24f
Kim, Hansung
2c7c135c-f00b-4409-acb2-85b3a9e8225f

[Unknown type: UNSPECIFIED]

Record type: UNSPECIFIED

Abstract

3D Human motion generation is pivotal across film, animation, gaming, and embodied intelligence. Traditional 3D motion synthesis relies on costly motion capture, while recent work shows that 2D videos provide rich, temporally coherent observations of human behavior. Existing approaches, however, either map high-level text descriptions to motion or rely solely on video conditioning, leaving a gap between generated dynamics and real-world motion statistics. We introduce MotionDuet, a multimodal framework that aligns motion generation with the distribution of video-derived representations. In this dual-conditioning paradigm, video cues extracted from a pretrained model (e.g., VideoMAE) ground low-level motion dynamics, while textual prompts provide semantic intent. To bridge the distribution gap across modalities, we propose Dual-stream Unified Encoding and Transformation (DUET) and a Distribution-Aware Structural Harmonization (DASH) loss. DUET fuses video-informed cues into the motion latent space via unified encoding and dynamic attention, while DASH aligns motion trajectories with both distributional and structural statistics of video features. An auto-guidance mechanism further balances textual and visual signals by leveraging a weakened copy of the model, enhancing controllability without sacrificing diversity. Extensive experiments demonstrate that MotionDuet generates realistic and controllable human motions, surpassing strong state-of-the-art baselines.

Text
2511.18209v3 - Author's Original
Download (4MB)

More information

Published date: 22 November 2025
Keywords: cs.GR

Identifiers

Local EPrints ID: 514726
URI: https://eprints.soton.ac.uk/id/eprint/514726
PURE UUID: 28e32354-0b39-4b70-a20b-e41746091718
ORCID for Pengcheng Fang: ORCID iD orcid.org/0009-0008-6215-4335
ORCID for Xiaohao Cai: ORCID iD orcid.org/0000-0003-0924-2834
ORCID for Min-Ling Zhang: ORCID iD orcid.org/0000-0003-3107-0856
ORCID for Hansung Kim: ORCID iD orcid.org/0000-0003-4907-0491

Catalogue record

Date deposited: 25 Sep 2026 16:45
Last modified: 26 Sep 2026 02:31

Export record

Altmetrics

Contributors

Author: Yi-Yang Zhang
Author: Tengjiao Sun
Author: Pengcheng Fang ORCID iD
Author: Deng-Bao Wang
Author: Xiaohao Cai ORCID iD
Author: Min-Ling Zhang ORCID iD
Author: Hansung Kim ORCID iD

Download statistics

Downloads from ePrints over the past year. Other digital versions may also be available to download e.g. from the publisher's website.

View more statistics

Atom RSS 1.0 RSS 2.0

Contact ePrints Soton: eprints@soton.ac.uk

ePrints Soton supports OAI 2.0 with a base URL of https://eprints.soton.ac.uk/cgi/oai2

This repository has been built using EPrints software, developed at the University of Southampton, but available to everyone to use.

We use cookies to ensure that we give you the best experience on our website. If you continue without changing your settings, we will assume that you are happy to receive cookies on the University of Southampton website.

×