Multimodal dance style transfer
Research output: Contribution to journal › Journal article › Research › peer-review
Documents
- Fulltext
Final published version, 2.8 MB, PDF document
This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.
Original language | English |
---|---|
Article number | 48 |
Journal | Machine Vision and Applications |
Volume | 34 |
Issue number | 4 |
Number of pages | 14 |
ISSN | 0932-8092 |
DOIs | |
Publication status | Published - 2023 |
Bibliographical note
Publisher Copyright:
© 2023, The Author(s).
- Dance motion, Generative models, Multimodal learning, Style transfer
Research areas
ID: 370580421