Multimodal dance style transfer

Research output: Contribution to journalJournal articleResearchpeer-review

Documents

  • Fulltext

    Final published version, 2.8 MB, PDF document

  • Wenjie Yin
  • Yin, Hang
  • Kim Baraka
  • Danica Kragic
  • Mårten Björkman

This paper first presents CycleDance, a novel dance style transfer system that transforms an existing motion clip in one dance style into a motion clip in another dance style while attempting to preserve the motion context of the dance. CycleDance extends existing CycleGAN architectures with multimodal transformer encoders to account for the music context. We adopt a sequence length-based curriculum learning strategy to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. Building upon CycleDance, we further propose StarDance, which enables many-to-many mappings across different styles using a single generator network. Additionally, we introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. To evaluate the performance of our approach, we perform an extensive ablation study and a human study with 30 participants, each with 5 or more years of dance experience. Our experimental results show that our approach can generate realistic movements with the target style, outperforming the baseline CycleGAN and its variants on naturalness, transfer strength, and content preservation. Our proposed approach has potential applications in choreography, gaming, animation, and tool development for artistic and scientific innovations in the field of dance.

Original languageEnglish
Article number48
JournalMachine Vision and Applications
Volume34
Issue number4
Number of pages14
ISSN0932-8092
DOIs
Publication statusPublished - 2023

Bibliographical note

Publisher Copyright:
© 2023, The Author(s).

    Research areas

  • Dance motion, Generative models, Multimodal learning, Style transfer

ID: 370580421