Synthesizing human motion with a global structure, such as a choreography, is a challenging task. Existing methods tend to concentrate on local smooth pose transitions and neglect the global context or the theme of the motion. In this work, we present a music-driven motion synthesis framework that generates long-term sequences of human motions which are synchronized with the input beats, and jointly form a global structure that respects a specific dance genre. In addition, our framework enables generation of diverse motions that are controlled by the content of the music, and not only by the beat. Our music-driven dance synthesis framework is a hierarchical system that consists of three levels: pose , motif , and choreography . The pose level consists of an LSTM component that generates temporally coherent sequences of poses. The motif level guides sets of consecutive poses to form a movement that belongs to a specific distribution using a novel motion perceptual-loss . And the choreography level selects the order of the performed movements and drives the system to follow the global structure of a dance genre. Our results demonstrate the effectiveness of our music-driven framework to generate natural and consistent movements on various dance types, having control over the content of the synthesized motions, and respecting the overall structure of the dance.

We introduce MotioNet , a deep neural network that directly reconstructs the motion of a 3D human skeleton from a monocular video. While previous methods rely on either rigging or inverse kinematics (IK) to associate a consistent skeleton with temporally coherent joint rotations, our method is the first data-driven approach that directly outputs a kinematic skeleton, which is a complete, commonly used motion representation. At the crux of our approach lies a deep neural network with embedded kinematic priors, which decomposes sequences of 2D joint positions into two separate attributes: a single, symmetric skeleton encoded by bone lengths, and a sequence of 3D joint rotations associated with global root positions and foot contact labels. These attributes are fed into an integrated forward kinematics (FK) layer that outputs 3D positions, which are compared to a ground truth. In addition, an adversarial loss is applied to the velocities of the recovered rotations to ensure that they lie on the manifold of natural joint rotations. The key advantage of our approach is that it learns to infer natural joint rotations directly from the training data rather than assuming an underlying model, or inferring them from joint positions using a data-agnostic IK solver. We show that enforcing a single consistent skeleton along with temporally coherent joint rotations constrains the solution space, leading to a more robust handling of self-occlusions and depth ambiguities.

Motion capture (mocap) technology is an efficient method for digitizing art performances, and is becoming increasingly popular in the preservation and dissemination of dance performances. Although technically the captured data can be of very high quality, dancing allows stylistic variations and improvisations that cannot be easily identified. The majority of motion analysis algorithms are based on ad-hoc quantitative metrics, thus do not usually provide insights on style qualities of a performance. In this work, we present a framework based on the principles of Laban Movement Analysis (LMA) that aims to identify style qualities in dance motions. The proposed algorithm uses a feature space that aims to capture the four LMA components (B ody , E ffort , S hape , S pace ), and can be subsequently used for motion comparison and evaluation. We have designed and implemented a prototype virtual reality simulator for teaching folk dances in which users can preview dance segments performed by a 3D avatar and repeat them. The user’s movements are captured and compared to the folk dance template motions; then, intuitive feedback is provided to the user based on the LMA components. The results demonstrate the effectiveness of our system, opening new horizons for automatic motion and dance evaluation processes.