This article reports on an interdisciplinary research project on movement sonification for sensori-motor learning. First, we describe different research fields which have contributed to movement sonification, from music technology including gesture-controlled sound synthesis, sonic interaction design, to research on sensori-motor learning with auditory-feedback. In particular, we propose to distinguish between sound-oriented tasks and movement-oriented tasks in experiment involving interactive sound feedback. We describe several research questions and recently published results on movement control, learning and perception. In particular, we studied the effect of the auditory feedback on movements considering several cases: from experiments on pointing and visuo-motor tracking to more complex tasks where interactive sound feedback can guide movements, or cases of sensory substitution where the auditory feedback can inform on object shapes. We also developed specific methodologies and technologies for designing the sonic feedback and movement sonification. We conclude with a discussion on key future research challenges in sensori-motor learning with movement sonification. We also point out towards promising applications such as rehabilitation, sport training or product design.

Studies of the nature of the neural mechanisms involved in goal-directed movements tend to concentrate on the role of vision. We present here an attempt to address the mechanisms whereby an auditory input is transformed into a motor command. The spatial and temporal organization of hand movements were studied in normal human subjects as they pointed towards unseen auditory targets located in a horizontal plane in front of them. Positions and movements of the hand were measured by a six infrared camera tracking system. In one condition, we assessed the role of auditory information about target position in correcting the trajectory of the hand. To accomplish this, the duration of the target presentation was varied. In another condition, subjects received continuous auditory feedback of their hand movement while pointing to the auditory targets. Online auditory control of the direction of pointing movements was assessed by evaluating how subjects reacted to shifts in heard hand position. Localization errors were exacerbated by short duration of target presentation but not modified by auditory feedback of hand position. Long duration of target presentation gave rise to a higher level of accuracy and was accompanied by early automatic head orienting movements consistently related to target direction. These results highlight the efficiency of auditory feedback processing in online motor control and suggest that the auditory system takes advantages of dynamic changes of the acoustic cues due to changes in head orientation in order to process online motor control. How to design an informative acoustic feedback needs to be carefully studied to demonstrate that auditory feedback of the hand could assist the monitoring of movements directed at objects in auditory space.

We introduce a robust method to recover optical absorption, reduced scattering, and single-scattering asymmetry coefficients (μ_a, ${\mathrm{\mu }}_s^{\prime }$, g ₁) of infinite turbid media over a range of (${\mathrm{\mu }}_s^{\prime }$/μ_a) spanning 3 orders of magnitude. This is accomplished through the spatially resolved measurement of irradiance at source-detector separations spanning 0.25–8 transport mean free paths (l*). These measurements are rapidly processed by a multistaged nonlinear optimization algorithm in which the measured irradiances are compared with predictions given by the δ-P ₁ variant of the diffusion approximation to the Boltzmann transport equation. The ability of the δ-P ₁ model to accurately describe radiative transport within media of arbitrary albedo and on spatial scales comparable to l* is the key element enabling the separation of g ₁ from ${\mathrm{\mu }}_s^{\prime }$.

Anticipatory behaviours are known to occur in music performance, notably on the control movements of instruments such as piano or drums. We studied such effects on bowed string movements, corresponding to a case where the control on sound is continuous. Movements were measured with an optical motion capture system combined with sensors on the bow. Bowing movements were analysed and compared on the basis of underlying effort costs, determined from their velocity profiles. Precisely, we used movement models that assume that jerk or impulse are minimized. These models were synthesized based on measurement data and then compared to velocity and acceleration profiles. Results on various musical cases involving separate strokes, scales, mixed bowing techniques and rhythms showed that this methodology can account, to some extent, for the different effort strategies used by the players. The presented modelling provides evidence of anticipatory behaviour during bowing movements.

Many technologies for promoting physical activity (PA) give limited importance to critical variables for engagement in PA, such as negative body perceptions. Here, we aim to address this gap by incorporating barriers and experienced body sensations into the design process for wearables and body-based devices thus expanding the design space for such technologies. We first report four co-design workshops with physically inactive participants (n=9); in these workshops, we explored tangible tools (i) to sensitize people to body sensations experienced when facing barriers to PA and (ii) to ideate how haptic and auditory feedback could transform body sensations to increase body-awareness and engagement in PA; results show several interactive sensorial patterns with potential to inform the design of body transformation wearables. These were validated through a follow-up workshop (n=13 participants) and reflections based on insights from a literature review. Our findings are significant for the design of ubiquitous technology to support and initiate PA in everyday contexts through a novel approach of transforming body perceptions/ sensations related to PA barriers. We contribute design inspirations and cards for identifying barriers to PA and to empower designers and researchers to integrate them early in the design process.