While recent studies have claimed that non-referential gestures (i.e., gestures that do not visually represent any semantic content in speech) are used to mark discourse-new and/or -accessible referents and focused information in adult speech, to our knowledge, no prior investigation has studied the relationship between information structure (IS) and gesture referentiality in children’s narrative speech from a developmental perspective. A longitudinal database consisting of 332 narratives performed by 83 children at two different time points in development was coded for IS and gesture referentiality (i.e., referential and non-referential gestures). Results revealed that at both time points, both referential and non-referential gestures were produced more with information that moves discourse forward (i.e., focus) and predication (i.e., comment) rather than topical or background information. Further, at 7–9 years of age, children tended to use more non-referential gestures to mark focus and comment constituents than referential gestures. In terms of the marking of the newness of discourse referents, non-referential gestures already seem to play a key role at 5–6 years old, whereas referential gestures did not show any patterns. This relationship was even stronger at 7–9 years old. All in all, our findings offer supporting evidence that in contrast with referential gestures, non-referential gestures have been found to play a key role in marking IS, and that the development of this relationship solidifies at a period in development that coincides with a spurt in non-referential gesture production.
Communication is multimodal in that speakers use not only their voices, but also co-speech gestures to communicate. Recent insights suggest that gestural behavior has a strong association with prosodic structure and that a single gesture can communicate various semantic and pragmatic meanings. This highlights the importance of developing a comprehensive, flexible, and transparent approach to gesture annotation that accounts for multiple dimensions of gesture, including a gesture’s form, prosodic properties, and semantic and pragmatic contributions. To address this need for an increasingly dimensionalized approach to multimodal data annotation, the main goal of this paper is to present and describe a novel labeling system for manual gestures. The MultiModal MultiDimensional (M3D) system consists of an open access package that has been developed in coordination with five different labs working on gesture and its interaction with speech. The package includes a set of reliable annotation guidelines, a validated training program, and two annotated audiovisual corpora that represent over 60 minutes of lecture-style speech.
7 March 2023 in Journal of Speech, Language, and Hearing Research
Purpose: This study aims to analyze the development of gesture–speech temporal alignment patterns in children's narrative speech from a longitudinal perspective and, specifically, the potential differences between different gesture types, namely, gestures that imagistically portray or refer to semantic content in speech (i.e., referential gestures) and those that lack semantic content (i.e., non-referential gestures). Method: This study uses an audiovisual corpus of narrative productions ( n = 332) from 83 children (43 girls, 40 boys) who participated in a narrative retelling task at two time points in development (at 5–6 and 7–9 years of age). The 332 narratives were coded for both manual co-speech gesture types and prosody. Gestural annotations included gesture phasing (i.e., preparation, stroke, hold, and recovery) and gesture types (in terms of referentiality, i.e., referential and non-referential), whereas prosodic annotations included pitch-accented syllables. Results: Results revealed that by ages 5–6 years, children already temporally aligned the stroke of both referential and non-referential gestures with pitch-accented syllables, showing no significant differences between these two gesture types. Conclusions: The results of the present study contribute to the view that both referential and non-referential gestures are aligned with pitch accentuation, and therefore, this is not only a characteristic of non-referential gestures. Our results also add support to McNeill's phonological synchronization rule from a developmental perspective and indirectly back up recent theories about the biomechanics of gesture–speech alignment, suggesting that this is an inherent ability of oral communication.
9 January 2025 in Language, Speech, and Hearing Services in Schools
Purpose: The present study aims to evaluate the feasibility and preliminary effectiveness of a novel multi-tiered narrative intervention program—the multimodal narrative (MMN) program—in Catalan that was co-created to boost preschool children's narrative and pragmatic skills. Method: First, we describe here in detail the novel program, which consisted of a set of interventions oriented around the retelling of a narrative in a multimodal fashion, that is, with an abundant use of appropriate gesture and facial expression and careful attention to the pragmatic aspects of communication. We then describe the results of a self-reported feasibility study (Study 1) after this program was trial-implemented by 31 preschool teachers and speech-language therapists in their respective professional contexts. A pre- and post-intervention pilot study (Study 2) was conducted in which the researchers measured the effect of the MMN intervention on the 51 children who participated in the trial implementation. Results: Results from Study 1 revealed that most professionals adhered to the intervention protocol, that they found it enjoyable and easy to implement, and that it fostered active participation on the part of children. Results from Study 2 revealed that after the intervention, the narrative and pragmatic skills of all the children had improved. Conclusion: These results suggest that a full-fledged implementation of the MMN intervention program is feasible and has the potential to improve children's narrative and pragmatic skills in both clinical and educational contexts.
To see whether communicative-based multimodal skills (compared to non-communicative motor skills) predicted complex language skills, this study examined the predictive power of multimodal and motor skills on narrative and expressive pragmatic abilities across two groups. Participants were children with typical development (N = 88, Mage = 5.34, 48% female) and with neurodevelopmental disorders (N = 51, Mage = 5.01, 25% female) mostly of white ethnicity (86.3%). We evaluated children's multimodal accuracy, motor skills, core language skills, and narrative and pragmatic skills. Results revealed that, in all groups, both multimodal skills and core language significantly predicted narrative (R2 = 0.569) and pragmatic skills (R2 = 0.621), while motor skills did not. These findings highlight the relevance of multimodality in the assessment of children's complex language skills.
6 November 2025 in American Journal of Speech-Language Pathology
Purpose: This study aims to describe how a multi-tiered multimodal narrative intervention program was developed by integrating research and real-life professional practice through a participatory co-creation process using multiple methods. Method: A total of 24 preschool teachers, 69 therapists, and three researchers working in Catalonia participated in a five-session iterative co-creation process. First, the initial narrative intervention prototype was presented. Then, using the nominal group technique, practitioners generated ideas associated with narrative-based training. These ideas underwent qualitative and quantitative analyses to ensure that practitioners' needs were reflected in the prototype's design, with corrections made when necessary. Participants then piloted the intervention in their professional contexts and provided feedback. This led to the final revision of the narrative intervention program, approved by all participants. Results: Qualitative and quantitative analyses led to a few changes in the intervention protocols but essentially confirmed the practicability of the multi-tiered intervention prototype, which might therefore ensure the intervention's feasibility and effectiveness. Conclusions: This study aligns with the implementation science approach where educational interventions must be based on science and informed by contextual factors affecting their implementation in real life. Our results underline the importance of following co-creation processes that systematically accommodate practitioners' perspectives to improve educational and clinical outcomes.