Objective : Video and sound acquisition and processing technologies have seen great improvements in recent decades, with many applications in the biomedical area. The aim of this paper is to review the overall state of the art of advances within these topics in paediatrics and to evaluate their potential application for monitoring in the neonatal intensive care unit (NICU). Approach : For this purpose, more than 150 papers dealing with video and audio processing were reviewed. For both topics, clinical applications are described according to the considered cohorts—full-term newborns, infants and toddlers or preterm newborns. Then, processing methods are presented, in terms of data acquisition, feature extraction and characterization. Main results : The paper first focuses on the exploitation of video recordings; these began to be automatically processed in the 2000s and we show that they have mainly been used to characterize infant motion. Other applications, including respiration and heart rate estimation and facial analysis, are also presented. Audio processing is then reviewed, with a focus on the analysis of crying. The first studies in this field focused on induced-pain cries and the newest ones deal with spontaneous cries; the analyses are mainly based on frequency features. Then, some papers dealing with non-cry signals are also discussed. Significance : Finally, we show that even if recent improvements in digital video and signal processing allow for increased automation of processing, the context of the NICU makes a fully automated analysis of long recordings problematic. A few proposals for overcoming some of the limitations are given.

This study was designed to test if heart rate variability (HRV) data from preterm and full-term infants could be used to estimate their functional maturational age (FMA), using a machine learning model. We propose that the FMA, and its deviation from the postmenstrual age (PMA) of the infants could inform physicians about the progress of the maturation of the infants. The HRV data was acquired from 50 healthy infants, born between 25 and 41 weeks of gestational age, who did not present any signs of abnormal maturation relative to their age group during the period of observation. The HRV features were used as input for a machine learning model that uses filtering and genetic algorithms for feature selection, and an ensemble machine learning (EML) algorithm, which combines linear and random forest regressions, to produce as output a FMA. Using HRV data, the FMA had a mean absolute error of 0.93 weeks, 95% CI [0.78, 1.08], compared to the PMA. These results demonstrate that HRV features of newborn infants can be used by an EML model to estimate their FMA. This method was also generalized using respiration rate variability (RRV) and bradycardia data, obtaining similar results. The FMA, predicted either by HRV, RRV or bradycardia, and its deviation from the true PMA of the infants, could be used as a surrogate measure of the maturational age of the infants, which could potentially be monitored non-invasively and in real-time in the setting of neonatal intensive care units.

Background: Sleep is an important determinant of brain development in preterm infants. Its temporal organization varies with gestational age (GA) and post-menstrual age (PMA) but little is known about how sleep develops in very preterm infants. The objective was to study the correlation between the temporal organization of quiet sleep (QS) and maturation in premature infants without severe complications during their neonatal hospitalization. Methods: Percentage of time spent in QS and average duration of time intervals (ADI) spent in QS were analyzed from a cohort of newborns with no severe complications included in the Digi-NewB prospective, multicentric, observational study in 2017–19. Three groups were analyzed according to GA: Group 1 (27–30 weeks), Group 2 (33–37 weeks), Group 3 (>39 weeks). Two 8-h video recordings were acquired in groups 1 and 2: after birth (T1) and before discharge from hospital (T2). The annotation of the QS phases was performed by analyzing video recordings together with heart rate and respiratory traces thanks to a dedicated software tool of visualization and annotation of multimodal long-time recordings, with a double expert reading. Results are expressed as median (interquartile range, IQR). Correlations were analyzed using a linear mixed model. Results: Five newborns were studied in each group (160 h of recording). Median time spent in QS increased from 13.0% [IQR: 13–20] to 28.8% [IQR: 27–30] and from 17.0% [IQR: 15–21] to 29.6% [IQR: 29.5–31.5] in Group 1 and 2, respectively. Median ADI increased from 54 [IQR: 53–54] to 288 s [IQR: 279–428] and from 90 [IQR: 84–96] to 258 s [IQR: 168–312] in Group 1 and 2. Both groups reach values similar to that of group 3, respectively 28.2% [IQR: 24.5–31.3] and 270 s [IQR: 210–402]. The correlation between PMA and time spent in QS or ADI were, respectively 0.73 (p < 10⁻⁴) and 0.46 (p = 0.06). Multilinear analysis using temporal organization of QS gave an accurate estimate of PMA (r² = 0.87, p < 0.001). Conclusion: The temporal organization of QS is correlated with PMA in newborns without severe complication. An automated standardized continuous behavioral quantification of QS could be interesting to monitor during the hospitalization stay in neonatal units.

Cry analysis is an important tool to evaluate the development of preterm infants. However, the context of Neonatal Intensive Care Units is challenging, since a wide variety of sounds can occur (e.g., alarms and adult voices). In this paper, a method to extract cries is proposed. It is based on an initial segmentation between silence and sound events, followed by feature extraction on the resulting audio segments and a cry and non-cry classification. A database of 198 cry events coming from 21 newborns and 439 non-cry events was created. Then, a set of features—including Mel-Frequency Cepstral Coefficients—issued from principal component analysis, was computed to describe each audio segment. For the first time in cry analysis, noise was handled using harmonic plus noise analysis. Several machine learning models have been compared. The K-Nearest Neighbours approach showed the best results with a precision of 92.9%. To test the approach in a monitoring application, 412 h of recordings were automatically processed. The cries automatically selected were replayed and a precision of 92.2% was obtained. The impact of errors on the fundamental frequency characterisation was also studied. Results show that despite a difficult context, automatic cry extraction for non-invasive monitoring of vocal development of preterm infants is achievable.