We introduce a novel system for automatic assessment of newborn and preterm infant behavior—including activity levels, behavioral states, and sleep–wake cycles—in clinical settings for streamlining care and minimizing healthcare professionals’ workload. While vital signs are routinely monitored, the previously mentioned assessments require labor-intensive direct observation. Research so far has already introduced non- and minimally invasive solutions. However, we developed a system that automatizes the preceding evaluations in a non-contact way using deep learning algorithms. In this work, we provide a Gated Recurrent Unit (GRU)-stack-based solution that works on a dynamic feature set generated by computer vision methods from the cameras’ video feed and patient monitor to classify the activity phases of infants adapted from the NIDCAP (Newborn Individualized Developmental Care Program) scale. We also show how pulse rate variability (PRV) data could improve the performance of the classification. The network was trained and evaluated on our own database of 108 h collected at the Neonatal Intensive Care Unit, Dept. of Neonatology of Pediatrics, Semmelweis University, Budapest, Hungary.