This paper presents the first multimodal neonatal pain dataset that contains visual, vocal, and physiological responses following clinically required procedural and postoperative painful procedures. It was collected from 58 neonates (27-41 gestational age) during their hospitalization in the neonatal intensive care unit. The visual and vocal data were recorded using an inexpensive RGB camera while the physiological responses (vital signs and cortical activity) were recorded using portable bedside monitors. The recorded behavioral and physiological responses were scored by expert nurses using two validated pain scales to obtain the ground truth labels. In addition to behavioral and physiological responses, our dataset contains clinical information such as the neonate’s age, gender, weight, pharmacological and non-pharmacological interventions, and previous painful procedures. The presented multimodal dataset can be used to develop artificial intelligence systems that monitor, assess, and predict neonatal pain based on the analysis of behavioral and physiological responses. It can also be used to advance the understanding of neonatal pain, which can lead to the development of effective pain prevention and treatment.