In this study, a contactless vital signs monitoring system was proposed, which can measure body temperature (BT), heart rate (HR) and respiration rate (RR) for people with and without face masks using a thermal and an RGB camera. The convolution neural network (CNN) based face detector was applied and three regions of interest (ROIs) were located based on facial landmarks for vital sign estimation. Ten healthy subjects from a variety of ethnic backgrounds with skin colors from pale white to darker brown participated in several different experiments. The absolute error (AE) between the estimated HR using the proposed method and the reference HR from all experiments is $2.70\pm 2.28$ beats/min (mean ± std), and the AE between the estimated RR and the reference RR from all experiments is $1.47\pm 1.33$ breaths/min (mean ± std) at a distance of 0.6–1.2 m.