Drowsiness behind the wheel is a major road safety issue with efforts focused on developing drowsy driving detection systems. However, most drowsy driving detection studies using physiological signals have focused on developing a 'black box' machine learning classifier, with much less focus on 'robustness' and 'explainability'-two crucial properties of a trustworthy machine learning model. Therefore, this study has focused on using multiple validation techniques to evaluate the overall performance of such a system using multiple supervised machine learning-based classifiers and then unbox the black box model using explainable machine learning. Driving was simulated via a 30-minute psychomotor vigilance task while the participants reported their level of subjective sleepiness with their physiological signals: electroencephalogram (EEG), electrooculogram (EOG) and electrocardiogram (ECG) being recorded. Six different techniques, comprising subject-dependent and independent techniques were applied for model validation and robustness testing with three supervised machine learning classifiers, namely K-nearest neighbours (KNN), support vector machines (SVM) and random forest (RF), and two explainable methods, namely SHapley Additive exPlanation (SHAP) analysis and partial dependency analysis (PDA) were leveraged for model interpretation. The study identified the leave one participant out, a subject-independent validation technique to be most useful, with the best sensitivity of 70.3 %, specificity of 82.2 %, and an accuracy of 80.1 % using the random forest classifier in addressing the autocorrelation issue due to inter-individual differences in physiological signals. Moreover, the explainable results suggest most important physiological features for drowsiness detection, with a clear cut-off in the decision boundary. The implication of the study will ensure a rigorous validation for robustness testing and an explainable machine learning approach to developing a trustworthy drowsiness detection system and enhancing road safety. The explainable machine learning-based results show promise in real-life deployment of the physiological-signal based in-vehicle trustworthy drowsiness detection system, with higher reliability and explainability, along with a lower system cost.