Identifying changes in driver’s vigilance under combined manual and automated driving conditions, as well as during prolonged driving, is crucial for reducing car crashes. Existing studies have not adequately considered the similarities and differences in physiological signals between different driving modes or the cumulative effects during extended driving periods. To address this gap, our study focuses on enhancing the feature selection method for driver’s vigilance recognition. A long-duration simulated car-following driving experiment was designed and conducted to simultaneously collect EEG, eye movement, EOG, and driving data. Similarities and differences in the physiological signals of vigilance between manual and automated driving are analyzed in terms of correlation and significance. The cumulative effects of physiological signals are investigated using time series analysis. The proposed feature selection method was validated using an LSTM-based driver’s vigilance recognition model. Results showed a recognition accuracy of 86.32% for manual driving, with a fluctuation rate of 1.18% over prolonged periods. For automated driving, the accuracy was 87.12%, with a fluctuation rate of 0.66%. By considering the similarities and differences in physiological signals between manual and automated driving modes and the cumulative effects, our study enhances the applicability and stability of the driver’s vigilance recognition model across different driving conditions. The validation results demonstrate that the proposed method improves the applicability and stability of the driver’s vigilance recognition model across different driving modes during extended driving periods.