The effects of fatigue on a driver’s autonomic nervous system (ANS) were investigated through heart rate variability (HRV) measures considering the difference of sex. Electrocardiogram (ECG) data from 18 drivers were recorded during a simulator-based driving experiment. Thirteen short-term HRV measures were extracted through time-domain and frequency-domain methods. First, differences in HRV measures related to mental state (alert or fatigued) were analyzed in all subjects. Then, sex-specific changes between alert and fatigued states were investigated. Finally, sex differences between alert and fatigued states were compared. For all subjects, ten measures showed significant differences (Mann-Whitney U test, p < 0.01) between different mental states. In male and female drivers, eight and four measures, respectively, showed significant differences between different mental states. Six measures showed significant differences between males and females in an alert state, while ten measures showed significant sex differences in a fatigued state. In conclusion, fatigue impacts drivers’ ANS activity, and this impact differs by sex; more differences exist between male and female drivers’ ANS activity in a fatigued state than in an alert state.