Achieving favorable compatibility between detection performance and inference efficiency has long been a core bottleneck restricting the practical deployment of vision-based driver fatigue detection algorithms on vehicle-mounted embedded devices. In this work, we propose an adaptive temporal-aware feature selection paradigm for driver fatigue detection to address temporal isolation and computational redundancy problems. Specifically, we propose a temporal dual-stream fusion (TDF) module to generate unified embeddings of highly discriminative video features, laying a solid feature foundation for superior final performance. Additionally, we design an adaptive feature selection (AFS) mechanism that evaluates the necessity of executing each stream based on confidence estimation, eliminating redundant computations with acceptable slight accuracy loss. In addition, the experimental results on the challenging NTHU-DDD dataset demonstrate the superiority of our method, especially from the perspective of balancing performance and efficiency.