Stroke patients often experience a substantial deterioration in physical fitness and are susceptible to fatigue during rehabilitation training due to damage inflicted on their central nervous system. In clinical practice, the implementation of computer vision algorithms for intelligent recognition of fatigue states in stroke patients during rehabilitation training can offer timely reminders and interventions based on the level of fatigue occurrence. This approach can effectively alleviate the substantial burden on medical personnel who would otherwise need to continuously monitor and document patients in real-time. A fatigue action recognition algorithm that combines 3D-CNN and Transformer is proposed. To effectively highlight the key areas related to fatigue action and suppress other visual noise in the natural shooting frame, a feature segmentation structure with 3D window and a window merging mechanism with self-training 3D bell weight are designed. By processing the mean of features in the time dimension, the representation coefficient of the action relationship contained in the sequence is obtained, enhancing the discernability between adjacent similar frames. Based on the fatigue driving dataset published by Tsinghua University in Taiwan and the private fatigue action dataset based on stroke patients, the proposed method has achieved a Top-1 accuracy of 75.35% and 70.83%, respectively. These results prove the feasibility and effectiveness of the 3D CNN-driven Transformer architecture.