Physical fatigue is a major cause of work-related accidents and musculoskeletal injuries in the construction industry, and additional means are needed for their identification and management to prevent long-term consequences. Based on recent scientific literature, fatigue can be detected with wearable inertial measurement units (IMUs). However, IMUs for detecting fatigue have been so far tested mainly in the laboratory; therefore, a research gap exists in application of IMU sensors for detecting fatigue in real-life work settings. The aim of this paper is to bring the fatigue trend detection with IMUs closer to real-life context by using wearable IMU sensor data from an actual construction site measuring actual workers with simulated work tasks. The paper also presents advancements in fatigue trend detection with frequency domain investigations to gain access to more detailed fatigue relevant features. Machine-learning methods are used to predict fatigue trends based on IMU data, resulting in fatigue trend detection accuracy that advances the state of the art. More knowledge is also unearthed about relevant sensor locations and features.