Timely and overall knowledge of the states and resource allocation of diverse activities on construction sites is critical to resource leveling, progress tracking, and productivity analysis. Despite its importance, this task is still performed manually. Previous studies have taken a significant step forward in introducing computer vision technologies, although they have been oriented toward limited classes of objects or limited types of activities. Furthermore, they especially focus on single activity recognition, where an image contains only the execution of an activity by one or a few objects. This paper introduces a two-step method for recognizing diverse construction activities in still site images. It detects 22 classes of construction-related objects using convolutional neural networks. With objects detected, semantic relevance representing the likelihood of the cooperation or coexistence between two objects in a construction activity, spatial relevance representing the two-dimensional pixel proximity in the image coordinates, and activity patterns are defined to recognize 17 types of construction activities. The advantage of the proposed method is its potential to recognize diverse concurrent construction activities in a fully automatic way. Therefore, it is possible to save managers’ valuable time in manual data collection and concentrate their attention on solving problems that necessarily demand their expertise.

Recording workers’ activities is an important, but burdensome, management task for site supervisors. The last decade has seen a growing trend toward vision‐based activity recognition. However, recognizing workers’ activities in far‐field surveillance videos is understudied. This study proposes a hierarchical statistical method for recognizing workers’ activities in far‐field surveillance videos. The method consists of two steps. First, a deep action recognition method was used to recognize workers’ actions, and a new fusion strategy was proposed to consider the characteristics of far‐field surveillance videos. The deep action recognition method with the new fusion strategy has achieved the comparable performance (0.84 average accuracy) on the far‐field surveillance data set in contrast to the original method on the public data sets. Second, a Bayesian nonparametric hidden semi‐Markov model was innovatively used to model and infer workers’ activities based on action sequences. The latent states of the fitted Bayesian model captured workers’ activities in terms of state duration distributions and state transition distributions, which are indispensable for understanding workers’ time allocation. It has been preliminarily illustrated that the activity information learned by the Bayesian model possesses the potential to implement objective work sampling, personal physical fatigue monitoring, trade‐level health risk assessment, and process‐based quality control. Also, the limitations of this study are discussed.