The safety of construction workers is always a major concern at construction sites as the construction industry is inherently dangerous with many factors influencing worker safety. Several studies concluded that psychological factors such as workload, organizational relationships, mental stress, job security, and job satisfaction have significant effects on workers’ safety. However, research on psychological factors that are characteristic of different age groups have been limited. The aim of this study was to examine the impact of psychological factors on the perception of worker safety for two different age groups. After an extensive literature review, different psychological factors were identified, and a hypothetical research model was developed based on psychological factors that could affect workers’ perception of safety. A survey instrument was developed, and data were collected from seven different construction sites in Pakistan. Structural equation modeling (SEM) was employed to test the hypothetical model for both age groups. The results revealed that workload and job satisfaction are significantly dominant factors on workers’ perception of safety in older workers, whereas organizational relationships, mental stress, and job security are dominant factors for younger workers at construction sites.