The flexible job-shop scheduling problem with worker flexibility (FJSPW) has been a frequently researched topic in recent years. In FJSPW, the number of workers is generally fewer than the number of machines, and workers are flexible enough to perform operations on multiple machines, thereby improving productivity. However, the consideration of workers' well-being is generally overlooked. Worker well-being is crucial, particularly in manufacturing processes where frequent manual handling of tasks in awkward positions is involved, to prevent fatigue and work-related musculoskeletal disorders. This paper proposes an FJSPW model that considers ergonomic factors (FJSPW-Ergo) to improve workers' productivity and well-being simultaneously. The ergonomic load is measured using the rapid entire body assessment (REBA) score. An improved non-dominated sorting genetic algorithm II (NSGA-II) is employed to solve the FJSPW-Ergo model with a three-layer encoding method, two crossover operators and two mutation methods to minimize the makespan and maximum REBA score (i.e., the highest ergonomic load experienced by any worker). Different combinations of jobs and machines with a variable number of workers are used to demonstrate the effectiveness of the proposed FJSPW-Ergo approach. The results show that FJSPW-Ergo is an effective method for simultaneously achieving productivity and ergonomic relief.