Few studies have been undertaken to understand the employment impact in patients with colorectal cancer and none in middle-aged individuals with cancer. This study described transitions in, and key factors influencing, work participation during the 12 months following a diagnosis of colorectal cancer. We enrolled 239 adults during 2010 and 2011who were employed at the time of their colorectal cancer diagnosis and were prospectively followed over 12 months. They were compared to an age- and gender-matched general population group of 717 adults from the Household, Income and Labour Dynamics in Australia (HILDA) Survey. Data were collected using telephone and postal surveys. Primary outcomes included work participation at 12 months, changes in hours worked and time to work re-entry. Multivariable logistic and Cox proportional hazards models were undertaken. A significantly higher proportion of participants with colorectal cancer (27%) had stopped working at 12 months than participants from the comparison group (8%) (p < 0.001). Participants with cancer who returned to work took a median of 91 days off work (25–75 percentiles: 14–183 days). For participants with cancer, predictors of not working at 12 months included: being older, lower BMI and lower physical well-being. Factors related to delayed work re-entry included not being university-educated, working for an employer with more than 20 employees in a non-professional or managerial role, longer hospital stay, poorer perceived financial status and having or had chemotherapy. In middle-adulthood, those working and diagnosed with colorectal cancer can expect to take around three months off work. Individuals treated with chemotherapy, without a university degree and from large employers could be targeted for specific assistance for a more timely work entry. ACTRN12611000530921