Studies had suggested an association between circadian disruptors (including night shift work, domestic light exposure at night, sleep duration, and circadian gene polymorphism) and breast cancer, while rare studies had been conducted in the Chinese population. This study was a case-control study conducted to explore the impact of circadian disruptors on the risk of breast cancer in China. Four hundred and sixty-four cases and 464 controls, admitted from the Department of Breast Surgery, Cancer Hospital, Chinese Academy of Medical Sciences, were included in this study. Adjusting age, BMI group, smoking, alcohol consumption, menopausal status, family history of breast cancer, duration of breastfeeding, age at menarche, number of pregnancies, age at first full-term pregnancy, use of estrogen and use of oral contraceptive, multivariate logistic regression analysis showed that the risk of breast cancer was higher in short sleep duration group (OR = 4.86, 95%CI: 1.73–17.33). Meanwhile, rs2292912 in CRY2, rs2253820 in PER1, rs2289591 in PER1 and rs3027188 in PER1 were positively associated with the risk of breast cancer. This study supported that the short duration of sleep and four SNPs in crucial circadian genes played a role in the development of breast cancer.