Purpose: Sexual function is crucial for the quality of life and can be highly affected by preoperative therapy and surgery. The aim of this study was to identify potential risk factors for poor sexual function and quality of life. Methods: Female patients were asked to complete the Female Sexual Function Index (FSFI-6). Male patients were demanded to answer the International Index of Erectile Function (IIEF-5). Results: In total, 79 patients filled in the questionary, yielding a response rate of 41.57%. The proportion of women was represented by 32.91%, and the median age was 76.0 years (66.0–81.0). Sexual dysfunction appeared in 88.46% of female patients. Severe erectile dysfunction occurred in 52.83% of male patients. Univariate analysis showed female patients (OR: 0.17, 95%CI: 0.05–0.64, p = 0.01), older age (OR: 0.34, 95%CI 0.11–1.01, p = 0.05), tumor localization under 6cm from the anal verge (OR: 4.43, 95%CI: 1.44–13.67, p = 0.01) and extension of operation (APR and ISR) (OR: 0.13, 95%CI: 0.03–0.59, p = 0.01) as significant risk factors for poor outcome. Female patients (OR: 0.12, 95%CI: 0.03–0.62, p = 0.01) and tumors below 6 cm from the anal verge (OR: 4.64, 95%CI: 1.18–18.29, p = 0.03) were shown to be independent risk factors for sexual dysfunction after multimodal therapy in the multivariate analysis. Quality of life was only affected in the case of extensive surgery (p = 0.02). Conclusion: Higher Age, female sex, distal tumors and extensive surgery (APR, ISR) are revealed risk factors for SD in this study. Quality of life was only affected in the case of APR or ISR.