Immune checkpoint inhibitors and immune-related biomarkers are increasingly investigated in rectal cancer (RC). We retrospectively analysed PD-L1 expression in diagnostic biopsy and resection samples from RC patients treated at our centre between 2000 and 2020. PD-L1 immunostaining (22C3 clone) was evaluated according to tumour proportion (TPS), immune cell (ICS), and the combined positive score (CPS). Eighty-three patients were included. At diagnosis, PD-L1 expression ≥1%/≥5% was observed in 15.4%/0%, 80.7%/37.4%, and 69.2%/25.6% of patients based on TPS, ICS, and CPS, respectively. At surgery, the respective figures were 4.6%/1.5%, 60.2%/32.5%, and 50.7%/26.2%. Using the 1% cut-off and regardless of the scoring system, PD-L1 was less expressed in surgery than biopsy samples (p ≤ 0.04). In paired specimens, PD-L1-ICS reduction was especially observed following neoadjuvant long-course (chemo)radiotherapy (p = 0.03). PD-L1-ICS of ≥5% in surgical samples (HR: 0.17; p = 0.02), and a biopsy-to-surgery increase in PD-L1-ICS (HR: 0.19; p = 0.04) was predictive for longer disease-free survival, while the PD-L1-ICS of either ≥1% (HR 0.28; p = 0.04) or ≥5% (HR 0.19; p = 0.03) in surgical samples and the biopsy-to-surgery increase in PD-L1-ICS (HR: 0.20; p = 0.04) were associated with better overall survival. Our study suggests that PD-L1 expression in RC is largely reflective of immune cell infiltration, and its presence/increase in surgical samples predicts better outcomes.