Tumor-infiltrating lymphocytes include heterogeneous populations of T lymphocytes that play crucial roles in the tumor immune response; importantly, their presence in the tumor tissue may predict clinical outcomes. Therefore, we herein studied the prognostic significance of the presence and location of CD3⁺, CD8⁺, and FoxP3⁺ T lymphocytes in colorectal cancer samples. In the intratumor analysis, our data did not reveal any association between lymphocyte infiltrations with clinical or pathological data. However, in the tumor margins, we found that the presence of high infiltrations of CD3⁺, CD8⁺, or FoxP3⁺ T lymphocytes were associated with TNM stages I-II (p = 0.021, p = 0.022, and p = 0.012, respectively) and absence of lymph node metastases (p = 0.010, p = 0.003, and p = 0.004, respectively). Despite these associations with good prognostic indicators, we were not able to find any statistically significant alterations in the overall survival of the patients, even though high infiltrations of FoxP3⁺ T lymphocytes in the tumor margins resulted in an increased overall survival of 14 months. Taken together, these data show that the presence of CD3⁺, CD8⁺, or FoxP3⁺T lymphocyte infiltrates in the tumor margins are associated with the pathogenesis of CRC, but only high Foxp3⁺ T lymphocyte infiltrations in the tumor invasive margins are inclined to indicate favorable prognosis.