The aim of our study was to determine the type and frequency of chromosomal aberrations and polymorphisms in men with different degrees of spermatogenic failure in comparison to men with normozoospermia, in order to find correlations between cytogenetic findings and the abnormal results of semen analysis. In our study, we performed cytogenetic analysis in 901 infertile men, divided into five groups according to semen analysis—normozoospermia (86), asthenozoospermia (394), oligoasthenozoospermia (182), severe male factor (100), and azoospermia (139). The frequency of polymorphisms was similar in all groups (11–16%, without significant differences). The frequency of numerical and structural aberrations increases with the degree of the spermatogenic failure (3.5% in normozoospermia, 5.6% in asthenozoospermia, 9.8% in oligoasthenozoospermia, 9% in severe male factor, and 13.5% in azoospermia). We found a significantly higher incidence of numerical chromosomal aberrations in severe male factor (7%) and azoospermia (9.3%). Oligoasthenozoospermia occured in 45% of cases with translocation, compared to 20% in the group with a normal karyotype. We revealed that chromosomal translocations are tightly associated with oligoasthenozoospermia, whereas numerical chromosomal aberrations—with severe male factor and azoospermia. The impact of chromosome polymorphisms on male infertility should be studied in greater detail.