The presented results make an original contribution to the development of knowledge on the prediction and/or modeling of the abrasive wear properties of polyurethanes. A series of segmented linear polyurethane elastomers (PUR)—In which the hard segments consist of 4,4′-methylene bis(phenylisocyanate) and 1,4-butanodiol, whilst polyether, polycarbonate, or polyester polyols constitute the soft segments—Were synthesized and characterized. The hardness and wear performance as functions of the variable chemical composition of polyurethane elastomers were evaluated in order to define the relationship between studied factors. The microstructure was characterized in detail, including analysis of the hydrogen bonding by Fourier transformed infrared (FT-IR) spectroscopy and the phase structure by X-ray scattering (WAXS) and differential scanning calorimetry (DSC) methods. The presented studies provide the key features of the polymer composition affecting the abrasive resistance as well as attempts to explain the origin of the differences in the polyurethane elastomers’ performance.