As an additive to improve the performance of asphalt binder, tire pyrolysis carbon black is gradually being used in road engineering, but the effect of carbon black (CB) with different particle sizes on asphalt modification remains to be further studied. In this study, three kinds of particle sizes and three kinds of contents of CB were used to modify asphalt, and different tests were conducted to research the high temperature performance and fatigue resistance of carbon black modified asphalt binder. It is found that the addition of CB can enhance the rutting resistance and medium temperature fatigue performance of virgin asphalt binder in general. However, for CB of 270 μm and 2.6 μm, its addition under certain contents lead to the decrease of high temperature performance and fatigue performance of the asphalt binder. For aged asphalt, the addition of CB decreases the rutting resistance and improves the fatigue resistance. The recommended content and particle size of CB are 2% and 2.6 μm. This study refines the complex effects of CB on asphalt properties, providing a reference for determining the size and content of CB in asphalt modification.