Coal mine waste rocks, mainly broken gangue, can be used as filling materials to backfill into goafs. Under the overburden load, the backfill body is vulnerable to compressive deformation and particle breakage. With the increase in mining depth, the overlying strata will impose different loads on waste rock filling materials at different loading velocities, which further affect the material compressive deformation and particle breakage. In this paper, an experimental scheme and a loading device are designed to study the influence of loading stress and velocity on the compressive deformation and particle size distributions of the backfill materials before and after compression. The results show that the axial strain of the gangue filling materials increases rapidly with the axial stress and then gradually stabilizes, showing a logarithmic functional relationship. Increasing the loading velocity will destroy the contact structures among the gangue particles and cause a larger deformation to the filling materials. When the loading stress is relatively low (5 MPa), the gangue particles with a size larger than 20 mm have a stronger bearing capacity compared with particles of 16–20 mm, which are the first particles to be crushed under these conditions. Further increasing the loading velocity will increase the breakage degree of the filling materials. The breakage ratio (B_M) has a logarithmic functional relationship with the loading stress and the loading velocity. When the ground stress is lower than 5 MPa, the content of coarse particles should be increased to enhance the bearing capacity of the gangue materials; when the ground stress is higher than 10 MPa, the content of fine particles should be increased to reduce the porosity ratio and the particle breakage ratio.