In order to simulate the fracture process of particle-reinforced materials on the micro-scale, an image-based double-smoothing cohesive finite element framework is proposed in the present paper. Two separate smoothing processes are performed to reduce the noise in the digital image and eliminate the jagged elements in the finite element mesh. The main contribution of the present study is the proposed novel image-based cohesive finite element framework, and this method improved the quality of the meshes effectively. Meanwhile, the artificial resistance due to the jagged element is reduced with the double-smoothing cohesive finite element framework during the crack propagation. Therefore, the image-based double-smoothing cohesive finite element framework is significant for the simulation of fracture mechanics.