Crack detection at an early stage is necessary to save people’s lives and to prevent the collapse of building/bridge structures. Manual crack detection is time-consuming, especially when a building structure is too high. Image processing, machine learning, and deep learning-based methods can be used in such scenarios to build an automatic crack detection system. This study uses a novel deep convolutional neural network, 3SCNet (3ScaleNetwork), for crack detection. The SLIC (Simple Linear Iterative Clustering) segmentation method forms the cluster of similar pixels and the LBP (Local Binary Pattern) finds the texture pattern in the crack image. The SLIC, LBP, and grey images are fed to 3SCNet to form pool of feature vector. This multi-scale feature fusion (3SCNet+LBP+SLIC) method achieved the highest sensitivity, specificity, an accuracy of 99.47%, 99.75%, and 99.69%, respectively, on a public historical building crack dataset. It shows that using SLIC super pixel segmentation and LBP can improve the performance of the CNN (Convolution Neural Network). The achieved performance of the model can be used to develop a real-time crack detection system.