This article uses cutting‐edge deep learning technology to identify structural damage from images for a civil engineering application. The public infrastructures of the country are generally inspected physically by a visual evaluation by qualified inspectors. However, manual inspections are pretty time‐consuming and often require too much labor. The number of experts capable of evaluating such structural damage is inadequate. As a result, computer vision‐based techniques for automatic damage detection have been developed. This paper’s civil infrastructure damages are classified into four damages of roads common in Indian highways and the concrete deterioration in the bridges. The convolutional neural network has become a standard tool for organizing and recognizing images. In this paper, an ensemble of three CNN models is proposed, and two are transfer learning‐based models. The proposed ensemble transfer learning model provided a validation accuracy of 87.1%.