An artificial intelligence (AI) method based on image deep learning is proposed to predict the entire displacement shape of a structure using the feature of partial displacements. The performance of the method was investigated through a structural test of a steel frame. An image-to-image regression (I2IR) training method was developed based on the U-Net layer for image recognition. In the I2IR method, the U-Net is modified to generate images of entire displacement shapes when images of partial displacement shapes of structures are input to the AI network. Furthermore, the training of displacements combined with the location feature was developed so that nodal displacement values with corresponding nodal coordinates could be used in AI training. The proposed training methods can consider correlations between nodal displacements in 3D space, and the accuracy of displacement predictions is improved compared with artificial neural network training methods. Displacements of the steel frame were predicted during the structural tests using the proposed methods and compared with 3D scanning data of displacement shapes. The results show that the proposed AI prediction properly follows the measured displacements using 3D scanning.