Soft sensors are attracting significant attention in human–machine interaction due to their high flexibility and adaptability. However, estimating motion state from these sensors is difficult due to their nonlinearity and noise. In this paper, we propose a deep learning network for a smart glove system to predict the moving state of a piezoelectric soft sensor. We implemented the network using Long-Short Term Memory (LSTM) units and demonstrated its performance in a real-time system based on two experiments. The sensor’s moving state was estimated and the joint angles were calculated. Since we use moving state in the sensor offset calculation and the offset value is used to estimate the angle value, the accurate moving state estimation results in good performance for angle value estimation. The proposed network performed better than the conventional heuristic method in estimating the moving state. It was also confirmed that the calculated values successfully mimic the joint angles measured using a leap motion controller.