Challenging due to the absence of color information and low resolution, infrared pedestrian tracking is still of great importance in video surveillance for its ability to distinguish and track persons in night vision. Recent researches show that the CF (Correlation Filters) based methods have a satisfying performance on the infrared pedestrian tracking task when combined with deep features. However, most of these algorithms only rely on the general convolutional features and online model updating to trace object variation, resulting in the lost track of pedestrians with large deformation and infrequent poses. In this paper, we propose a long-term inter-frame graph memory feature as a stronger descriptor for infrared pedestrians. We show that this cross-frame memory graph can enrich deep features with time-variant pedestrian appearance information, boosting the robustness and accuracy of the baseline CF tracker. Extensive experiments and analyses are carried out on the widely-used PTB-TIR dataset, demonstrating the effectiveness and universality of the proposed graph memory features for infrared pedestrians.