Various data representation algorithms have been proposed for gene expression. There are some shortcomings in traditional gene expression methods, such as learning the ideal affinity matrix to effectively capture the geometric structure of genetic data space, and reducing noises and outliers influences of data input. We propose a novel matrix factorization algorithm called Robust semi-nonnegative matrix factorization (RSNMF) with adaptive graph regularization, which simultaneously performs matrix robust factorization with learning affinity matrix in a unified optimization framework. RSNMF also uses a loss function based on l2;1-norm to improve the robustness of the model against noises and outliers. A novel Augmented Lagrange multiplier (ALM) is designed to obtain the optimal solution of RSNMF. The results of extensive experiments that were performed on gene expression datasets demonstrate that RSNMF outperforms the other algorithms, which validates the effectiveness and robustness of RSNMF.