Although matrix model-based approaches to collaborative filtering (CF), such as latent factor models, achieve good accuracy in review rating prediction, they still face data sparsity problems. Many recent studies have exploited review text information to improve the performance of predictions. The review content that they use, however, is usually on the coarse-grained text level or sentence level. In this paper, we propose a joint model that incorporates review text information with matrix factorization for review rating prediction. First, we adopt an aspect extraction method and propose a simple and practical algorithm to represent the review by aspects and sentiments. Then, we propose two similarity measures: aspect-based user similarity and aspect-based product similarity. Finally, aspect-based user and product similarity measures are incorporated into a matrix factorization to build a joint model for rating prediction. To this end, our model can alleviate the data sparsity problem and obtain interpretability for the recommendation. We conducted experiments on two datasets. The experimental results demonstrate the effectiveness of the proposed model.