In this study, the crystal plasticity finite element method was established by coupling the crystal plasticity and finite element method (FEM). The effect of rolling deformation and slip system of polycrystalline Al-Mg-Si aluminum alloy was investigated. The results showed that there was a pronounced heterogeneity in the stress and strain distribution of the material during cold rolling. The maximum strain and shear strain occurred at surface of the material. The smaller the grain size, the lower the strain concentration at the grain boundary. Meanwhile, a smaller strain difference existed between the grain interior and near the boundary. The rotation of grains leads to significant differences in deformation and rotation depending on their initial orientations during the rolling process. The slip system of (11-1)<-110> had a large effect on the plastic deformation, (111) is second, and the effect of (1-11) slip system on the plastic deformation is the smallest. After deformation, the grain orientation concentration was increased with deformation. Therefore, both the strength and volume fraction of texture were increased with the increase in rolling deformation. The experimental results of EBSD indicated that the large rolling reduction resulted in severe grain twisting, so the texture strength was increased. The simulation results were in close agreement with the experimental results. This study provides a theoretical basis for the rolling process, microstructure, and performance control of aluminum alloys.