Teaching is the primary component of the educational process for college students. To achieve the training objectives of different subjects, every University has arranged various types of courses. The ultimate purpose of teaching activities for these courses consists of improving the students’ knowledge level, ability level, and quality level. However, most of the current teaching activities only focus on the evaluation system and performance evaluation method for each course, thus lacking an overall academic performance and evaluation pipeline designed for the influence of different courses on students’ future development. Bearing the above-mentioned analysis in mind, we introduced machine learning-based techniques into students’ academic performance analysis and leveraged the learning-based approaches of rank models to establish a framework for students’ learning ability analysis. By employing imaging sensors, the capability of multiple media including image and video has also been embedded into the proposed architecture. Meanwhile, we exploited the pipeline for different students’ academic performance to reveal the academic commonality among students and the different influence of the primary courses for students’ future development between different groups.