The aim of this study was to analyse the association of the use of the mobile phone with physical fitness (PF) and academic performance in secondary school students and its gender-related differences. A total of 501 high school students participated in the study (236 girls and 265 boys; 12–18 years). Use of the mobile phone and sample distributions were done with the Mobile-Related Experience Questionnaire (CERM): low use of mobile phone (LMP = 10–15 points), medium use of mobile phone (MMP = 16–23 points) and high use of mobile phone (HMP = 24–40 points). PF via Eurofit test battery and academic performance were recorded, and gender was used as a differentiating factor. The HMP group registered lower values than the LMP group for academic performance (Spanish: 4.78 ± 2.26 vs. 3.90 ± 1.96 points; p = 0.007, Mathematics: 4.91 ± 2.23 vs. 4.00 ± 1.84 points; p = 0.007) and PF (Abdominals: 6.83 ± 2.40 vs. 5.41 ± 2.46 points; p < 0.001, Broad jump: 6.24 ± 3.02 vs. 4.94 ± 2.28 points; p = 0.013). The boy students showed greater values than girl students for PF in the LMP (medicine-ball-throw: 6.34 ± 2.24 vs. 5.28 ± 1.86 points, p = 0.007) and MMP (medicine-ball-throw: 6.49 ± 2.52 vs. 5.02 ± 1.68 points; p < 0.001) groups, but no gender-related differences were found in the HMP group. In conclusion, high use of the mobile phone was related to worse results in the PF tests and academic performance. Gender-related differences were found for academic performance regardless of the use of the mobile, but for physical fitness no gender differences were found in HMP group.