Mineral identification is an important part of geological analysis. Traditional identification methods rely on either the experience of the appraisers or the various measuring instruments, and the methods are either easily influenced by appraisers’ experience or require too much work. To solve the above problems, there are studies using image recognition and intelligent algorithms to identify minerals. However, current studies cannot identify many minerals, and the accuracy is low. To increase the number of identified minerals and accuracy, we propose a method that uses both mineral photo images and the Mohs hardness in deep neural networks to identify the minerals. The experimental results showed that the method can reach 90.6% top-1 accuracy and 99.6% top-5 accuracy for 36 common minerals. An app based on the model was implemented on smartphones with no need for accessing the internet and communication signals. Tested on 73 real mineral samples, the app achieved top-1 accuracy of 89% when the mineral image and hardness are both used and 71.2% when only the mineral image is used.