Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Doctor performance evaluation (DPE) is an important task in mHealth, which aims to evaluate the overall quality of online diagnosis and patient outcomes so that high customer satisfaction and loyalty can be attained. However in reality most customers trend not to give ratings to doctors’ performance. Therefore it is imperative to develop a model to make DPE automatically. When making auto-evaluation of doctors’ performance, we expect to rate the doctors’ performance into a score label that is as close as possible to the true one. Objective: This study aims to do DPE automatically from online textual consultation contents between doctors and customers by a novel machine learning method. Methods: We propose a solution that models DPE as an ordinal regression problem. In doing so, a combined support vector machine and ordinal partitioning model (called SVMOP for short), along with an innovative prediction function is developed to capture the ordering labels hidden in preferences over DPE. Specifically, when conducting the feature engineering, in addition to the basic text features, eight handcrafted features extracted from over 70,000 medical entries, are added and then further boosted by Gradient Boosting Decision Tree. Results: Real data sets from one of the largest mobile doctor-patient communication platforms in China are used in our study. In according with the statistics, 64% of data on mHealth platforms lack the evaluation labels from customers. Experimental results reveal that our approach can well support doctor performance evaluation automatically. Specifically, compared with other auto-evaluation models, SVMOP improves mean absolute error (MAE) by 0.1, mean square error (MSE) by 0.5, pairwise accuracy (PAcc) by 5%; the handcrafted features that we suggest, improve MAE by 0.1, MSE by 0.2, PAcc by 3%. After boosting, the performance can be further improved. Furthermore, based on SVMOP, we could also mine some predictive features such as useful polite expressions and sentiment words, which can be further applied to guide the development of mHealth platforms. Conclusions: It is the first time to model the problem of doctor performance evaluation as an ordinal regression problem. Experiments show that the performance of our proposed model with revised prediction function is better than many other machine learning methods on MAE, MSE as well as PAcc. In summary, with this model the mHealth platform could not only make an auto-evaluation of doctors’ performance online but also mine the most effective features, which could guide the promotion of doctors’ performance and the development of mHealth platforms.