The increasing cost of healthcare services is accelerating the development of the telehealth system to fulfill the necessity of delivering an efficient and cost-effective remote healthcare services. Moreover, the ageing of the global population and the disruption of the COVID-19 pandemic are creating a rapid rise of demand for healthcare services. This includes those who are in need of remote monitoring for chronic conditions through rehabilitation exercises. Therefore, this paper presents a telehealth system using PoseNet for in-home rehabilitation, with built-in statistical computation for doctors to analyze the patient’s recovery status. This system enables patients to perform rehabilitation exercises at home using an ordinary webcam. The PoseNet skeleton-tracking method is applied to detect and track the patients’ angular movements for both elbows and knees. By using this system, the measurement of the elbow and knee joint angles can be calculated and recorded while patients are performing rehabilitation exercises in front of the laptop webcam. After the patients complete their rehabilitation exercises, the skeleton results of four body parts will be generated. Based on the same actions performed by patients on selected days, the doctors can examine and evaluate the deviation rate of patients’ angular movements between different days to determine the recovery rate.