Physical activity recognition using embedded sensors has enabled many context-aware applications in different areas, such as healthcare. Initially, one or more dedicated wearable sensors were used for such applications. However, recently, many researchers started using mobile phones for this purpose, since these ubiquitous devices are equipped with various sensors, ranging from accelerometers to magnetic field sensors. In most of the current studies, sensor data collected for activity recognition are analyzed offline using machine learning tools. However, there is now a trend towards implementing activity recognition systems on these devices in an online manner, since modern mobile phones have become more powerful in terms of available resources, such as CPU, memory and battery. The research on offline activity recognition has been reviewed in several earlier studies in detail. However, work done on online activity recognition is still in its infancy and is yet to be reviewed. In this paper, we review the studies done so far that implement activity recognition systems on mobile phones and use only their on-board sensors. We discuss various aspects of these studies. Moreover, we discuss their limitations and present various recommendations for future research.

The position of on-body motion sensors plays an important role in human activity recognition. Most often, mobile phone sensors at the trouser pocket or an equivalent position are used for this purpose. However, this position is not suitable for recognizing activities that involve hand gestures, such as smoking, eating, drinking coffee and giving a talk. To recognize such activities, wrist-worn motion sensors are used. However, these two positions are mainly used in isolation. To use richer context information, we evaluate three motion sensors (accelerometer, gyroscope and linear acceleration sensor) at both wrist and pocket positions. Using three classifiers, we show that the combination of these two positions outperforms the wrist position alone, mainly at smaller segmentation windows. Another problem is that less-repetitive activities, such as smoking, eating, giving a talk and drinking coffee, cannot be recognized easily at smaller segmentation windows unlike repetitive activities, like walking, jogging and biking. For this purpose, we evaluate the effect of seven window sizes (2–30 s) on thirteen activities and show how increasing window size affects these various activities in different ways. We also propose various optimizations to further improve the recognition of these activities. For reproducibility, we make our dataset publicly available.

Although running deep learning algorithms is challenging due to resource constraints on mobile and wearable devices, they provide performance improvements compared to lightweight or shallow architectures. The widespread application areas for ondevice deep learning include computer vision, image processing, natural language processing, and audio classification. However, mobile and wearable sensing applications are also gaining attention. They can benefit from on-device deep learning, given that these devices are integrated with various sensors and produce large amounts of data. This paper reviews state-of-the-art studies on on-device deep learning for mobile and wearable devices, particularly from the sensor data analytics perspective. We first discuss the general optimization techniques of deep learning algorithms to meet the resource limitations of the devices. Then, we elaborate on model update and personalization techniques and review the studies by classifying them according to several aspects, including application areas, sensors, types of devices, utilized DL algorithms, mode of implementation, methods for optimizing DL algorithms for the target devices, training method, implementation toolkit/platform, performance metrics and resource consumption analysis. Finally, we discuss the open issues and future research directions about on-device deep learning for mobile and wearable sensing applications.

Phone placement, i.e., where the phone is carried/stored, is an important source of information for context-aware applications. Extracting information from the integrated smart phone sensors, such as motion, light and proximity, is a common technique for phone placement detection. In this paper, the efficiency of an accelerometer-only solution is explored, and it is investigated whether the phone position can be detected with high accuracy by analyzing the movement, orientation and rotation changes. The impact of these changes on the performance is analyzed individually and both in combination to explore which features are more efficient, whether they should be fused and, if yes, how they should be fused. Using three different datasets, collected from 35 people from eight different positions, the performance of different classification algorithms is explored. It is shown that while utilizing only motion information can achieve accuracies around 70%, this ratio increases up to 85% by utilizing information also from orientation and rotation changes. The performance of an accelerometer-only solution is compared to solutions where linear acceleration, gyroscope and magnetic field sensors are used, and it is shown that the accelerometer-only solution performs as well as utilizing other sensing information. Hence, it is not necessary to use extra sensing information where battery power consumption may increase. Additionally, I explore the impact of the performed activities on position recognition and show that the accelerometer-only solution can achieve 80% recognition accuracy with stationary activities where movement data are very limited. Finally, other phone placement problems, such as in-pocket and on-body detections, are also investigated, and higher accuracies, ranging from 88% to 93%, are reported, with an accelerometer-only solution.

Human activity recognition and behavior monitoring in a home setting using wireless sensor networks (WSNs) provide a great potential for ambient assisted living (AAL) applications, ranging from health and wellbeing monitoring to resource consumption monitoring. However, due to the limitations of the sensor devices, challenges in wireless communication and the challenges in processing large amounts of sensor data in order to recognize complex human activities, WSN-based AAL systems are not effectively integrated in the home environment. Additionally, given the variety of sensor types and activities, selecting the most suitable set of sensors in the deployment is an important task. In order to investigate and propose solutions to such challenges, we introduce a WSN-based multimodal AAL system compatible for homes with multiple residents. Particularly, we focus on the details of the system architecture, including the challenges of sensor selection, deployment, networking and data collection and provide guidelines for the design and deployment of an effective AAL system. We also present the details of the field study we conducted, using the systems deployed in two different real home environments with multiple residents. With these systems, we are able to collect ambient sensor data from multiple homes. This data can be used to assess the wellbeing of the residents and identify deviations from everyday routines, which may be indicators of health problems. Finally, in order to elaborate on the possible applications of the proposed AAL system and to exemplify directions for processing the collected data, we provide the results of several human activity inference experiments, along with examples on how such results could be interpreted. We believe that the experiences shared in this work will contribute towards accelerating the acceptance of WSN-based AAL systems in the home setting.

Authenticating a user in the right way is essential to IT systems, where the risks are becoming more and more complex. Especially in the mobile world, banking applications are among the most delicate systems requiring strict rules and regulations. Existing approaches often require point-of-entry authentication accompanied by a one-time password as a second-factor authentication. However, this requires active participation of the user and there is continuous authentication during a session. In this paper, we investigate whether it is possible to continuously authenticate users via behavioral biometrics with a certain performance on a mobile banking application. A currently used mobile banking application in Turkey is chosen as the case, and we developed a continuous authentication scheme, named DAKOTA, on top of this application. The DAKOTA system records data from the touch screen and the motion sensors on the phone to monitor and model the user's behavioral patterns. Forty-five participants completed the predefined banking transactions. This data is used to train seven different classification algorithms. The results reveal that binary-SVM with RBF kernel reaches the lowest error scores, 3.5% equal error rate (EER). Using the end-to-end DAKOTA system, we investigate the performance in real-time, both in terms of authentication accuracy and resource usage. We show that it does not bring extra overhead in terms of power and memory usage compared to the original banking application and we can achieve a 90% true positive recognition rate, on average.