There is a research field of human activity recognition that automatically recognizes a user's physical activity through sensing technology incorporated in smartphones and other devices. When sensing daily activity, various measurement conditions, such as device type, possession method, wearing method, and measurement application, are often different depending on the user and the date of the measurement. Models that predict activity from sensor values are often implemented by machine learning and are trained using a large amount of activity-labeled sensor data measured from many users who provide labeled sensor data. However, collecting activity-labeled sensor data using each user's individual smartphones causes data being measured in inconsistent environments that may degrade the estimation accuracy of machine learning. In this study, I propose an activity recognition method that is robust to different sampling rates - even in the measurement environment. The proposed method applies an adversarial network and data augmentation by downsampling to a common activity recognition model to achieve the acquisition of feature representations that make the sampling rate unspecifiable. Using the Human Activity Sensing Consortium (HASC), which is a dataset of basic activity recognition using smartphone sensors, I conducted an evaluation experiment to simulate an environment in which various sampling rates were measured. As a result, I found that estimation accuracy was reduced by the conventional method in the above environment and could be improved by my proposed method.

The feature extraction of human activity recognition (HAR) based on sensor data has been studied as a hand-crafted method. The significant feature extraction ability is a key factor in improving the accuracy of HAR. Recently, deep learning methods have been employed for feature extraction. In this paper, we review previous studies on deep learning methods in HAR and discuss suitable models for feature extraction. First, we applied various convolutional neural networks to clarify the effective architecture for HAR. Afterward, we developed advanced models by embedding submodules, such as self-attention and recurrent neural networks, often adopted in recent studies. Comparative experiments on HASC, UCI, and WISDM public datasets showed that Inception-V3, which used cross-channel multi-size convolution transformation, outperformed other backbones. Through comparative experiments after embedding submodules, submodules do not always have a positive effect on accuracy. Compared with other submodules, SENet has a positive effect. We conclude that it is essential to select an appropriate backbone model before applying the submodules, and submodules are unnecessary in some cases.

Deep learning (DL) models for sensor-based human activity recognition (HAR) are still in their nascent stages compared with image recognition. HAR’s inference is generally implemented on edge devices such as smartphones because of the secure privacy. However, lightweight DL models for HAR while meeting the hardware limitations are lacking. In this study, using the neural architecture search (NAS), we investigated an effective DL model architectures that can be used for inference on smartphones. We designed multiple search spaces for the type of convolution, the kernel size of the convolution process, the type of skip operation, the number of layers, and the number of output filters by Bayesian optimization. We propose models called mobile-aware convolutional neural network (CNN) for sensor-based HAR by NAS (MarNASNets). We constructed four MarNASNet networks, MarNASNets-A to D, each with a different model size and a parameter search space of four patterns. Experimental results show that MarNASNets achieve the same accuracy as existing CNN architectures with fewer parameters and are effective model architectures for on-device and sensor-based HAR. We also developed Activitybench, an iOS app, for measuring model performance on smartphones, and evaluated the on-device performance of each model. MarNASNets explored achieved accuracy comparable to existing CNN models with smaller model sizes. MarNASNet-C achieved accuracies of 92.60, 94.52, and 88.92 % for HASC, UCI, and WISDM, respectively. Especially for HASC and UCI, MarNASNet-C achieved the highest accuracies despite the small model size. Their latency was also comparable to that of existing CNN models, enabling real-time on-device inference.

Sensor-based human activity recognition (HAR) is a paramount technology in the Internet of Things services. HAR using representation learning, which automatically learns a feature representation from raw data, is the mainstream method because it is difficult to interpret relevant information from raw sensor data to design meaningful features. Ensemble learning is a robust approach to improve generalization performance; however, deep ensemble learning requires various procedures, such as data partitioning and training multiple models, which are time-consuming and computationally expensive. In this study, we propose Easy Ensemble (EE) for HAR, which enables the easy implementation of deep ensemble learning in a single model. In addition, we propose various techniques (input variationer, stepwise ensemble, and channel shuffle) for the EE. Experiments on a benchmark dataset for HAR demonstrated the effectiveness of EE and various techniques and their characteristics compared with conventional ensemble learning methods.

Resource management for fisheries plays a pivotal role in fostering a sustainable fisheries industry. In Japan, resource surveys rely on manual measurements by staff, incurring high costs and limitations on the number of feasible measurements. This study endeavors to revolutionize resource surveys by implementing image-recognition technology. Our methodology involves developing a system that detects individual fish regions in images and automatically identifies crucial keypoints for accurate fish length measurements. We use grounded-segment-anything (Grounded-SAM), a foundation model for fish instance segmentation. Additionally, we employ a Mask Keypoint R-CNN trained on the fish image bank (FIB), which is an original dataset of fish images, to accurately detect significant fish keypoints. Diverse fish images were gathered for evaluation experiments, demonstrating the robust capabilities of the proposed method in accurately detecting both fish regions and keypoints.

The digitization of catch information for the promotion of sustainable fisheries is gaining momentum globally. However, the manual measurement of fundamental catch information, such as species identification, length measurement, and fish count, is highly inconvenient, thus intensifying the call for its automation. Recently, image recognition systems based on convolutional neural networks (CNNs) have been extensively studied across diverse fields. Nevertheless, the deployment of CNNs for identifying fish species is difficult owing to the intricate nature of managing a plethora of fish species, which fluctuate based on season and locale, in addition to the scarcity of public datasets encompassing large catches. To overcome this issue, we designed a transferable pre-trained CNN model specifically for identifying fish species, which can be easily reused in various fishing grounds. Utilizing an extensive fish species photographic database from a Japanese museum, we developed a transferable fish identification (TFI) model employing strategies such as multiple pre-training, learning rate scheduling, multi-task learning, and metric learning. We further introduced two application methods, namely transfer learning and output layer masking, for the TFI model, validating its efficacy through rigorous experiments.