24 August 2022 in IEEE Geoscience and Remote Sensing Letters
Despite the enormous success of the ’pre-training and fine-tuning’ paradigm, widespread across machine learning, it has yet to pervade remote sensing (RS). To help rectify this, we pre-train a vision transformer (ViT) on 1.3 million satellite-derived RS images. We pre-train SatViT using a state-of-the-art self-supervised learning algorithm called masked autoencoding (MAE), which learns general representations by reconstructing held-out image patches. Crucially, this approach does not require annotated data, allowing us to pre-train on unlabeled images acquired from Sentinel-1 & 2. After fine-tuning, SatViT outperforms state-of-the-art ImageNet and RS-specific pre-trained models on both of our downstream tasks. We further improve overall accuracy (by 3.2% and 0.21%) by continuing to pre-train SatViT—still using MAE—on the unlabelled target datasets. Most importantly, we release our code, pre-trained model weights, and tutorials aimed at helping researchers fine-tune our models. (https://github.com/antofuller/SatViT).
13 February 2023 in IEEE Transactions on Instrumentation and Measurement
Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region of interest detection and movement detection for noncontact-based monitoring systems. In a home setting, pose detection can be used to detect sleep positions which has shown to have a strong influence on multiple health factors. However, pose monitoring during sleep is challenging due to heavy occlusions from blanket coverings and low lighting. To address this, we present a novel dataset, Simultaneously-collected multimodal Mannequin Lying pose (SMaL) dataset, for under the cover infant pose estimation. We collect depth and pressure imagery of an infant mannequin in different poses under various cover conditions. We successfully infer full body pose under the cover by training state-of-art pose estimation methods and leveraging existing multimodal adult pose datasets for transfer learning. We demonstrate a hierarchical pretraining strategy for transformer-based models to significantly improve performance on our dataset. Our best performing model was able to detect joints under the cover within 25mm 86% of the time with an overall mean error of 16.9mm. Data, code and models publicly available at https://github.com/DanielKyr/SMaL.
15 February 2023 in arXiv
Infant pose monitoring during sleep has multiple applications in both healthcare and home settings. In a healthcare setting, pose detection can be used for region of interest detection and movement detection for noncontact based monitoring systems. In a home setting, pose detection can be used to detect sleep positions which has shown to have a strong influence on multiple health factors. However, pose monitoring during sleep is challenging due to heavy occlusions from blanket coverings and low lighting. To address this, we present a novel dataset, Simultaneously-collected multimodal Mannequin Lying pose (SMaL) dataset, for under the cover infant pose estimation. We collect depth and pressure imagery of an infant mannequin in different poses under various cover conditions. We successfully infer full body pose under the cover by training state-of-art pose estimation methods and leveraging existing multimodal adult pose datasets for transfer learning. We demonstrate a hierarchical pretraining strategy for transformer-based models to significantly improve performance on our dataset. Our best performing model was able to detect joints under the cover within 25mm 86% of the time with an overall mean error of 16.9mm. Data, code and models publicly available at https://github.com/DanielKyr/SMaL
9 December 2024 in IEEE Geoscience and Remote Sensing Letters
Cloud properties underpin accurate climate modeling and are often derived from the individual particles comprising a cloud. Studying these cloud particles is challenging due to their intricate shapes, called “habits”, and manual classification via probe-generated images is time-consuming and subjective. We propose a novel method for habit representation learning that uses minimal labeled data by leveraging self-supervised learning (SSL) with Vision Transformers on a newly acquired dataset of 124, 000 images captured by the novel High-Volume Precipitation Spectrometer ver. 4 (HVPS-4) probe. Our approach significantly outperforms ImageNet pretraining by 48% on a 293-sample annotated dataset. Notably, we present the first SSL scheme for learning habit representations, leveraging data collected in-flight from the probe. Our results demonstrate that self-supervised pretraining significantly improves habit classification even when using single-channel HVPS-4 data. We achieve further gains using sequential views and a soft contrastive objective tailored for sequential, in-flight measurements. Our work paves the way for applying SSL to multi-view and multi-scale data from advanced cloud-particle imaging probes, enabling comprehensive characterization of the flight environment. We publicly release data, code, and models associated with this study.