This paper describes a low-cost, robust, and accurate remote eye-tracking system that uses an industrial prototype smartphone with integrated infrared illumination and camera. Numerous studies have demonstrated the beneficial use of eye-tracking in domains such as neurological and neuropsychiatric testing, advertising evaluation, pilot training, and automotive safety. Remote eye-tracking on a smartphone could enable the significant growth in the deployment of applications in these domains. Our system uses a 3D gaze-estimation model that enables accurate point-of-gaze (PoG) estimation with free head and device motion. To accurately determine the input eye features (pupil center and corneal reflections), the system uses Convolutional Neural Networks (CNNs) together with a novel center-of-mass output layer. The use of CNNs improves the system’s robustness to the significant variability in the appearance of eye-images found in handheld eye trackers. The system was tested with 8 subjects with the device free to move in their hands and produced a gaze bias of 0.72°. Our hybrid approach that uses artificial illumination, a 3D gaze-estimation model, and a CNN feature extractor achieved an accuracy that is significantly (400%) better than current eye-tracking systems on smartphones that use natural illumination and machine-learning techniques to estimate the PoG.

The most accurate remote Point of Gaze (PoG) estimation methods that allow free head movements use infrared light sources and cameras together with gaze estimation models. Current gaze estimation models were developed for desktop eye-tracking systems and assume that the relative roll between the system and the subjects’ eyes (the ’R-Roll’) is roughly constant during use. This assumption is not true for hand-held mobile-device-based eye-tracking systems. We present an analysis that shows the accuracy of estimating the PoG on screens of hand-held mobile devices depends on the magnitude of the R-Roll angle and the angular offset between the visual and optical axes of the individual viewer. We also describe a new method to determine the PoG which compensates for the effects of R-Roll on the accuracy of the POG. Experimental results on a prototype infrared smartphone show that for an R-Roll angle of ${90}^{°}$ , the new method achieves accuracy of approximately $1^{°}$ , while a gaze estimation method that assumes that the R-Roll angle remains constant achieves an accuracy of ${3.5}^{°}$ . The manner in which the experimental PoG estimation errors increase with the increase in the R-Roll angle was consistent with the analysis. The method presented in this paper can improve significantly the performance of eye-tracking systems on hand-held mobile-devices.

The presence of cameras and powerful computers on modern mobile devices gives rise to the hope that they can perform computer vision tasks as we walk around. However, the computational demand and energy consumption of computer vision tasks such as object detection, recognition and tracking make this challenging. At the same time, a fixed vision hard core on the SoC contained in a mobile chip may not have the flexibility needed to adapt to new situations, or evolve as new algorithms are discovered. This may mean that computer vision on a mobile device is the killer application for FPGAs, and could motivate the inclusion of FPGAs, in some form, within modern smartphones. In this paper we present a novel hardware architecture for object detection, that is bit-for-bit compatible with the object classifiers in the widely-used open source OpenCV computer vision software. The architecture is novel, compared to prior work in this area, in two ways: its memory architecture, and its particular SIMD-type of processing. The implementation, which consists of the full system, not simply the kernel, outperforms a same-generation technology mobile processor by a factor of 59 times, and is 13.5 times more energy-efficient.

Aerial-photo interpreted inventories of forest resources, including tree species composition, are valuable in forest resource management, but are expensive to create and can be relatively inaccurate. Because of differences among tree species in their spectral properties and seasonal phenologies, it might be possible to improve such forest resource inventory information (FRI) by using it in concert with multispectral satellite information from multiple time periods. We used Sentinel-2 information from nine spectral bands and 12 dates within a two-year period to model multivariate percent tree species composition in >51,000 forest stands in the FRI of south-central Ontario, Canada. Accuracy of random forest (RF) and convolutional neural network (CNN) predictions were tested using species-specific basal area information from 155 0.25-ha field plots. Additionally, we created models using the Sentinel-2 information in concert with the field data and compared the accuracy of these models and the FRI-based models by use of basal areas from a second (13.7-ha) field data set. Based on average R² values across species in the two field data sets, the Sentinel-FRI models outperformed the FRI, showing 1.5- and 1.7-fold improvements relative to the FRI for RF and 2.1- and 2.2-fold improvements for CNN (mean R²: 0.141–0.169 (FRI); 0.217–0.295 (RF); 0.307–0.352 (CNN)). Models created with the field data performed even better: improvements relative to the FRI were 2.1-fold for RF and 2.8-fold for CNN (mean R²: 0.169 (FRI); 0.356 (RF); 0.469 (CNN)). As predicted, R² values between FRI- and field-trained predictions were higher than R² values with the FRI. Of the 21 tree species evaluated, 8 relatively rare species had poor models in all cases. Our multivariate approach allowed us to use more FRI stands in model creation than if we had been restricted to stands dominated by single species and allowed us to map species abundances at higher resolution. It might be possible to improve models further by use of tree stem maps and incorporation of the effects of canopy disturbances.