We have developed a three-dimensional (3D) dynamic integral-imaging (InIm)-system-based optical see-through augmented reality display with enhanced depth range of a 3D augmented image. A focus-tunable lens is adopted in the 3D display unit to relay the elemental images with various positions to the micro lens array. Based on resolution priority integral imaging, multiple lenslet image planes are generated to enhance the depth range of the 3D image. The depth range is further increased by utilizing both the real and virtual 3D imaging fields. The 3D reconstructed image and the real-world scene are overlaid using an optical see-through display for augmented reality. The proposed system can significantly enhance the depth range of a 3D reconstructed image with high image quality in the micro InIm unit. This approach provides enhanced functionality for augmented information and adjusts the vergence-accommodation conflict of a traditional augmented reality display.

Multidimensional optical imaging systems for information processing and visualization technologies have numerous applications in fields such as manufacturing, medical sciences, entertainment, robotics, surveillance, and defense. Among different three-dimensional (3-D) imaging methods, integral imaging is a promising multiperspective sensing and display technique. Compared with other 3-D imaging techniques, integral imaging can capture a scene using an incoherent light source and generate real 3-D images for observation without any special viewing devices. This review paper describes passive multidimensional imaging systems combined with different integral imaging configurations. One example is the integral-imaging-based multidimensional optical sensing and imaging systems (MOSIS), which can be used for 3-D visualization, seeing through obscurations, material inspection, and object recognition from microscales to long range imaging. This system utilizes many degrees of freedom such as time and space multiplexing, depth information, polarimetric, temporal, photon flux and multispectral information based on integral imaging to record and reconstruct the multidimensionally integrated scene. Image fusion may be used to integrate the multidimensional images obtained by polarimetric sensors, multispectral cameras, and various multiplexing techniques. The multidimensional images contain substantially more information compared with two-dimensional (2-D) images or conventional 3-D images. In addition, we present recent progress and applications of 3-D integral imaging including human gesture recognition in the time domain, depth estimation, mid-wave-infrared photon counting, 3-D polarimetric imaging for object shape and material identification, dynamic integral imaging implemented with liquid-crystal devices, and 3-D endoscopy for healthcare applications.

Conventional polarimetric imaging may perform poorly in photon-starved environments. In this Letter, we demonstrate the potential of integral imaging and dedicated algorithms for extracting three-dimensional (3D) polarimetric information in low light, and reducing the effects of measurement uncertainty. In our approach, the Stokes polarization parameters are measured and statistically analyzed in low illumination conditions through 3D-reconstructed polarimetric images with dedicated algorithms to improve the signal-to-noise ratio (SNR). The 3D volumetric degree of polarization (DoP) of the scene is calculated by statistical algorithms. We show that the 3D polarimetric information of the object can be statistically extracted from the Stokes parameters and 3D DoP images. Experimental results along with a novel statistical analysis verify the feasibility of the proposed approach for polarimetric 3D imaging in photon-starved environments and show that it outperforms its two-dimensional counterpart in terms of SNR. To the best of our knowledge, this is the first report of novel optical experiments along with novel statistical analysis and dedicated algorithms to recover 3D polarimetric imaging signatures in low light.

We propose an underwater optical signal detection system based on multi-dimensional integral imaging with spatially distributed multiple light sources and four-dimensional (4D) spatial-temporal correlation. We demonstrate our system for the detection of optical signals in turbid water. A 4D optical signal is generated from a three-dimensional (3D) spatial distribution of underwater light sources, which are temporally encoded using spread spectrum techniques. The optical signals are captured by an array of cameras, and 3D integral imaging reconstruction is performed, followed by multi-dimensional correlation to detect the optical signal. Inclusion of multiple light sources located at different depths allows for successful signal detection at turbidity levels not feasible using only a single light source. We consider the proposed system under varied turbidity levels using both Pseudorandom and Gold Codes for temporal signal coding. We also compare the effectiveness of the proposed underwater optical signal detection system to a similar system using only a single light source and compare between conventional and integral imaging-based signal detection. The underwater signal detection capabilities are measured through performance-based metrics such as receiver operating characteristic (ROC) curves, the area under the curve (AUC), and the number of detection errors. Furthermore, statistical analysis, including Kullback-Leibler divergence and Bhattacharya distance, shows improved performance of the proposed multi-source integral imaging underwater system. The proposed integral-imaging based approach is shown to significantly outperform conventional imaging-based methods.

Automated human gesture recognition is receiving significant research interest, with applications ranging from novel acquisition techniques to algorithms, data processing, and classification methodologies. This tutorial presents an overview of the fundamental components and basics of the current 3D optical image acquisition technologies for gesture recognition, including the most promising algorithms. Experimental results illustrate some examples of 3D integral imaging, which are compared to conventional 2D optical imaging. Examples of classifying human gestures under normal and degraded conditions, such as low illumination and the presence of partial occlusions, are provided. This tutorial is aimed at an audience who may or may not be familiar with gesture recognition approaches, current 3D optical image acquisition techniques, and classification algorithms and methodologies applied to human gesture recognition.

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.