Light in flight was captured by a single shot of a newly developed backside-illuminated multi-collection-gate image sensor at a frame interval of 10 ns without high-speed gating devices such as a streak camera or post data processes. This paper reports the achievement and further evolution of the image sensor toward the theoretical temporal resolution limit of 11.1 ps derived by the authors. The theoretical analysis revealed the conditions to minimize the temporal resolution. Simulations show that the image sensor designed following the specified conditions and fabricated by existing technology will achieve a frame interval of 50 ps. The sensor, 200 times faster than our latest sensor will innovate advanced analytical apparatuses using time-of-flight or lifetime measurements, such as imaging TOF-MS, FLIM, pulse neutron tomography, PET, LIDAR, and more, beyond these known applications.

The paper summarizes the evolution of the Backside-Illuminated Multi-Collection-Gate (BSI MCG) image sensors from the proposed fundamental structure to the development of a practical ultimate-high-speed silicon image sensor. A test chip of the BSI MCG image sensor achieves the temporal resolution of 10 ns. The authors have derived the expression of the temporal resolution limit of photoelectron conversion layers. For silicon image sensors, the limit is 11.1 ps. By considering the theoretical derivation, a high-speed image sensor designed can achieve the frame rate close to the theoretical limit. However, some of the conditions conflict with performance indices other than the frame rate, such as sensitivity and crosstalk. After adjusting these trade-offs, a simple pixel model of the image sensor is designed and evaluated by simulations. The results reveal that the sensor can achieve a temporal resolution of 50 ps with the existing technology.

The paper presents an ultra-high-speed image sensor for motion pictures of reproducible events emitting very weak light. The sensor is backside-illuminated. Each pixel is equipped with the multiple collection gates (MCG) at the center of the front side. Each collection gate is connected to an in-pixel large memory unit, which can accumulate image signals captured by repetitive imaging. The combination of the backside illumination, image signal accumulation, and slow readout from the in-pixel signal storage after an image capturing operation offers a very high sensitivity. Pipeline signal transfer from the MCG to the in-pixel memory units enables the sensor to achieve a large frame count and a very high frame rate at the same time. A test sensor was fabricated with a pixel count of 32 × 32 pixels. Each pixel is equipped with four collection gates, each connected to a memory unit with 305 elements; thus, with a total frame count of 1220 (305 × 4) frames. The test camera achieved 25 Mfps, while the sensor was designed to operate at 50 Mfps.

The theoretical temporal resolution limit $t_T$ of a silicon photodiode (Si PD) is 11.1 ps. We call “super temporal resolution” the temporal resolution that is shorter than that limit. To achieve this resolution, Germanium is selected as a candidate material for the photodiode (Ge PD) for visible light since the absorption coefficient of Ge for the wavelength is several tens of times higher than that of Si, allowing a very thin PD. On the other hand, the saturation drift velocity of electrons in Ge is about 2/3 of that in Si. The ratio suggests an ultra-short propagation time of electrons in the Ge PD. However, the diffusion coefficient of electrons in Ge is four times higher than that of Si. Therefore, Monte Carlo simulations were applied to analyze the temporal resolution of the Ge PD. The estimated theoretical temporal resolution limit is 0.26 ps, while the practical limit is 1.41 ps. To achieve a super temporal resolution better than 11.1 ps, the driver circuit must operate at least 100 GHz. It is thus proposed to develop, at first, a short-wavelength infrared (SWIR) ultra-high-speed image sensor with a thicker and wider Ge PD, and then gradually decrease the size along with the progress of the driver circuits.

In contemporary scenarios, privacy is paramount, especially in applications such as video interviews. This study introduces a privacy-focused real-time face and hair swapping method designed to conceal identity while retaining essential facial attributes of the original subject. Unlike conventional face swapping methods that rely on a reference image of a realistic person, our proposed method eliminates this need for enhanced privacy. Instead, this work introduces the use of a synthetic face image generated by a Generative Adversarial Networks (GANs), offering a secure solution that addresses the heightened importance of privacy in face and hair swapping applications, particularly in sensitive contexts like interviews. The proposed method in this study is an efficient 3-phase pipeline capable of performing these operations in real-time from a standard camera. This novel approach ensures a seamless integration of anonymity and attribute preservation, paving the way for more discreet and privacy-centric real-time face and hair swapping technologies, not only in video interviews but also in entertainment and communication applications such as video teleconferencing and streaming media.

In recent years, deep learning methodologies have been increasingly applied to the intricate challenges of visual-inertial odometry (VIO), especially in scenarios with rapid movements and scenes lacking clear structure. This paper introduces a novel hybrid approach that leverages the inherent strengths of traditional VIO techniques, while harnessing the potential of advanced machine learning technologies. By seamlessly integrating an iterated extended Kalman filter with deep learning techniques, our approach systematically takes into account uncertainties, thereby enhancing the overall reliability and robustness of the system. The proposed algorithm has been rigorously evaluated on the KITTI and EuroC datasets, outperforming other deep learning VIO methods. It achieved a translation error of 2.28% and a rotation error of 0.226 degrees per 100 meters on the KITTI odometry dataset.