This paper presents a novel dual-camera system that can simultaneously capture zoomed-in images using an ultrafast pan-tilt camera and a fixed wide-view camera using deep learning methods. An ultrafast pan-tilt camera can function as multiple virtual pan-tilt cameras by synchronizing a high-frame-rate zooming-view camera and an ultrafast pan-tilt mirror device that can switch over 500 different views in a second. A wide-view camera can obtain images in a fixed view in which multiple targets to be tracked are captured in low resolution and then recognized by processing the images using deep learning methods at a rate of dozens of frames per second. Based on the positions of all targets, recognized by the wide-view camera, the pan and tilt angles of multiple pan-tilt cameras are virtually controlled using an ultrafast pan-tilt camera through multithread viewpoint control to simultaneously capture the zoomed-in images of all targets. Our developed system can operate 10 virtual pan-tilt cameras (25 fps) with multithread viewpoint control and 4 ms time granularity in synchronization with convolutional neural-network-based recognition model operating at 25 fps, which is accelerated by a general-purpose computing on graphics processing units. The effectiveness of our system was demonstrated by the results of several experiments conducted on simultaneous zoom shooting of multiple running objects (persons and cars) in the range of approximately 80 m or higher in a natural outdoor scene, which was formerly too wide for a single fixed camera to capture clearly and simultaneously.

This study proposes a visual tracking system with 500-fps image processing that can detect and track a fast-moving appearance-varying target in a 360-degree-field of view (FOV) from high-spatial-resolution images. The system consists of a high-speed camera, a pan-tilt Galvano-mirror system, and a mirror-drive high-speed pan control system with 360-degree-rotational capacity. Omnidirectional visual tracking of a remote object using complex visual features becomes possible due to the acquisition of high-spatial-resolution images of the target object via the system. We introduce an intelligent high frame rate (HFR) target-tracking algorithm that can simultaneously detect and track an appearance-varying object against a complex background for high-speed visual feedback control. Furthermore, the proposed surveillance system enables robust visual tracking under low-light conditions using a 3-D light detection and ranging (LiDAR) sensor system and a simultaneously controlled active spotlight projection system. The effectiveness of the proposed omnidirectional visual tracking system was demonstrated by conducting several omnidirectional detecting and tracking experiments on persons dozens of meters away using a 25-mm focal length lens under both daytime and night-time conditions.

This paper presents a novel concept of real-time catadioptric stereo tracking using a single ultrafast mirror-drive pan-tilt active vision system that can simultaneously switch between hundreds of different views in a second. By accelerating video-shooting, computation, and actuation at the millisecond-granularity level for time-division multithreaded processing in ultrafast gaze control, the active vision system can function virtually as two or more tracking cameras with different views. It enables a single active vision system to act as virtual left and right pan-tilt cameras that can simultaneously shoot a pair of stereo images for the same object to be observed at arbitrary viewpoints by switching the direction of the mirrors of the active vision system frame by frame. We developed a monocular galvano-mirror-based stereo tracking system that can switch between 500 different views in a second, and it functions as a catadioptric active stereo with left and right pan-tilt tracking cameras that can virtually capture 8-bit color $512\times 512$ images each operating at 250 fps to mechanically track a fast-moving object with a sufficient parallax for accurate 3D measurement. Several tracking experiments for moving objects in 3D space are described to demonstrate the performance of our monocular stereo tracking system.

This study develops an omnidirectional panoramic video system that can simultaneously observe zoom-in images at dozens of different viewpoints in a 360-degree field of view by synchronizing hundreds of frames per second video shooting, with a rotation of tens-of-rotation-per-second mirror-rotator. When video shooting a scene with a certain exposure utilizing a mirror-rotator, apparent motion in the captured images becomes too large to observe non-blurred images, owing to its high-speed rotation. To reduce motion blurs caused by the rotation of a mirror-rotator, our system introduces an ultrafast pan-tilt mirror mechanism that can cancel apparent motion in images by executing frame-by-frame viewpoint control, so that the apparent speed in images becomes zero when the camera shutter is open, and it returns to its home position when the shutter is closed. Our system can capture 40 omnidirectionally different view images at 10 fps in real time by synchronizing 400-fps video shooting with frame-by-frame ultrafast pan-tilt mirror control for motion blur reduction when a mirror-rotator rotates at 10 rps. Its performance is verified by experimental results for several omnidirectional real scenes, and our system can capture clear and bright panoramic zoom-in images without decreasing the camera exposure when omnidirectionally observing dozens-of-meters-distance outdoor scenes.

This study proposes a visual tracking system that can detect and track multiple fast-moving appearance-varying targets simultaneously with 500 fps image processing. The system comprises a high-speed camera and a pan-tilt galvanometer system, which can rapidly generate large-scale high-definition images of the wide monitored area. We developed a CNN-based hybrid tracking algorithm that can robustly track multiple high-speed moving objects simultaneously. Experimental results demonstrate that our system can track up to three moving objects with velocities lower than 30 m per second simultaneously within an 8-m range. The effectiveness of our system was demonstrated through several experiments conducted on simultaneous zoom shooting of multiple moving objects (persons and bottles) in a natural outdoor scene. Moreover, our system demonstrates high robustness to target loss and crossing situations.

This paper presents a novel view and scanning-depth expansion photographic microscope using ultrafast switching mirrors and high-speed vision that can simultaneously extend both the field of view and the scanning-depth range of a typical optical microscope in real time. Two ultrafast switching mirrors that can switch at several hundred hertz are used for micro-gaze control via synchronization through a high-speed vision system. The field of view can be changed while maintaining a constant length of the optical path by adjusting the angles of the two mirrors, which ensures that the extended field of view is within the focal range. We evaluated and designed mirror angle combinations for view expansion at multiple depths, making it possible to extend view and depth in real time without considering the complex design and computing required by computational imaging micro view expansion approaches. The effectiveness of our system was demonstrated by several experiments, including micro view expansion at a single depth and multiple depths in real time. In addition, by evaluating the image quality of the view expansion image, our proposed system can automatically output always-focused view expansion images in real time even when samples move at different depths.