In this paper, we propose a new depth image-based rending (DIBR) scheme for 3DTV applications, where a background sprite model is utilized for dis-occlusion/hole filling purpose. Dissimilar to traditional spatial (e.g., interpolation or inpainting) and temporal methods, our algorithm is capable of recovering the holes with true background information by incrementally integrating the spatial and temporal information of the video in a unified background sprite model. The technique of background sprite model construction in this paper is featured of resolving camera motions existing in most of the consumer videos, accurate registration of multi-frame information, and efficient memory use and computation in realistic 3DTV application. To register/stitch each input frame to the background sprite model accurately, foreground removal considering color and depth information and an adaptive key-frame-based scheme are developed for transform computation. Experimental results show that our proposed scheme has a large temporal reference distance and can retrieve true background information accurately, thus leading to better quality after novel view synthesis compared to existing spatial or spatio-temporal algorithms, especially for videos with significant camera motions or complex backgrounds.

This paper presents an RGB-NIR (Near Infrared) dual-modality technique to analyze the remote photoplethysmogram (rPPG) signal and hence estimate the heart rate (in beats per minute), from a facial image sequence. Our main innovative contribution is the introduction of several denoising techniques such as Modified Amplitude Selective Filtering (MASF), Wavelet Decomposition (WD), and Robust Principal Component Analysis (RPCA), which take advantage of RGB and NIR band characteristics to uncover the rPPG signals effectively through this Independent Component Analysis (ICA)-based algorithm. Two datasets, of which one is the public PURE dataset and the other is the CCUHR dataset built with a popular Intel RealSense D435 RGB-D camera, are adopted in our experiments. Facial video sequences in the two datasets are diverse in nature with normal brightness, under-illumination (i.e., dark), and facial motion. Experimental results show that the proposed method has reached competitive accuracies among the state-of-the-art methods even at a shorter video length. For example, our method achieves MAE = 4.45 bpm (beats per minute) and RMSE = 6.18 bpm for RGB-NIR videos of 10 and 20 s in the CCUHR dataset and MAE = 3.24 bpm and RMSE = 4.1 bpm for RGB videos of 60-s in the PURE dataset. Our system has the advantages of accessible and affordable hardware, simple and fast computations, and wide realistic applications.

For bedridden elderly people, pressure ulcer is the most common and serious complication and could be prevented by regular repositioning. However, due to a shortage of long-term care workers, repositioning might not be implemented as often as required. Posture monitoring by using modern health/medical caring technology can potentially solve this problem. We propose a RGB-D camera system to recognize the posture of the bedridden elderly patients based on the analysis of 3D human skeleton which consists of articulated joints. Since practically most bedridden patients were covered with a blanket, only four 3D joints were used in our system. After the recognition of the posture, a warning message will be sent to the caregiver for assistance if the patient stays in the same posture for more than a predetermined period (e.g., two hours). Experimental results indicate that our proposed method is capable of achieving a high accuracy in posture recognition (above 95%). To the best of our knowledge, this application of using human skeleton analysis for patient care is novel. The proposed scheme is promising for clinical applications and will undertake an intensive test in health care facilities in the near future after redesigning a proper RGB-D (Red-Green-Blue-Depth) camera system. In addition, a desktop computer can be used for multi-point monitoring to reduce cost, since real-time processing is not required in this application.

In this article, an automatic stereoscopic video conversion scheme which accepts MPEG-encoded videos as input is proposed. Our scheme is depth-based, relying on spatio-temporal analysis of the decoded video data to yield depth perception cues, such as temporal motion and spatial contrast, which reflect the relative depths between the foreground and the background areas. Our scheme is shot-adaptive, demanding that shot change detection and shot classification be performed for tuning of algorithm or parameters that are used for depth cue combination. The above-mentioned depth estimation is initially block-based, followed by a locally adaptive joint trilateral upsampling algorithm to reduce the computing load significantly. A recursive temporal filter is used to reduce the possible depth fluctuations (and also artifacts in the synthesized images) resulting from wrong depth estimations. The traditional Depth-Image-Based-Rendering algorithm is used to synthesize the left- and right-view frames for 3D display. Subjective tests show that videos converted by our scheme provide comparable perceived depth and visual quality with those converted from the depth data calculated by stereo vision techniques. Also, our scheme is shown to outperform the well-known TriDef software in terms of human’s perceived 3D depth. Based on the implementation by using “OpenMP” parallel programming model, our scheme is capable of executing in real-time on a multi-core CPU platform.

Eldering caring is important in today’s aging society, especially that accident anticipation/prevention plays an important role. In this paper, a novel approach to preventing elderly accidents based on a very low-resolution thermopile sensor array (TPA) (only 32×32 pixels) is proposed for prediction of bed-exit event that might lead to elderly falls in home caring. Low-resolution TPA sensor, capable of collecting far infrared energy, ensures cost-effective monitoring, no interference with user’s daily life, and most importantly privacy-preservation. Since most of the fall accidents occur when the elderly attempts to get off the bed without assistance, it is thus the focus of this paper to monitor his/her posture and action via TPA image sensor and then predict that an action of getting off the bed will occur in a near future (e.g., S seconds later). Our system can raise an alarm to the caregivers so that they can intervene and offer the necessary assistance. A deep-learning model based on CNN-RNN (Convolutional neural network-Recurrent neural network) architecture was designed which is capable of predicting the elderly bed-exit intention by S = 5.78 seconds in advance of the action onset at an accuracy of 99.37% according to our dataset evaluation. Our system is also suitable for on-line real-time operation which will be helpful to elderly caring in our society.

In computer vision, accurately estimating a 3D human skeleton from a single RGB image remains a challenging task. Inspired by the advantages of multi-view approaches, we propose a method of predicting enhanced 2D skeletons (specifically, predicting the joints’ relative depths) from multiple virtual viewpoints based on a single real-view image. By fusing these virtual-viewpoint skeletons, we can then estimate the final 3D human skeleton more accurately. Our network consists of two stages. The first stage is composed of a two-stream network: the Real-Net stream predicts 2D image coordinates and the relative depth for each joint from the real viewpoint, while the Virtual-Net stream estimates the relative depths in virtual viewpoints for the same joints. Our network’s second stage consists of a depth-denoising module, a cropped-to-original coordinate transform (COCT) module, and a fusion module. The goal of the fusion module is to fuse skeleton information from the real and virtual viewpoints so that it can undergo feature embedding, 2D-to-3D lifting, and regression to an accurate 3D skeleton. The experimental results demonstrate that our single-view method can achieve a performance of 45.7 mm on average per-joint position error, which is superior to that achieved in several other prior studies of the same kind and is comparable to that of other sequence-based methods that accept tens of consecutive frames as the input.