Deep learning image processing models have had remarkable success in recent years in generating high quality images. Particularly, the Improved Denoising Diffusion Probabilistic Models (DDPM) have shown superiority in image quality to the state-of-the-art generative models, which motivated us to investigate their capability in the generation of the synthetic electrocardiogram (ECG) signals. In this work, synthetic ECG signals are generated by the Improved DDPM and by the Wasserstein GAN with Gradient Penalty (WGAN-GP) models and then compared. To this end, we devise a pipeline to utilize DDPM in its original 2D form. First, the 1 D ECG time series data are embedded into the 2 D space, for which we employed the Gramian Angular Summation/Difference Fields (GASF/GADF) as well as Markov Transition Fields (MTF) to generate three 2 D matrices from each ECG time series, which when put together, form a 3-channel 2 D datum. Then 2 D DDPM is used to generate 2 D 3-channel synthetic ECG images. The 1 D ECG signals are created by de-embedding the 2 D generated image files back into the 1 D space. This work focuses on unconditional models and the generation of Normal Sinus Beat ECG signals exclusively, where the Normal Sinus Beat class from the MIT-BIH Arrhythmia dataset is used in the training phase. The quality , distribution , and the authenticity of the generated ECG signals by each model are quantitatively evaluated and compared. Our results show that in the proposed pipeline and in the particular setting of this paper, the WGAN-GP model is consistently superior to DDPM in all the considered metrics.

In movie productions, the Foley artist is responsible for creating an overlay sound track that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener, thereby reinforcing the director's intention for a given scene. The artist must decide what artificial sound captures the essence of both the sound and action depicted in the scene. In this paper, we present AutoFoley, a fully automated deep-learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced sound track. An important performance criterion of the synthesized sound track is that it can be time-synchronized with the input video, which provides a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as interframe relations in fast-moving video clips by incorporating an interpolation technique and temporal relational networks (TRN). We employ a robust multiscale recurrent neural network (RNN) associated with a convolutional neural network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large-scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. While the Foley dataset was limited to short-duration videos representing focused representative activities, this dataset demonstrates the capabilities of our proposed system. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley shows that more than 73% of the test subjects considered the generated sound track as original, which is a noteworthy improvement in cross-modal research in sound synthesis.

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of the visual and audio features. In this paper we introduce a novel method for guiding a class-conditioned GAN to synthesize representative audio with temporally extracted visual information. We accomplish this visual-to-sound generation task by adapting the synchronicity traits between the audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading to the generation of visually aligned realistic soundtracks. We expanded our previously proposed Automatic Foley data set. We evaluated FoleyGAN's synthesized sound output through human surveys that show noteworthy (on average 81%) audio-visual synchronicity performance. Our approach outperforms other baseline models and audio-visual data sets in statistical and ablation experiments achieving improved IS, FID and NDB scores. In ablation analysis we showed the significance of our visual and temporal feature extraction method as well as augmented performance of our generation network. Overall, our FoleyGAN model showed sound retrieval accuracy of 76.08% surpassing existing visual-to-audio synthesis deep neural networks.

Formation control has been a popular research topic in recent years. This is primarily due to the the use of formation control to coordinate the movement of a swarm of robots. In this paper, with the primary objective of guaranteeing image overlap, a novel bird flocking inspired formation control algorithm was developed to control a swarm of unmanned aerial vehicles (UAVs). Our approach uses only stereo cameras, global positioning system (GPS), and inertial measurement units (IMUs), and removes the need of feature or pattern matching. By utilizing stereo cameras, virtual tracking points can be calculated. The convergence of these tracking points are chosen as the primary formation control goal because if the virtual tracking points are converged, it is guaranteed that the field of view of the cameras of the various UAVs always have some overlap. This allows for different types of stitching methods to be used to form a larger image with no gaps, depending on the configuration of tracking points. Five different control algorithms were developed: direct method, proportional integral derivative, bang-bang, short window model predictive, and long window model predictive. The formation controller was tested in three simulation environments to verify the robustness and the effectiveness of our algorithm.

A great number of computer vision publications have focused on distinguishing between human action recognition and classification rather than the intensity of actions performed. Indexing the intensity which determines the performance of human actions is a challenging task due to the uncertainty and information deficiency that exists in the video inputs. To remedy this uncertainty, in this paper we coupled fuzzy logic rules with the neural-based action recognition model to rate the intensity of a human action as intense or mild. In our approach, we used a Spatio-Temporal LSTM to generate the weights of the fuzzy-logic model, and then demonstrate through experiments that indexing of the action intensity is possible. We analyzed the integrated model by applying it to videos of human actions with different action intensities, and were able to achieve an accuracy of 89.16% on our intensity indexing generated dataset. The integrated model demonstrates the ability of a neuro-fuzzy inference module to effectively estimate the intensity index of human actions.

Researchers have extensively explored indoor localization in recent years. Robotic applications often require precise positioning, which is difficult when access to a GPS is limited or compromised. Many of the existing approaches require either prior knowledge of the local environment and fixed landmarks or complex and expensive hardware to achieve the necessary degree of accuracy. In this article, we examine the use of the HTC virtual reality hardware along with our novel approach, VIVEPOSE, to perform indoor robotic localization. We then compare the accuracy of our proposed indoor localization system to current approaches that use traditional odometry sensor-based localization. Finally, we present and demonstrate a leader-follower approach using VIVEPOSE and show how the HTC Vive tracker can be successful for indoor localization for a robotics application.