In movie productions, the Foley artist is responsible for creating an overlay sound track that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener, thereby reinforcing the director's intention for a given scene. The artist must decide what artificial sound captures the essence of both the sound and action depicted in the scene. In this paper, we present AutoFoley, a fully automated deep-learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced sound track. An important performance criterion of the synthesized sound track is that it can be time-synchronized with the input video, which provides a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as interframe relations in fast-moving video clips by incorporating an interpolation technique and temporal relational networks (TRN). We employ a robust multiscale recurrent neural network (RNN) associated with a convolutional neural network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large-scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. While the Foley dataset was limited to short-duration videos representing focused representative activities, this dataset demonstrates the capabilities of our proposed system. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley shows that more than 73% of the test subjects considered the generated sound track as original, which is a noteworthy improvement in cross-modal research in sound synthesis.

Deep learning based visual-to-sound generation systems have been developed that identify and create audio features from video signals. However, these techniques often fail to consider the time-synchronicity of the visual and audio features. In this paper we introduce a novel method for guiding a class-conditioned GAN to synthesize representative audio with temporally extracted visual information. We accomplish this visual-to-sound generation task by adapting the synchronicity traits between the audio-visual modalities. Our proposed FoleyGAN model is capable of conditioning action sequences of visual events leading to the generation of visually aligned realistic soundtracks. We expanded our previously proposed Automatic Foley data set. We evaluated FoleyGAN's synthesized sound output through human surveys that show noteworthy (on average 81%) audio-visual synchronicity performance. Our approach outperforms other baseline models and audio-visual data sets in statistical and ablation experiments achieving improved IS, FID and NDB scores. In ablation analysis we showed the significance of our visual and temporal feature extraction method as well as augmented performance of our generation network. Overall, our FoleyGAN model showed sound retrieval accuracy of 76.08% surpassing existing visual-to-audio synthesis deep neural networks.

The Internet of Things has enabled a wide variety of new applications not possible only a short time ago. Sensing data found at the Edge of the network, close to the environment where people are found, is a critical component with many modern applications. Often restrictions at the device level or on the available bandwidth limit the ability to capture all locally available data required for processing and analysis. In this research, we present a novel method for extracting sound from video data where no original sound was present. Our novel method of sound synthesis first uses the image features output from a Convolutional Neural Network (CNN) to determine class prediction weights using an advanced Long Short Term Memory (LSTM) network. A Generative Adversarial Network (GAN) is then used to generate the representative sound of the predicted class for the input video sample. By combining the output of many Auto Sound Generators in a System of Systems framework, we show that new applications emerge that were never before possible.

Self Organized Data Aggregation Technique applying Evolutionary Game Theory is one of the most effective methods to select the optimum strategy for Delay Tolerant Network (DTN). In DTN, reliable end-to-end data transfer is provided by custody transfer mechanism where special nodes (custodians) exchange information with custody following hop-by-hop manner. For multi cluster Delay Tolerant Network, message ferrying approach is considered here. Now, the strategy selection (to become an aggregator or sender) is considered as a game in Evolutionary Game Theory, where, each node will draw result after interaction. For strategy selection, Imitation Update Rule of Evolutionary Game Theory has been utilized. We will focus mainly on the Birth-Death and Death-Birth Update Rules of Evolutionary Game Theory and select the better Update rules among three update rules.

In movie productions, the Foley Artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener thereby reinforcing the Directors's intention for a given scene. In this paper, we present AutoFoley, a fully-automated deep learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in the applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced soundtrack. An important performance criterion of the synthesized soundtrack is to be time-synchronized with the input video, which provides for a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as inter-frame relations in fast moving video clips by incorporating an interpolation technique and Temporal Relationship Networks (TRN). We employ a robust multi-scale Recurrent Neural Network (RNN) associated with a Convolutional Neural Network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley show over 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in cross-modal research in sound synthesis.