1 August 2019 in Proceedings of the IEEE
Deep learning is currently widely used in a variety of applications, including computer vision and natural language processing. End devices, such as smartphones and Internet-of-Things sensors, are generating data that need to be analyzed in real time using deep learning or used to train deep learning models. However, deep learning inference and training require substantial computation resources to run quickly. Edge computing, where a fine mesh of compute nodes are placed close to end devices, is a viable way to meet the high computation and low-latency requirements of deep learning on edge devices and also provides additional benefits in terms of privacy, bandwidth efficiency, and scalability. This paper aims to provide a comprehensive review of the current state of the art at the intersection of deep learning and edge computing. Specifically, it will provide an overview of applications where deep learning is used at the network edge, discuss various approaches for quickly executing deep learning inference across a combination of end devices, edge servers, and the cloud, and describe the methods for training deep learning models across multiple edge devices. It will also discuss open challenges in terms of systems performance, network technologies and management, benchmarks, and privacy. The reader will take away the following concepts from this paper: understanding scenarios where deep learning at the network edge can be useful, understanding common techniques for speeding up deep learning inference and performing distributed training on edge devices, and understanding recent trends and opportunities.
3 November 2016 in IEEE Access
With the rise and widespread deployment of a vast array of wireless radio access technologies (e.g., 3G, 4G/LTE, 802.11, Bluetooth, and Femto) and the growth of interest in potential 5G technologies such as millimeter-wave radio, coupled with the rapid increase in the number of network edge devices with multiple radio interfaces, the question of network control and client-to-base-station association becomes an important issue. Older, well-studied centralized control schemes where a single computational entity harvests channel information from individual clients in order to determine optimal resource allocations for each client is no longer tenable: such methods require significant signaling overhead which does not scale well with the expected number of hundreds of thousands of smart client devices with multiple radio interfaces capable of leveraging many different radio access technologies (RATs). With the rise of these smart devices, which come with significant computational power, it is now possible to ask the question: can the network allow the client control over RAT selection and association in order to meet some client-driven or network-driven objective, and to what degree does the network assist the client in making these choices? This question becomes particularly important given the increasing interest in standardization and deployment of client-controlled edge networking, or Fog networking. In this paper, we explore the spectrum of client-controlled HetNets for 5G networks: from the fully devolved distributed local control approach, where clients make local decisions without any assistance from the network, to the hybrid control approach where clients may make decisions given some global information provided by the network.
12 December 2022 in IEEE Access
Dense video captioning is a very challenging task because it requires a high-level understanding of the video story, as well as pinpointing details such as objects and motions for a consistent and fluent description of the video. Many existing solutions divide this problem into two sub-tasks, event detection and captioning, and solve them sequentially (“localize-then-describe” or reverse). Consequently, the final outcome is highly dependent on the performance of the preceding modules. Detecting events or generating sentences directly from vast amounts of video information is prone to loss of elaborateness (accuracy and fluency), and hence propagates as a cascade effect to subsequent modules. Parallel decoding approaches have been proposed to overcome these limitations, which leads to another challenge: the bottleneck at the branch point. In this paper, we decompose this sequential approach by proposing a parallel pathway dense video captioning framework (PPVC) that localizes and describes events simultaneously without any bottlenecks. We introduce a representation organization network at the branching point of the parallel pathway to organize the encoded video feature by considering the entire storyline. Then, an event localizer focuses to localize events without any event proposal generation network, a sentence generator describes events while considering the fluency and coherency of sentences. A gating network controls the flow of organized representations to filter out unnecessary information. Our method has several advantages over existing work: (i) the final output does not depend on the output of the preceding modules, (ii) it improves existing parallel decoding methods by relieving the bottleneck of information. We evaluate the performance of PPVC on large-scale benchmark datasets, the ActivityNet Captions, and YouCook2. PPVC not only outperforms existing algorithms on the majority of metrics but also improves on both datasets by 5.4% and 4.9% compared to the state-of-the-art parallel decoding method.
24 May 2023 in IEEE Access
Dense video captioning aims to localize and describe events for storytelling in untrimmed videos. It is a conceptually very challenging task that requires concise, relevant, and coherent captioning based on high-quality event localization. Unlike simple temporal action localization tasks without overlapping events, dense video captioning requires detecting multiple/overlapping regions in order to branch out the video story. Most existing methods generate numerous candidate event proposals and then eliminate duplicate ones using a event proposal selection algorithm (e.g., non-maximum suppression) or generate event proposals directly through box prediction and binary classification mechanisms, similar to object detection tasks. Despite these efforts, the aforementioned approaches tend to fail to localize overlapping events into different stories, hindering high-quality captioning. In this paper, we propose SBS, a dense video captioning framework with a gradual approach that addresses the challenge of localizing overlapping events and eventually constructs high-quality captioning. SBS accurately estimates the number of explicit events for each video snippet and then detects the boundaries context/activities, which are the details for generating the event proposals. Based on both the number of events and boundaries, SBS generates the event proposals. SBS encodes the context of the event sequence and finally generates sentences describing the event proposals. Our framework is fairly effective in localizing multiple/overlapping events, thus experimental results showthe state-of-the-art performance compared to the existing methods.