20 August 2019 in IEEE Transactions on Circuits and Systems for Video Technology
In this work, we propose a division-and-summarization (DaS) framework for dense video captioning. After partitioning each untrimmed long video as multiple event proposals, where each event proposal consists of a set of short video segments, we extract visual feature (e.g., C3D feature) from each segment and use the existing image/video captioning approach to generate one sentence description for this segment. Considering that the generated sentences contain rich semantic descriptions about the whole event proposal, we formulate the dense video captioning task as a visual cue aided sentence summarization problem and propose a new two stage Long Short Term Memory (LSTM) approach equipped with a new hierarchical attention mechanism to summarize all generated sentences as one descriptive sentence with the aid of visual features. Specifically, the first-stage LSTM network takes all semantic words from the generated sentences and the visual features from all segments within one event proposal as the input, and acts as the encoder to effectively summarize both semantic and visual information related to this event proposal. The second-stage LSTM network takes the output from the first-stage LSTM network and the visual features from all video segments within one event proposal as the input, and acts as the decoder to generate one descriptive sentence for this event proposal. Our comprehensive experiments on the ActivityNet Captions dataset demonstrate the effectiveness of our newly proposed DaS framework for dense video captioning.
1 March 2019 in International Journal of Advanced Robotic Systems
The brain–computer interface-based rehabilitation robot has quickly become a very important research area due to its natural interaction. One of the most important problems in brain–computer interface is that large-scale annotated electroencephalography data sets required by advanced classifiers are almost impossible to acquire because biological data acquisition is challenging and quality annotation is costly. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed with the test data. It can be considered a powerful tool for solving the problem of insufficient training data. There are two basic issues with transfer learning, under transfer and negative transfer. We proposed a novel brain–computer interface framework by using autoencoder-based transfer learning, which includes three main components: an autoencoder framework, a joint adversarial network, and a regularized manifold constraint. The autoencoder framework automatically encodes and reconstructs data from source and target domains and forces the neural network to learn to represent these domains reliably. The joint adversarial network aims to force the network to learn to encode more appropriately for the source domain and target domain simultaneously, thereby overcoming the problem of under transfer. The regularized manifold constraint aims to avoid the problem of negative transfer by avoiding geometric manifold structure in the target domain being destroyed by the source domain. Experiments show that the brain–computer interface framework proposed by us can achieve better results than state-of-the-art approaches in electroencephalography signal classification tasks. This is helpful in aiding our rehabilitation robot to understand the intention of patients and can help patients to carry out rehabilitation exercises effectively.
24 July 2018 in IEEE Transactions on Fuzzy Systems
While tactile sensing is becoming an indispensable robotic ability for object recognition and grasping manipulation, it still remains challenging for us to deal with the tactile data as the force distribution over the array sensors continuously changes as a function of time. In this paper, we present an efficient feature extractor named Linear Dynamic Systems-based Fuzzy C-means Method (LDS-FCM) to encode the tactile sequences, both spatially and temporally. To this end, we decompose every input sequence into multiple sub-sequences, each of which is locally described by a finite-ordered observability matrix of the LDS model. A fuzzy c-means method is then applied to cluster the local LDS descriptors for learning a codebook. Conditioned on the resulting codebook, the global tactile representation is formulated by employing two different frameworks to integrate the subsequences within each tactile sequence, namely, the Vector of Locally Aggregated Descriptor (VLAD) and Bag-of-Word (BoW) approaches. The effectiveness of the proposed model is verified by a variety of experimental evaluations on five benchmark datasets. It shows that our proposed method achieves a higher classification accuracy than the state-of-the-art models with a large margin.
22 March 2018 in IEEE Transactions on Cognitive and Developmental Systems
It is hard to grasp objects based on BCI by brain-actuated robot arm and hand due to its high degree of freedom. Shared control strategy and hybrid BCI are research trends to solve this control problem of brain-actuated discrete event system (DES). We propose a new shared control method based on Fused Fuzzy Petri Nets (FFPNs) for combining the robot automatic control and the brain-actuated control. This method takes the advantages of both fuzzy control and Petri Nets such as easy modeling, robustness and effectiveness. Both MATLAB simulation test and Barrett robot hand practical experiments show that the proposed method performs much better than automatic control or BCI control independently. In the online BCI practical experiment, the user successfully control the Barrett robot hand to grasp object avoiding obstacle in whole ten random scenes by our shared control method and hybrid BCI. Compared with BCI control, the user needs not to synchronously work according to the specific paradigm in the whole process. Meanwhile, our method improves safety and robustness by comparing with automatic control.