The high redundancy among keyframes is a critical issue for the prior summarizing methods in dealing with user-created videos. To address the critical issue, we present a Graph Attention Networks (GAT) adjusted Bi-directional Long Short-term Memory (Bi-LSTM) model for unsupervised video summarization. First, the GAT is adopted to transform an image's visual features into higher-level features by the Contextual Features based Transformation (CFT) mechanism. Specifically, a novel Salient-Area-Size-based spatial attention model is presented to extract frame-wise visual features on the observation that humans tend to focus on sizable and moving objects. Second, the higher-level visual features are integrated with semantic features processed by Bi-LSTM to refine the frame-wise probability of being selected as keyframes. Extensive experiments demonstrate that our method outperforms state-of-the-art methods.
16 November 2023 in IEEE Transactions on Learning Technologies
Higher education is rapidly growing in the online learning landscape. However, current personalized recommendation techniques struggle with precise extraction of complex mathematical semantics, hindering accurate perception of learners' cognitive states and relevance of recommendations. This paper proposes a framework for extracting complex mathematical semantics and providing personalized exercise recommendations. We design a tree-based position encoding method to enhance the accuracy of positional representation for mathematical expressions in pre-trained model, aiming to improve the performance of downstream tasks. We propose an automatic method for extracting knowledge attributes based on expert annotations, enabling interpretable cognitive diagnosis. Furthermore, we employ sequential pattern mining to discover the knowledge usage patterns in exercises, generate learning paths using a multi-layer knowledge graph, and leverage cognitive diagnostic results to enhance the relevance of recommendations. Experimental results show a 2.0% improvement in mathematical symbol embedding on mathematical formula retrieval tasks, and knowledge attribute extraction accuracy ranging from 66.5% to 81.7%. Learners' post-test scores significantly improve during group testing, with good consistency between online cognitive diagnosis and self-diagnosis.
13 July 2023 in IEEE Transactions on Image Processing
End-to-end Long Short-Term Memory (LSTM) has been successfully applied to video summarization. However, the weakness of the LSTM model, poor generalization with inefficient representation learning for inputted nodes, limits its capability to efficiently carry out node classification within user-created videos. Given the power of Graph Neural Networks (GNNs) in representation learning, we adopted the Graph Information Bottle (GIB) to develop a Contextual Feature Transformation (CFT) mechanism that refines the temporal dual-feature, yielding a semantic representation with attention alignment. Furthermore, a novel Salient-Area-Size-based spatial attention model is presented to extract frame-wise visual features based on the observation that humans tend to focus on sizable and moving objects. Lastly, semantic representation is embedded within attention alignment under the end-to-end LSTM framework to differentiate indistinguishable images. Extensive experiments demonstrate that the proposed method outperforms State-Of-The-Art (SOTA) methods.
4 September 2014 in Journal of Visual Communication and Image Representation
In this paper, a novel approach to encode lenslet (LL) images is proposed. The method departs from traditional block-based coding structures and employs a hexagonal-shaped pixel cluster, called macro-pixel, as an elementary coding unit. A novel prediction mode based on dictionary learning is proposed, whereby macro-pixels are represented by a sparse linear combination of atoms from a generic dictionary. Additionally, an optimized linear prediction mode and a directional prediction mode specifically designed for macro-pixels are proposed. Rate-distortion optimization is utilized to select the best intra prediction mode for each macro-pixel. Experimental results on the light field image data set show that the proposed coding system outperforms HEVC and the state-of-the-art in LL image coding with an average peak signal to noise ratio gain of 3.33 and 1.41 dB, respectively, and with rate savings of 67.13% and 34.30%, respectively.