6 February 2025 in Patterns
Large language models (LLMs) have demonstrated performance approaching human levels in tasks such as long-text comprehension and mathematical reasoning, but they remain black-box systems. Understanding the reasoning bottlenecks of LLMs remains a critical challenge, as these limitations are deeply tied to their internal architecture. Attention heads play a pivotal role in reasoning and are thought to share similarities with human brain functions. In this review, we explore the roles and mechanisms of attention heads to help demystify the internal reasoning processes of LLMs. We first introduce a four-stage framework inspired by the human thought process. Using this framework, we review existing research to identify and categorize the functions of specific attention heads. Additionally, we analyze the experimental methodologies used to discover these special heads and further summarize relevant evaluation methods and benchmarks. Finally, we discuss the limitations of current research and propose several potential future directions.
28 September 2018 in IEEE Transactions on Knowledge and Data Engineering
Given the edge list of a social network, the node embedding method learns the structural features for every node and embeds the features into a vector space. The current related work on node embedding exploits only a portion of existing networks, e.g., static networks. However, social networks are inherently hierarchical and dynamic systems in which the topology changes constantly and the strength of influence of information among neighbors varies with different numbers of hops. We propose a highly efficient node embedding method, DNPS, that is faster and more accurate than state-of-the-art methods and that can further boost the training progress, especially under dynamic conditions. In this paper, we attempt to model the hierarchical and dynamic features of social networks by designing a damping-based sampling algorithm corresponding to a local search-based incremental learning algorithm, which can easily be extended to large-scale scenarios. We conduct extensive experiments on six real-world social networks with three challenging tasks, including missing link prediction, dynamic link prediction and multi-label classification. The results of the experiments on these tasks demonstrate that the proposed method significantly outperforms the existing methods with different settings.