13 May 2024 in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Developing chatbots as personal companions has long been a goal of artificial intelligence researchers. Recent advances in Large Language Models (LLMs) have delivered a practical solution for endowing chatbots with anthropomorphic language capabilities. However, it takes more than LLMs to enable chatbots that can act as companions. Humans use their understanding of individual personalities to drive conversations. Chatbots also require this capability to enable human-like companionship. They should act based on personalized, real-time, and time-evolving knowledge of their users. We define such essential knowledge as the common ground between chatbots and their users, and we propose to build a common-ground-aware dialogue system from an LLM-based module, named OS-1, to enable chatbot companionship. Hosted by eyewear, OS-1 can sense the visual and audio signals the user receives and extract real-time contextual semantics. Those semantics are categorized and recorded to formulate historical contexts from which the user's profile is distilled and evolves over time, i.e., OS-1 gradually learns about its user. OS-1 combines knowledge from real-time semantics, historical contexts, and user-specific profiles to produce a common-ground-aware prompt input into the LLM module. The LLM's output is converted to audio, spoken to the wearer when appropriate. We conduct laboratory and in-field studies to assess OS-1's ability to build common ground between the chatbot and its user. The technical feasibility and capabilities of the system are also evaluated. Our results show that by utilizing personal context, OS-1 progressively develops a better understanding of its users. This enhances user satisfaction and potentially leads to various personal service scenarios, such as emotional support and assistance.
21 October 2025 in ACM Transactions on Design Automation of Electronic Systems
Customization is a fundamental aspect of embedded system development, requiring developers to acquire extensive domain-specific knowledge from technical documents. However, the sheer volume of these documents, coupled with the intricate relationships between their contents, makes it challenging to efficiently retrieve the necessary information. This challenge highlights the need for a structured approach to organize domain knowledge, with explicit representation of interrelationships. Moreover, while advanced large language models (LLMs) show promise in aiding embedded system development, they often lack the specialized knowledge needed to address domain-specific queries effectively. In this article, we present HSG-RAG, a knowledge base construction and retrieval method tailored for embedded system development that leverages knowledge graphs to represent the hierarchical structure within technical documentation. Unlike prior retrieval-augmented generation (RAG) or GraphRAG-based approaches, which build the index either rely on semantic similarity or keyword co-occurrence, HSG-RAG captures the inherited dependency and hierarchical relationships in the documents, which benefits the retrieval in both performance and efficiency. We also introduce a benchmark for evaluating the effectiveness of RAG systems in solving real-world challenges in embedded systems, particularly for multi-hop question answering. Experimental results show that HSG-RAG outperforms both RAG and GraphRAG, generating more specific and concise responses.