Large language models (LLMs) have emerged as a powerful tool for creating personalized knowledge experiences for users, often serving as their own interface through text-based chatbots. The interpretation of user intent and generation of output occur implicitly within the model’s architecture. We propose an alternative approach in a system we call Bespoke where the LLM acts as an agent to explicitly reason about user intent, plan, and generate graphical interfaces to fulfill that intent. This approach enables the creation of visually rich interactions that complement chat-based interactions. By employing a step-by-step reasoning process to reduce ambiguity and keep the model on track, we compose interfaces from a toolkit of widgets, providing a designed and tailored user experience. Our early experiment shows that the output interface differs depending on the interpreted intent. In the current version, these interactions are multimodal in the automatic generation of UI; in future versions, this paradigm can be extended to multiple modalities of input and output. This agentive approach moves the interface towards a personalized, bespoke experience with multimodal interaction that adapts to the user’s intentions. See video demonstration here [2].
Large Multimodal Models (LMMs) like OpenAI's GPT4o and Google's Gemini, introduced in 2024, process multiple modalities, enabling significant advances in multimodal interaction. Inspired by frameworks for self-driving cars and AGI, this paper proposes "Levels of Multimodal Interaction" to guide research and development. The four levels are: basic multimodality (0), single modalities in turn-taking; combined multimodality (1), fused interpretation of multiple modalities; humanlike (2), natural interaction flow with additional communication signals; and beyond humanlike (3), surpassing human capabilities and include underlying hidden signals with the potential for transformational human-AI integration. LMMs have progressed from Level 0 to 1, with Level 2 next. Level 3 sets a speculative target that multimodal interaction research could help achieve, where interaction becomes more natural and ultimately surpasses human capabilities. Eventually, such Level 3 multimodal interaction could lead to greater human-AI integration and transform human performance. This anticipated shift, in turn, introduces considerations, particularly around safety, agency and control of AI systems.