Multimodal learning systems have been found to be effective in studies investigating cognitive theory of multimedia learning. Yet this research is rarely put into practice in Science, Technology, Engineering, and Math (STEM) learning environments, which are dominated by visual graphics. Introducing multimodal learning systems into STEM settings and allowing students to access dual channel cueing beyond visual perception may help more students process information in their preferred modality. The purpose of this study was to investigate the usability, effectiveness, and design of multimodal interfaces for enhancing access to graphical representations. We used existing theories of multisensory information processing to study how sighted participants could learn and interpret spatial primitives and graphical concepts presented via three non-visual conditions: natural language (NL) descriptions, haptic renderings, and a NL-Haptic combination. Results showed that access to haptic-only renderings produced the least accurate responses, whereas NL descriptions with and without haptics led to similar performance by participants when learning graphical content without vision. Performance was also impacted by graphical content complexity with the highest level of accuracy observed for closed forms, compared to paired line segments and line/polygon intersections. We argue that universally designed, multimodal learning environments can transcend traditional, visual diagrams by utilizing non-visual channels and commercial hardware to support learners with different sensory abilities, preferences, and processing needs. Findings contribute to extending theoretical insights of non-visual information processing to better understand multisensory learning in sighted individuals.
30 May 2026 in Universal Access in the Information Society
A persistent and vexing challenge for blind and low vision (BLV) travelers involves finding assigned and/or available seating in different transportation modalities. This is particularly problematic as autonomous vehicle (AV) rideshares and shuttles become more prevalent given the absence of assistance from human operators. While audio-based smartphone systems are the current standard for BLV interaction, emerging evidence suggests that multisensory systems, combining haptics and spatialized audio, may improve accessibility across the trip. We present a user study with 8 BLV participants comparing 1) audio-only descriptions and 2) multisensory information using vibro-audio maps and environmental audio cues to assist in seat finding. Results suggest higher accuracy, faster completion times, and stronger user preference for the multisensory condition. Participants also emphasized the importance of well-timed, contextually relevant information and greater control over information delivery. These findings highlight the promise of multisensory systems to improve independent travel for BLV users in future AVs.
1. The rapid pace of data aggregation and information communication requires a labor force that understands how to accurately interpret pre-processed data through graphical representations. These t...