James F. Mullen

James F. Mullen

Auto-generated profile, based on publicly available information

James F. Mullen

Computer Science, Virginia Tech, Ashburn, Virginia, United States of America, 20147

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
25 December 2023 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
VDVishnu Sashank Dorbala
Vishnu Sashank Dorbala
JMJames F. Mullen
James F. Mullen
DMDinesh Manocha
Dinesh Manocha
We present language-guided exploration (LGX), a novel algorithm for Language-Driven Zero-Shot Object Goal Navigation (L-ZSON), where an embodied agent navigates to an uniquely described target object in a previously unseen environment. Our approach makes use of large language models (LLMs) for this task by leveraging the LLM's commonsense-reasoning capabilities for making sequential navigational decisions. Simultaneously, we perform generalized target object detection using a pre-trained Vision-Language grounding model. We achieve state-of-the-art zero-shot object navigation results on RoboTHOR with a success rate (SR) improvement of over 27% over the current baseline of the OWL-ViT CLIP on Wheels (OWL CoW). Furthermore, we study the usage of LLMs for robot navigation and present an analysis of various prompting strategies affecting the model output. Finally, we showcase the benefits of our approach via real-world experiments that indicate the superior performance of LGX in detecting and navigating to visually unique objects.

13 September 2021 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
JMJames Mullen
James Mullen
JMJosh Mosier
Josh Mosier
SCSounak Chakrabarti
Sounak Chakrabarti
ACAnqi Chen
Anqi Chen
TWTyler White
Tyler White
DLDylan Losey
Dylan Losey
Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arms motion, the robot gathers information about the humans desired task. But how does the human know what their robot has inferred Todays approaches often focus on conveying intent: for instance, using legible motions or gestures to indicate what the robot is planning. However, closing the loop on robot inference requires more than just revealing the robots current policy: the robot should also display the alternatives it thinks are likely, and prompt the human teacher when additional guidance is necessary. In this paper we propose a multimodal approach for communicating robot inference that combines both passive and active feedback. Specifically, we leverage information-rich augmented reality to passively visualize what the robot has inferred, and attention-grabbing haptic wristbands to actively prompt and direct the humans teaching. We apply our system to shared autonomy tasks where the robot must infer the humans goal in real-time. Within this context, we integrate passive and active modalities into a single algorithmic framework that determines when and which type of feedback to provide. Combining both passive and active feedback experimentally outperforms single modality baselines; during an in-person user study, we demonstrate that our integrated approach increases how efficiently humans teach the robot while simultaneously decreasing the amount of time humans spend interacting with the robot.

17 July 2024 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
JMJames F. Mullen
James F. Mullen
PGPrasoon Goyal
Prasoon Goyal
RPRobinson Piramuthu
Robinson Piramuthu
MJMichael Johnston
Michael Johnston
DMDinesh Manocha
Dinesh Manocha
RGReza Ghanadan
Reza Ghanadan
Home robots intend to make their users lives easier. Our work moves toward more helpful home robots by enabling them to inform their users of dangerous or unsanitary anomalies in the home. Some examples of these anomalies include the user leaving their milk out, forgetting to turn off the stove, or leaving poison accessible to children. To enable home robots with these abilities, we have created a new dataset, which we call SafetyDetect. The SafetyDetect dataset consists of 1000 anomalous home scenes, each of which contains unsafe or unsanitary situations for an agent to detect. Our approach utilizes large language models (LLMs) alongside both a graph representation of the scene which encodes relationships between the objects in the scene. Our key insight is that this connected scene graph and the object relationships it encodes enables the LLM to better reason about the scene - especially as it relates to detecting dangerous or unsanitary situations. Our most promising approach utilizes GPT-4 and pursues a classification technique where object relations from the scene graph are classified as normal, dangerous, unsanitary, or dangerous for children. This method is able to correctly identify over 90% of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conduct real world experiments on a ClearPath TurtleBot where we generate a scene graph from visuals of the real world scene, and run our approach with no modification. This setup resulted in little performance loss. The SafetyDetect Dataset and code will be released to the public upon this papers publication.

16 April 2025 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
JMJames F. Mullen
James F. Mullen
DKDhruva Kumar
Dhruva Kumar
XQXuewei Qi
Xuewei Qi
RMRajasimman Madhivanan
Rajasimman Madhivanan
ASArnie Sen
Arnie Sen
RKRichard Kim
Richard Kim
In the United States alone accidental home deaths exceed 128,000 per year. Our work aims to enable home robots who respond to emergency scenarios in the home, preventing injuries and deaths. We introduce a new dataset of household emergencies based in the ThreeDWorld simulator. Each scenario in our dataset begins with an instantaneous or periodic sound which may or may not be an emergency. The agent must navigate the multi-room home scene using prior observations, alongside audio signals and images from the simulator, to determine if there is an emergency or not. In addition to our new dataset, we present a modular approach for localizing and identifying potential home emergencies. Underpinning our approach is a novel probabilistic dynamic scene graph (P-DSG), where our key insight is that graph nodes corresponding to agents can be represented with a probabilistic edge. This edge, when refined using Bayesian inference, enables efficient and effective localization of agents in the scene. We also utilize multi-modal vision-language models (VLMs) as a component in our approach, determining object traits (e.g. flammability) and identifying emergencies. We present a demonstration of our method completing a real-world version of our task on a consumer robot, showing the transferability of both our task and our method. Our dataset will be released to the public upon this papers publication.