Xiaojian Ma

Xiaojian Ma

Auto-generated profile, based on publicly available information

Xiaojian Ma

State Key Laboratory for General Artificial Intelligence, Beijing, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
5 December 2024 in IEEE Transactions on Pattern Analysis and Machine Intelligence
Verified
Journal Article
Closed access
ZWZihao Wang
Zihao Wang
SCShaofei Cai
Shaofei Cai
ALAnji Liu
Anji Liu
YJYonggang Jin
Yonggang Jin
JHJinbing Hou
Jinbing Hou
YLYitao Liang
Yitao Liang
Achieving human-like planning and control with multimodal observations in an open world is a key milestone for more functional generalist agents. Existing approaches can handle certain long-horizon tasks in an open world. However, they still struggle when the number of open-world tasks could potentially be infinite and lack the capability to progressively enhance task completion as game time progresses. We introduce JARVIS -1, an open-world agent that can perceive multimodal input (visual observations and human instructions), generate sophisticated plans, and perform embodied control, all within the popular yet challenging open-world Minecraft universe. Specifically, we develop JARVIS -1 on top of pre-trained multimodal language models, which map visual observations and textual instructions to plans. The plans will be ultimately dispatched to the goal-conditioned controllers. We outfit JARVIS -1 with a multimodal memory, which facilitates planning using both pre-trained knowledge and its actual game survival experiences. JARVIS -1 is the existing most general agent in Minecraft, capable of completing over 200 different tasks using control and observation space similar to humans. These tasks range from short-horizon tasks, e.g., “chopping trees” to long-horizon ones, e.g., “obtaining a diamond pickaxe”. JARVIS -1 performs exceptionally well in short-horizon tasks, achieving nearly perfect performance. In the classic long-term task of ObtainDiamondPickaxe , JARVIS -1 surpasses the reliability of current state-of-the-art agents by 5 times and can successfully complete longer-horizon and more challenging tasks. Furthermore, we show that JARVIS -1 is able to self-improve following a life-long learning paradigm thanks to multimodal memory, sparking a more general intelligence and improved autonomy. The project can be found in https://craftjarvis-jarvis1.github.io/ .

30 September 2024
Verified
Book Chapter
Closed access
PDF file
YFYue Fan
Yue Fan
XMXiaojian Ma
Xiaojian Ma
RWRujie Wu
Rujie Wu
YDYuntao Du
Yuntao Du
JLJiaqi Li
Jiaqi Li
QLQing Li
Qing Li

18 March 2026 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
JGJun Guo
Jun Guo
XMXiaojian Ma
Xiaojian Ma
YFYue Fan
Yue Fan
HLHuaping Liu
Huaping Liu
QLQing Li
Qing Li
Open-vocabulary 3D scene understanding presents a significant challenge in computer vision, with wide-ranging applications in embodied agents and augmented reality systems. Existing methods adopt neural rendering methods as 3D representations and jointly optimize color and semantic features to achieve rendering and scene understanding simultaneously. In this paper, we introduce Semantic Gaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Our key idea is to distill knowledge from 2D pretrained models to 3D Gaussians. Unlike existing methods, we design a versatile projection approach that maps various 2D semantic features from pre-trained image encoders into a novel semantic component of 3D Gaussians, which is based on spatial relationships and needs no additional training. We further build a 3D semantic network that directly predicts the semantic component from raw 3D Gaussians for fast inference. The quantitative results on ScanNet segmentation and LERF object localization demonstrate the superior performance of our method. Additionally, we explore several applications of Semantic Gaussians, including object part segmentation, instance segmentation, scene editing, and spatiotemporal segmentation, with better qualitative results over 2D and 3D baselines, highlighting its versatility and effectiveness in supporting diverse downstream tasks.

30 September 2024
Verified
Book Chapter
Closed access
PDF file
ZZZiyu Zhu
Ziyu Zhu
ZZZhuofan Zhang
Zhuofan Zhang
XMXiaojian Ma
Xiaojian Ma
XNXuesong Niu
Xuesong Niu
YCYixin Chen
Yixin Chen
QLQing Li
Qing Li

12 January 2026 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
JGJun Guo
Jun Guo
XMXiaojian Ma
Xiaojian Ma
YWYikai Wang
Yikai Wang
MYMin Yang
Min Yang
HLHuaping Liu
Huaping Liu
QLQing Li
Qing Li
This paper investigates training better visual world models for robot manipulation, i.e., models that can predict future visual observations by conditioning on past frames and robot actions. Specifically, we consider world models that operate on RGB-D frames (RGB-D world models). As opposed to canonical approaches that handle dynamics prediction mostly implicitly and reconcile it with visual rendering in a single model, we introduce FlowDreamer, which adopts 3D scene flow as explicit motion representations. FlowDreamer first predicts 3D scene flow from the past frame and action conditions with a U-Net, and then a diffusion model will predict the future frame utilizing the scene flow. FlowDreamer is trained end-to-end despite its modularized nature. We conduct experiments on 4 different benchmarks, covering both video prediction and visual planning tasks. The results demonstrate that FlowDreamer achieves better performance compared to other baseline RGB-D world models by 7% on semantic similarity, 11% on pixel quality, and 6% on success rate in various robot manipulation domains.