Zhiyuan Xu

Zhiyuan Xu

Auto-generated profile, based on publicly available information

Zhiyuan Xu

Midea Group, Shanghai, China

Research Focus & Metrics

Academic Career Length

14Years (2013-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 February 2025 in IEEE Robotics and Automation Letters
Verified
Journal Article
Closed access
JWJunjie Wen
Junjie Wen
YZYichen Zhu
Yichen Zhu
JLJinming Li
Jinming Li
MZMinjie Zhu
Minjie Zhu
ZTZhibin Tang
Zhibin Tang
JTJian Tang
Jian Tang
Vision-Language-Action (VLA) models have shown remarkable potential in visuomotor control and instruction comprehension through end-to-end learning processes. However, current VLA models face significant challenges: they are slow during inference and require extensive pre-training on large amounts of robotic data, making real-world deployment difficult. In this letter, we introduce a new family of compact vision-language-action models, called TinyVLA, which offers two key advantages over existing VLA models: (1) faster inference speeds, and (2) improved data efficiency, eliminating the need for pre-training stage. Our framework incorporates two essential components to build TinyVLA: (1) initializing the policy backbone with robust, high-speed multimodal models, and (2) integrating a diffusion policy decoder during fine-tuning to enable precise robot actions. We conducted extensive evaluations of TinyVLA in both simulation and on real robots, demonstrating that our approach significantly outperforms the state-of-the-art VLA model, OpenVLA, in terms of speed and data efficiency, while delivering comparable or superior performance. Additionally, TinyVLA exhibits strong generalization capabilities across various dimensions, including language instructions, novel objects, unseen positions, changes in object appearance, background variations, and environmental shifts, often matching or exceeding the performance of OpenVLA. We believe that TinyVLA offers an interesting perspective on utilizing pre-trained multimodal models for policy learning.

14 December 2017 in IEEE Transactions on Industrial Informatics
Verified
Journal Article
Closed access
BZBo Zhang
Bo Zhang
CLChi Harold Liu
Chi Harold Liu
JTJian Tang
Jian Tang
ZXZhiyuan Xu
Zhiyuan Xu
JMJian Ma
Jian Ma
WWWendong Wang
Wendong Wang
Mobile Crowdsourcing (MCS) is now an important source of information for smart cities, especially with the help of unmanned aerial vehicles (UAVs) and driverless cars. They are equipped with different kinds of high-precision sensors, and can be scheduled/controlled completely during data collection, which will make MCS system more robust. However, they are limited to energy constraint, especially for long-term, long-distance sensing tasks, and cities are almost too crowded to set stationary charging station. Towards this end, in this paper we propose to leverage emerging Deep Reinforcement Learning (DRL) techniques for enabling model-free unmanned vehicles control, and present a novel and highly effective control framework, called "DRL-RVC". It utilizes the powerful Convolutional Neural Network (CNN) for feature extraction of the necessary information (including sample distribution, traffic flow, and etc.), then makes decisions under the guidance of the Deep Q Network (DQN). That is, UAVs will cruise in the city without control and collect most required data in the sensing region, while mobile unmanned charging station will reach the charging point in the shortest possible time. Finally, we validate and evaluate the proposed framework via extensive simulations based on a real dataset in Rome. Extensive simulation results well justify the effectiveness and robustness of our approach.

1 August 2017
Verified
Conference Paper
Closed access
YLYing Liu
Ying Liu
BLBrent Logan
Brent Logan
NLNing Liu
Ning Liu
ZXZhiyuan Xu
Zhiyuan Xu
JTJian Tang
Jian Tang
YWYangzhi Wang
Yangzhi Wang
In this paper, we propose the first deep reinforce-ment learning framework to estimate the optimal Dynamic Treat-ment Regimes from observational medical data. This framework is more flexible and adaptive for high dimensional action and state spaces than existing reinforcement learning methods to model real life complexity in heterogeneous disease progression and treatment choices, with the goal to provide doctor and patients the data-driven personalized decision recommendations. The proposed deep reinforcement learning framework contains a supervised learning step to predict the most possible expert actions; and a deep reinforcement learning step to estimate the long term value function of Dynamic Treatment Regimes. We motivated and implemented the proposed framework on a data set from the Center for International Bone Marrow Transplant Research (CIBMTR) registry database, focusing on the sequence of prevention and treatments for acute and chronic graft versus host disease. We showed results of the initial implementation that demonstrates promising accuracy in predicting human expert decisions and initial implementation for the reinforcement learning step.

28 October 2024
Verified
Conference Paper
Open access
YZYichen Zhu
Yichen Zhu
MZMinjie Zhu
Minjie Zhu
WZWill Zhang
Will Zhang
ZXZhiyuan Xu
Zhiyuan Xu
YPYaxin Peng
Yaxin Peng
In this paper, we introduce LLaVA-φ (LLaVA-Phi), an efficient multi-modal assistant that harnesses the power of the recently advanced small language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a notable advancement in the realm of compact multi-modal models. It demonstrates that even smaller language models, with as few as 2.7B parameters, can effectively engage in intricate dialogues that integrate both textual and visual elements, provided they are trained with high-quality corpora. Our model delivers commendable performance on publicly available benchmarks that encompass visual comprehension, reasoning, and knowledge-based perception. Beyond its remarkable performance in multi-modal dialogue tasks, our model opens new avenues for applications in time-sensitive environments and systems that require real-time interaction, such as embodied agents. It highlights the potential of smaller language models to achieve sophisticated levels of understanding and interaction, while maintaining greater resource efficiency. The project is available at https://github.com/zhuyiche/llava-phi **REMOVE 2nd URL**://github.com/zhuyiche/llava-phi.

22 September 2021 in IEEE Transactions on Multimedia
Verified
Journal Article
Closed access
CYChengxiang Yin
Chengxiang Yin
JTJian Tang
Jian Tang
TYTongtong Yuan
Tongtong Yuan
ZXZhiyuan Xu
Zhiyuan Xu
YWYanzhi Wang
Yanzhi Wang
Fine-grained instance segmentation is considerably more complicated and challenging than semantic segmentation. Most existing instance segmentation methods only focus on accuracy without paying much attention to inference latency, which, is critical to real-time applications, such as autonomous driving. In this paper, we aim to bridge the gap between semantic segmentation and instance segmentation by presenting a novel real-time model for instance segmentation, Sem2Ins, which effectively generates instance boundaries according to a semantic segmentation by leveraging conditional generative adversarial networks (cGANs) coupled with deep supervision and a weighted fusion layer. Specifically, supervision is imposed on each output layer, and features from different levels are fused to produce a well-generated boundary map. Sem2Ins has the following desirable features: 1) Combined with some fast semantic segmentation methods, Sem2Ins runs at a real-time speed that is fairly well-balanced against accuracy; 2) Sem2Ins works flexibly with any semantic segmentation model for instance segmentation, and if the given semantic segmentation is sufficiently good, Sem2Ins even achieves state-of-the-art in terms of accuracy; 3) deep supervision and weighted fusion can be leveraged to generate high-quality boundaries; and 4) Sem2Ins can be easily extended to panoptic segmentation. Extensive experiments performed on the Cityscapes, WildDash, KITTI and COCO benchmarks have demonstrated that 1) Sem2Ins, when combined with PSPNet and DDRNet-23-Slim, consistently outperforms the state-of-the-art real-time solution (Box2Pix) in terms of both speed and accuracy; and 2) Sem2Ins combined with DPC performs comparably to some powerful detect-and-segment approaches.

20 September 2019 in IEEE Transactions on Neural Networks and Learning Systems
Verified
Journal Article
Closed access
CYChengxiang Yin
Chengxiang Yin
JTJian Tang
Jian Tang
ZXZhiyuan Xu
Zhiyuan Xu
YWYanzhi Wang
Yanzhi Wang
Video question answering (VideoQA) is a very important but challenging multimedia task, which automatically analyzes questions and videos and generates accurate answers. However, research on VideoQA is still in its infancy. In this article, we propose a novel memory augmented deep recurrent neural network (MA-DRNN) model for VideoQA, which features a new method for encoding videos and questions, and memory augmentation using the emerging differentiable neural computer (DNC). Specifically, we encode textual (questions) information before visual (videos) information, which leads to better visual-textual representations. Moreover, we leverage DNC (with an external memory) for storing and retrieving useful information in questions and videos, and modeling the long-term visual-textual dependence. To evaluate the proposed model, we conducted extensive experiments using the VTW data set and MSVD-QA data set, which are both Widely used large-scale video data sets for language-level understanding. The experimental results have well validated the proposed model and showed that it outperforms the state-of-the-art in terms of various accuracy-related metrics.