Model-based reinforcement learning (MBRL) achieves significant sample efficiency in practice in comparison to model-free RL, but its performance is often limited by the existence of model prediction error. To reduce the model error, standard MBRL approaches train a single well-designed network to fit the entire environment dynamics, but this wastes rich information on multiple sub-dynamics, which can be modeled separately, allowing us to construct the world model more accurately. In this paper, we propose environment dynamics decomposition (ED2), a novel world model construction framework that models the environment in a decomposing manner. ED2 contains two key components: sub-dynamics discovery (SD2) and dynamics decomposition prediction (D2P). SD2 discovers the sub-dynamics in an environment automatically and D2P constructs the decomposed world model following the sub-dynamics. ED2 can be easily combined with the existing MBRL algorithms and empirical results show that ED2 significantly reduces the model error, increases the sample efficiency, and achieves higher asymptotic performance when combined with the state-of-the-art MBRL algorithms on various continuous control tasks.

Learning scalable reward signals for robotic manipulation remains challenging: manually designed dense rewards require task-specific expertise, while collecting human preference annotations is costly and difficult to scale. It is also unclear whether a general-purpose multimodal large language model (MLLM) can evaluate task progress reliably from visual trajectories alone. We introduce CriticGPT, an MLLM-based critic that compares pairs of robot-manipulation videos, produces natural-language analyses, and outputs preference feedback that can be distilled into a reward model for downstream policy learning. On Meta-World, CriticGPT achieves high preference accuracy and supports in-benchmark task transfer to tasks not used during fine-tuning. An independent study on 300 manually annotated trajectory pairs shows 92% agreement between CriticGPT and human judgments, compared with 86% agreement between the automatic supervision template and human judgments. Factor World experiments further show that the frozen critic remains robust under controlled visual distribution shifts, without implying broad cross-benchmark generalization. Finally, CriticGPT-derived rewards provide useful policy-learning signals on several evaluated Meta-World tasks.

Model-based reinforcement learning (MBRL) achieves significant sample efficiency in practice in comparison to model-free RL, but its performance is often limited by the existence of model prediction error. To reduce the model error, standard MBRL approaches train a single well-designed network to fit the entire environment dynamics, but this wastes rich information on multiple sub-dynamics which can be modeled separately, allowing us to construct the world model more accurately. In this paper, we propose the Environment Dynamics Decomposition (ED2), a novel world model construction framework that models the environment in a decomposing manner. ED2 contains two key components: sub-dynamics discovery (SD2) and dynamics decomposition prediction (D2P). SD2 discovers the sub-dynamics in an environment automatically and then D2P constructs the decomposed world model following the sub-dynamics. ED2 can be easily combined with existing MBRL algorithms and empirical results show that ED2 significantly reduces the model error, increases the sample efficiency, and achieves higher asymptotic performance when combined with the state-of-the-art MBRL algorithms on various continuous control tasks. Our code is open source and available at https://github.com/ED2-source-code/ED2.

Yibin Chen, Jinyi Liu, Yan Zheng, Yifu Yuan, Jianye Hao. Findings of the Association for Computational Linguistics: ACL 2025. 2025.