arxiv:2503.10480

World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning

Published on Mar 13

· Submitted by

sinwang on Mar 14

Upvote

Authors:

Siyin Wang ,

Zhaoye Fei ,

Shiduo Zhang ,

Jinlan Fu ,

Xipeng Qiu

Abstract

Recent advances in large vision-language models (LVLMs) have shown promise for embodied task planning, yet they struggle with fundamental challenges like dependency constraints and efficiency. Existing approaches either solely optimize action selection or leverage world models during inference, overlooking the benefits of learning to model the world as a way to enhance planning capabilities. We propose Dual Preference Optimization (D^2PO), a new learning framework that jointly optimizes state prediction and action selection through preference learning, enabling LVLMs to understand environment dynamics for better planning. To automatically collect trajectories and stepwise preference data without human annotation, we introduce a tree search mechanism for extensive exploration via trial-and-error. Extensive experiments on VoTa-Bench demonstrate that our D^2PO-based method significantly outperforms existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and LLaMA-3.2 (11B), achieving superior task success rates with more efficient execution paths.

View arXiv page View PDF Add to collection

Community

sinwang

Paper author Paper submitter Mar 14

•

edited Mar 14

📣 World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning

🤔 Current LVLMs struggle with grounding in embodied environments, how can we make AI agents understand the physical world like humans?

🔑 Key insight: When agents perform tasks, they need both "WHAT to do" and a mental model of "WHAT WILL HAPPEN after each action"! This internal #WorldModel is fundamental to human planning capabilities 🧠 #CognitiveAI

sinwang

Paper author Paper submitter Mar 14

•

edited Mar 14

Current methods focus mainly on action selection, Our D²PO framework jointly optimizes TWO aspects through preference learning:
• Action Selection: choosing the right actions
• State Prediction: building an internal world model that predicts environmental changes

sinwang

Paper author Paper submitter Mar 14

🌲 We also introduce a tree search mechanism to automatically collect trajectories through trial-and-error, eliminating human annotation while gathering diverse interaction experiences! This greatly improves data efficiency