01
Multi-view 2D Waypoint Prediction
Given multi-view RGB observations and a language instruction, a fine-tuned VLM predicts corresponding 2D waypoint sequences in text form.
Overview
A fundamental modality gap between vision-language models (VLMs) and robot actions limits the transfer of pretrained vision-language capabilities to manipulation. Trajectory-based intermediate representations offer a promising bridge, but 2D trajectories lack explicit 3D grounding, while depth-lifted trajectories are sensitive to depth noise and free-space ambiguity.
We introduce 3D Consistent Waypoints (3DWay), an efficient, explicit, and actionable 3D trajectory representation. Instead of directly regressing 3D coordinates with a VLM, 3DWay predicts multi-view consistent 2D waypoints and reconstructs them via geometric triangulation. The resulting 3D trajectories can be directly executed for simple tasks or used as explicit geometric priors for foundation VLAs.
3D-Consistent Representation
Actionable 3D task trajectories without depth sensing.
VLM-Compatible Generation
Multi-view consistent 2D prediction followed by geometric reconstruction.
Flexible Manipulation Interface
Direct execution or foundation-VLA integration for improved generalization.
Method
How 3DWay Works
Let the VLM reason in image space; let geometry recover the 3D structure.
02
Geometric Triangulation
With known camera parameters, corresponding 2D waypoints are triangulated into explicit 3D points, forming the complete 3DWay trajectory.
03
Policy Execution
3DWay supports direct waypoint execution or Adaptive Waypoint-Guided Fine-tuning for foundation VLAs.
Results
Generalization, Data Efficiency, and Real-World Transfer
64.0%
RLBench Unseen
Direct execution of 3DWay demonstrates strong generalization and effective transfer of pretrained VLM capabilities.
17.6 → 46.1%
Few-shot π0
3DWay-augmented fine-tuning substantially boosts few-shot VLA performance with only 10 demonstrations per task.
21.7 → 65.8%
Real Robot
3DWay integration substantially improves average success on real-world basic tasks.
Key ablation: removing multi-view consistent supervision drops RLBench unseen success from 64.0% to 22.0%.
Visualization
Multi-view Consistency and Open-World Generalization
Demos
Real-World Manipulation
Citation
@inproceedings{3dway2026,
title = {3DWay: Generalizing Robot Manipulation via 3D Consistent Waypoints},
author = {Huang, Ziqin and Li, Yingyue and Zhang, Chenyangguang and Zhang, Ruida and Chen, Yuxin and Wang, Gu and Liu, Xingyu and Tomizuka, Masayoshi and Ji, Xiangyang},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}