WorldBridge Bridging Video and Action for Task Generalization in World Action Models

WorldBridge Team, Joy Future Academy

Read the paper ↗ Explore robot demonstrations ↓

OOD task generalization

Heres Figure 1: In-domain only, adding OOD videos, and adding few OOD pairs. Original paper figure with animated video-to-action paths.

Task generalization = Video generalization + Action generalization

Generalizing to a new task requires both predicting what should happen and knowing how to act.

ID samplesOOD samplesJoint coverage

01 / The finding

What WAMs need:
more video, fewer action labels.

Broad video coverage supplies task knowledge. A small amount of action supervision grounds it in execution, while labeling every video brings little additional benefit in these experiments.

Add videos, then progressively add action labels.

After the video-only setting, keep 550 OOD videos per task fixed.

Video supervisionAction supervision
  1. 0 videos, 0 with action annotations
    25.07%
    Baseline
  2. 550 videos, 0 with action annotations
    55.90%
    +30.83 pts
  3. 550 videos, 1 with action annotations
    70.70%
    +14.80 pts
  4. 550 videos, 5 with action annotations
    84.67%
    +13.97 pts
  5. 550 videos, 20 with action annotations
    90.03%
    +5.36 pts
  6. 550 videos, 50 with action annotations
    92.27%
    +2.24 pts
  7. 550 videos, 100 with action annotations
    93.17%
    +0.90 pts
  8. 550 videos, 550 with action annotations
    92.83%
    −0.34 pts
Video alone adds 30.83 points. Sparse action labels bring further gains, which diminish at higher budgets. All gains are percentage-point changes from the preceding row.

More video matters. Full action labels are not required.

Four supervision budgets make the trade-off clear.

Less video

100 videos,100 with action annotations
88.83%OOD task success

Reference budget

More video · sparse actions

550 videos,20 with action annotations
90.03%OOD task success

+1.20 pts with 80% fewer action-annotated videos than the reference

More video · partial actions

550 videos,100 with action annotations
93.17%OOD task success

Highest measured success

More video · full actions

550 videos,550 with action annotations
92.83%OOD task success

Annotating all 550 videos adds no gain over annotating 100

Scale video coverage

With 100 action-annotated videos in both settings, more videos raise success from 88.83% to 93.17%: +4.34 points.

Keep action labels sparse

550 videos, 20 with action annotations, outperform 100 videos, 100 with action annotations, with 80% fewer action-annotated videos.

Full labeling is unnecessary here

550 videos, 100 with action annotations, achieve 93.17% success, compared with 92.83% for 550 videos, all with action annotations—using 81.8% fewer action-annotated videos.

Counts are per OOD task. ID supervision stays fixed at 550 videos per task, all with action annotations. Action-annotated videos are included in the total video count; counts refer to complete demonstrations, not individual action steps. ID/OOD is a fixed task partition.

02 / Simulation Benchmark

Leading results on diverse manipulation benchmarks.

96.23%RoboTwin 2.0Average success rate80.9%RoboTwin C2RAverage success rate82.8%RoboCasa GR1Success rate57.6%DOMINO Level-1Success rate83.0%RMBenchJoint-best success rate

03 / The model

Building WorldBridge, step by step
WorldBridge model architecture: T5 language conditioning, visual memory and current frames encoded by a video VAE, video and action experts connected through shared attention and trained with joint flow matching. The unified 46D action space maps robot and egocentric data into canonical slots with validity masks and grouped normalization.
Six stages reveal the model’s components and connections. This is a conceptual construction sequence; the two experts are trained jointly. The full architecture pauses briefly before the animation loops. View full-size PDF ↗
01

Unified pretraining

Robot trajectories and human egocentric video share a unified processing pipeline and a 46D action space. Before curation, the data collection includes approximately 13,976 hours of robot data and 14,128 hours of human egocentric data.

02

Dual-scale visual memory

Short-term observations capture local motion; long-term observations track task progress. Both provide temporal context beyond the current frame.

03

Directional attention

The action expert reads predicted visual dynamics. A structured attention mask connects video and action experts without feeding action tokens back into video generation.

04

Action forcing

Different actions within a training chunk receive different noise levels, allowing the action expert to use partially clean actions alongside future video features.

04 / Physical capabilities

Six tasks. Two robot platforms.

Long-horizon execution, precise placement and coordinated manipulation on AgiBot G2 and ARX AC-1.

Evaluation protocolTask-specific fine-tuning · 20 trials per task · varied initial positions or orientations · complete-task success

Arranging flowers

    Complete-task success

    72.5% average · +15.0 percentage points over π₀.₅ · +25.8 points over Heres without joint pretraining

    Success rate over 20 trials per task; average across six tasks. All methods are fine-tuned on the same task demonstrations.

    05 / Ablation Study

    Action forcing + visual memory

    The benefit of joint data pretraining

    RoboTwin 2.0 · Clean-to-Random

    No robot or egocentric data pretraining62.7%
    Clean 82.9% · Randomized 42.6%Baseline
    Robot data pretraining77.4%
    Clean 87.0% · Randomized 67.7%
    Robot + egocentric data pretraining80.9%
    Clean 88.3% · Randomized 73.4%

    Deployment efficiency

    957.2 → 103.9 ms median model-call latency
    957.2 msJoint video/action · 10/10 steps
    269.7 msDecoupled video/action · 1/10 steps
    126.6 msCompilation, without CUDA Graphs
    103.9 msFinal · CUDA Graph replay

    Citation

    Download .bib
    @misc{cen2026worldbridgebridgingvideoaction,
      title={WorldBridge: Bridging Video and Action for Task Generalization in World Action Models},
      author={Jun Cen and Yubo Guo and Hao Li and Liangkai Li and Yaoxu Lyu and Meng Wei and Zeqi Xiao and Sizhe Yang and Yuqiang Yang and Yifan Ye and Shanghang Zhang and Duo Zheng},
      year={2026},
      eprint={2610.40748},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.40748},
}