RAP-WAM.
Learning World-Action Models through
Predictive Rehearsal and Action Perturbations
Turn a video prior into a robot policy. Learn from predicted observations, then use action-induced video responses to guide supervision across space and time.
Circle-size selection
Real-robot task-macro success
Over the strongest of five baselines
Shared real-robot comparison
RoboDojo simulation evaluation
A video model can do more than predict.
Future observations provide dense supervision for robot control. RAP-WAM asks how a joint predictor can make that supervision more useful—through its own predictions and its responses to actions.
The base is adapted directly from a video-pretrained backbone using target-task demonstrations, without intermediate pretraining on external embodied data. Agent-guided refinement then retains task-specific improvements.
Read the abstract+
Rehearse the future.
Probe the response.
Three connected designs adapt the video prior and organize supervision. Explore each step below.
From the predictor
to the physical world.
Explore the real-robot comparisons and the base-to-system gains in simulation. Every value comes from the manuscript's result records.
The system selects one retained checkpoint per task. Aggregates average within each capability, then equally across the five groups. Score measures partial completion.
Explore all 42 task results+
| Task | Capability | Base SR | System SR | Gain |
|---|
Sixteen tasks.
See them in motion.
Reasoning about sizes and numbers, coordinating two hands, and assembling everyday objects. Open any recording to watch the complete task.
The badges report task-level success rates from the paper. The videos are selected execution recordings. Phone-box assembly is supplementary and is excluded from the 15-task macro average.
What does the model
see next?
Compare recorded and predicted frames from the same training window. Move the divider, change the task, or advance through the future.
51 paired examples: all 16 real-robot tasks and 35 simulation task conditions. Burger-box assembly adds a saved final-step training example.
A future shaped
by action.
The predictor generates future observations together with robot actions. These paired examples show how its forecasts compare with the recorded continuation.
Metrics average the five displayed future offsets. These are training-sample diagnostics, separate from the physical success-rate evaluation.
See the appendix for the full analysis ↗Prediction error tells
only part of the story.
of response-top20 positions fall outside error-top20, across 315 fixed-output simulation windows.
Action perturbations reveal where video predictions depend on actions. Response, current error, and the rehearsal prior guide spatial visual supervision; spatially pooled response guides supervision across future action steps.
Go deeper.
The full method, evaluation, and supplementary visualizations are in the manuscript.
Citation
@misc{he2026rapwam,
title={RAP-WAM: Learning World-Action Models through Predictive Rehearsal and Action Perturbations},
author={Yuanpeng He and Ceyao Zhang and Fangjing Li and Kefei Zhu and Lijian Li and Yueheng Li and Tianjia He and Wenpin Jiao and Zhi Jin and Yaodong Yang and Yuanpei Chen},
year={2026},
eprint={2610.16443},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2610.16443}
}