Learning to perform unseen robot manipulation tasks from a single human demonstration video.
Figure 1. CapPro uses a phase-aware representation of a single human video to provide task-specific guidance for a pretrained VLA policy. Auxiliary phase-caption supervision is used during training only. Click figure to enlarge ↗
Vision-language-action (VLA) models have demonstrated strong manipulation capabilities, yet adapting them to previously unseen tasks often requires additional robot demonstrations and fine-tuning. CapPro explores a different setting: a single egocentric human demonstration video provides task-specific guidance for a pretrained VLA policy. The framework learns compact, phase-aware video representations and integrates the resulting guidance into robot action conditioning, without relying on embodiment-aligned human-to-robot trajectories or target-task robot fine-tuning.
This is a short project-page summary; the manuscript abstract may be revised independently.
Converts the human demonstration into a compact visual representation, preserving task-relevant spatial and temporal information. Context-dependent queries learn execution-stage information using phase-caption supervision during training.
Combines video features and query-derived guidance with the robot observation and language instruction, conditioning the pretrained VLA policy for action generation on an unseen manipulation task.
Source and unseen target tasks with different levels of task shift, including object, interaction, and compositional changes.
Evaluation on physical manipulation tasks, with demonstrations and representative execution sequences to be added.