E$^3$C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Control
Quick Take
E$^3$C is a novel video diffusion framework for egocentric video generation, enhancing visual fidelity and control over human dynamics. It outperforms strong baselines on the Nymeria benchmark by improving camera motion accuracy and enabling intuitive scene editing.
Key Points
- E$^3$C constructs a semi-dense 3D memory from context frames.
- It uses skeleton renderings for exo human control and 3D joints for ego control.
- An ego motion encoder maintains control even when body parts are occluded.
- The framework shows improved object consistency and camera motion accuracy.
- Intuitive scene editing capabilities are enabled through structured conditions.
Article Content
From source RSS / original summaryarXiv:2605. 26316v1 Announce Type: new Abstract: Controllable and physically grounded egocentric video generation is essential for embodied agents to reason about how their own and others' actions manifest and change the world.
Compared to generic video synthesis, egocentric generation is especially challenging: the camera is tightly coupled to the actor, leading to rapid viewpoint changes and frequent self-occlusions; the underlying actions are subtle, articulated, and often only partially visible; and both the people and the scene state must evolve consistently with the specified controls.
We present E$^3$C, a controllable video diffusion framework for egocentric generation that builds structured and compact conditions disentangling persistent scene structure from human-driven dynamics. From context frames, E$^3$C constructs a semi-dense point cloud-based 3D memory and augments each point with appearance descriptors from video-VAE features. Rendering this memory into target viewpoints produces conditioning aligned with the target frames. Human dynamics are modeled separately.
The observed people in the scene are controlled by skeleton renderings (exo human control), while the camera wearer is specified by their 3D body joints and 6DoF wrist motion (ego human control). To preserve ego human control when the wearer's body parts are invisible, we introduce an ego motion encoder that produces persistent cross-attention tokens.
Experiments on Nymeria show that E$^3$C improves visual fidelity, camera-motion accuracy, object consistency, and ego & exo human control over strong baselines, while also enabling intuitive scene editing.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.
