TrackCraft3R: Repurposing Video Diffusion Transformers for Dense 3D Tracking
3d ago
Quick Take
TrackCraft3R repurposes video diffusion transformers for efficient dense 3D tracking from monocular video.
Key Points
- Introduces a feed-forward dense 3D tracking method.
- Utilizes dual-latent representation for improved tracking.
- Achieves state-of-the-art performance with reduced memory usage.
📖 Reader Mode
~2 min readAbstract:Dense 3D tracking from monocular video is fundamental to dynamic scene understanding. While recent 3D foundation models provide reliable per-frame geometry, recovering object motion in this geometry remains challenging and benefits from strong motion priors learned from real-world videos. Existing 3D trackers either follow iterative paradigms trained from scratch on synthetic data or fine-tune 3D reconstruction models learned from static multi-view images, both lacking real-world motion priors. Pre-trained video diffusion transformers (video DiTs) offer rich spatio-temporal priors from internet-scale videos, making them a promising foundation for 3D tracking. However, their frame-anchored formulation, which generates each frame's content, is fundamentally mismatched with reference-anchored dense 3D tracking, which must follow the same physical points from a reference frame across time. We present TrackCraft3R, the first method to repurpose a video DiT as a feed-forward dense 3D tracker. Given a monocular video and its frame-anchored reconstruction pointmap, TrackCraft3R predicts a reference-anchored tracking pointmap that follows every pixel of the first frame across time in a single forward pass, along with its visibility. We achieve this through two designs: (i) a dual-latent representation that uses per-frame geometry latents and reference-anchored track latents as dense queries, and (ii) temporal RoPE alignment, which specifies the target timestamp of each track latent. Together, these designs convert the per-frame generative paradigm of video DiTs into a reference-anchored tracking formulation with LoRA fine-tuning. TrackCraft3R achieves state-of-the-art performance on standard sparse and dense 3D tracking benchmarks, while running 1.3x faster and using 4.6x less peak memory than the strongest prior method. We further demonstrate robustness to large motions and long videos.
| Comments: | Project page and code are available at this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2605.12587 [cs.CV] |
| (or arXiv:2605.12587v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12587 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jisu Nam [view email]
[v1]
Tue, 12 May 2026 17:59:27 UTC (6,725 KB)
— Originally published at arxiv.org
More from arXiv cs.CV
See more →2d ago
FeaturedOriginal
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
AI Summary
CoReDiT enhances Diffusion Transformers by optimizing token pruning for efficiency and quality.
