WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories

WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories

   Title: 
   

  Content selection saved. Describe the issue below:

   Description: 
    
 

License: CC BY 4.0

arXiv:2603.02049v1 [cs.CV] 02 Mar 2026

WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories

Yisu Zhang 1,2 1 1 1 Equal Contribution. 2 2 footnotemark: 2 Project Lead. 3 3 footnotemark: 3 Corresponding author ,   Chenjie Cao 2 1 1 1 Equal Contribution. 2 2 footnotemark: 2 Project Lead. 3 3 footnotemark: 3 Corresponding author ,   Tengfei Wang 2 2 2 footnotemark: 2 ,

Xuhui Zuo 2 ,   Junta Wu 2 ,   Jianke Zhu 1 3 3 footnotemark: 3 ,   Chunchao Guo 2
1 Zhejiang University    2 Tencent Hunyuan

Abstract

Recent advances in foundational Video Diffusion Models (VDMs) have yielded significant progress. Yet, despite the remarkable visual quality of generated videos, reconstructing consistent 3D scenes from these outputs remains challenging, due to limited camera controllability and inconsistent generated content when viewed from distinct camera trajectories. In this paper, we propose WorldStereo , a novel framework that bridges camera-guided video generation and 3D reconstruction via two dedicated geometric memory modules. Formally, the global-geometric memory enables precise camera control while injecting coarse structural priors through incrementally updated point clouds. Moreover, the spatial-stereo memory constrains the model’s attention receptive fields with 3D correspondence to focus on fine-grained details from the memory bank. These components enable WorldStereo to generate multi-view-consistent videos under precise camera control, facilitating high-quality 3D reconstruction. Furthermore, the flexible control branch-based WorldStereo shows impressive efficiency, benefiting from the distribution matching distilled VDM backbone without joint training. Extensive experiments across both camera-guided video generation and 3D reconstruction benchmarks demonstrate the effectiveness of our approach. Notably, we show that WorldStereo acts as a powerful world model, tackling diverse scene generation tasks (whether starting from perspective or panoramic images) with high-fidelity 3D results. Models will be released on https://github.com/FuchengSu/WorldStereo.

Figure 1 :
WorldStereo enables high-quality 3D scene generation based on single-view or panoramic inputs. The input reference views are framed in green. We present point clouds reconstructed from videos generated by WorldStereo: the top two perspective scenes use WorldMirror  [ 58 ] , while the bottom two panoramic scenes are aligned via monocular depth maps  [ 84 ] .

1

Introduction

Recently, remarkably developed Video Diffusion Models (VDMs)  [ 7 , 99 , 8 , 47 , 67 , 46 , 78 ] have made impressive advances, achieving remarkable performance in photorealistic video synthesis. These models have showcased significant potential across a broad range of applications, including virtual reality, digital content creation, and embodied AI. Meanwhile, camera-controllable VDMs have also enabled substantial progress, incorporating various camera and action controls  [ 31 , 3 , 116 , 32 , 53 ] , as well as leveraging explicit guidance like point clouds  [ 106 , 22 , 60 , 66 , 50 , 64 , 13 ] , optical flow  [ 43 , 9 ] , and tracking points  [ 28 , 79 ] .

Despite these breakthroughs, current camera-guided VDMs remain limited in recovering consistent and reliable 3D scene reconstructions, even when paired with advanced feed-forward 3D reconstruction approaches  [ 96 , 83 , 82 , 44 ] , failing to step toward generalizable world models. A fundamental challenge lies in enabling VDMs to generate videos that capture sufficiently diverse and comprehensive viewpoints of the target scene, while maintaining precise camera control and high-fidelity visual quality . Existing camera-guided VDMs struggle to preserve consistency across varied camera trajectories, leading to ambiguous and blurry reconstructions. Intuitively, extending them into longer sequences can capture richer viewpoint coverage with natural consistency from the global attention of diffusion transformers (DiTs)  [ 63 ] , but this often comes at inferior video quality and prohibitive computation for both fine-tuning and inference. On the other hand, autoregressive (AR) VDMs  [ 14 , 40 ] improved the efficiency for long video generation via sequential synthesis, yet they suffer from limited camera precision  [ 72 ] and error accumulation  [ 40 ] . Moreover, both long video generation and AR models lack mature open-source communities compared to regular VDMs like HunyuanVideo  [ 46 ] , CogVideoX  [ 99 ] , and Wan  [ 78 ] .

Table 1 : Different video generation schemes for 3D reconstruction. Long-Bi VDMs produce long trajectories in a single pass to cover diverse viewpoints. AR models sequentially generate long videos in an autoregressive manner. Multi-Bi-Mem (ours) achieves multiple consistent generations based on a powerful open-released VDM  [ 78 ] with complementary viewpoints and memory mechanisms for integrated reconstruction.

Paradigms
Long-Bi
AR
Multi-Bi-Mem

Receptive Field Bidirectional Autoregressive Bidirectional

Trajectory Length/Num Long/Single Long/Single Medium/Multiple

Consistency Precise Camera Control

Video Quality

Efficiency

Community Progress

To overcome these challenges, we present WorldStereo , a novel framework that bridges the gap between camera-guided VDMs and 3D scene reconstruction by enabling consistent multi-trajectory video generations with geometry-aware memories , as summarized in Table   1 . Specifically, two complementary mechanisms, global-geometric and spatial-stereo memory, are incorporated into off-the-shelf VDMs to memorize coarse structures and fine-grained details, respectively. These two components enable precise and coherent video synthesis across diverse trajectories. WorldStereo subtly sidesteps the long sequence generation and largely preserves the generalization and usability of pre-trained VDMs, resulting in impressive 3D reconstruction as shown in Figure   1 .

Formally, WorldStereo is built upon the camera-guided VDM, Uni3C  [ 13 ] , with the extended capability to memorize consistent generation along different trajectories. We first augment Uni3C’s point cloud guidance as an incrementally updated Global-Geometric Memory (GGM) through iterative feed-forward 3D reconstruction. However, we find that GGM fails to preserve fine-grained details. Inspired by stereo matching  [ 61 ] , we introduce the Spatial-Stereo Memory (SSM): learning spatial coherence between the generated novel views and retrieved views from a deliberate memory bank by establishing explicit 3D correspondences. Furthermore, we constrain the attention receptive fields of each novel view to focus on the specific retrieved one, thereby enhancing detailed consistency. Additionally, WorldStereo inherits the flexibility of Uni3C: all pixel-wise aligned conditions are injected via the ControlNet branch, which is highly compatible with the distribution matching distillation (DMD)  [ 102 , 101 ] and eliminates the need for joint training. This technique significantly accelerates WorldStereo under 4-step inference without compromising generalization and consistency.

Extensive experiments confirm the effectiveness of WorldStereo, including both in-domain and out-of-distribution benchmarks. Our approach achieves superior camera-motion accuracy and higher-quality video generation. To further demonstrate its contributions to 3D reconstruction, we present a new 3D reconstruction benchmark to evaluate the output quality of camera-guided VDMs. We meticulously process and crop the ground-truth point clouds for Tanks-and-Temples  [ 45 ] and MipNeRF360  [ 4 ] , enabling comprehensive assessment of 3D consistency, visual quality, and camera trajectory fidelity. Our main contributions are summarized as follows:

•
Geometry-aware memory enhanced VDM. WorldStereo contains two complementary memory mechanisms to generate multi-trajectory consistent videos tailored for 3D reconstruction.

•
Flexible framework with efficient inference. Leveraging pixel-wise aligned ControlNet injection, WorldStereo can be accelerated via DMD without requiring joint training or sacrificing generalization.

•
Customized evaluation. We present a new 3D reconstruction benchmark to evaluate the outcome quality of camera-guided VDMs.

2

Related Work

Camera-Guided Video Generation. Taming high fidelity video generations  [ 95 , 117 , 55 , 46 , 99 , 78 ] under controllable viewpoints and camera trajectories serves as a pivotal pathway for world simulation. Some works tried to implicitly control the VDM under specific camera motion via tailored LoRA tuning  [ 30 , 6 , 74 , 114 ] . Meanwhile, implicitly discrete controls are also widely adopted in action-based VDMs  [ 113 , 48 ] and AR models  [ 20 , 29 , 33 ] . MotionCtrl  [ 88 ] first explicitly injected camera poses into the pre-trained VDM. Subsequent works further extend the camera presentation into Plücker ray  [ 31 , 50 , 97 , 3 , 32 ] . To enhance the camera control precision under metric scales, point clouds  [ 106 , 22 , 60 , 66 , 50 , 64 , 13 ] , mesh  [ 36 , 98 ] , optical flow  [ 43 , 9 ] , and tracking points  [ 28 , 79 ] have been employed for camera-guided VDMs, spanning both training-based and training-free manners  [ 37 , 35 , 107 ] . Despite the precise camera controllability achieved by these pioneering efforts, we note that they struggle to produce convincing 3D reconstruction due to constrained video lengths and inadequate consistency caused by memoryless visual conflicts.

Memory-based Video Generation. Existing video generation fails to maintain geometrically accurate structures during long-trajectory synthesis. A straightforward solution involves training VDMs with extended context lengths  [ 73 , 77 , 16 ] to memorize more visual clues, yet computationally costly. To reduce the of long contexts computation, concurrent works either compress prior frames into a condensed context window  [ 27 , 110 ] or inject historical frames via attention mechanisms  [ 118 , 68 , 94 ] , which inevitably cause information loss and compromise 3D consistency. Another research direction iteratively reconstructs 3D representations from historical frames, utilizing these as either conditional guidance  [ 60 , 106 , 66 , 28 , 91 ] or retrieved references  [ 105 , 49 ] for future contexts synthesis. But they also suffer from error accumulation and degraded fine-grained details. Our geometry-aware memory preserves both 3D consistency and high-fidelity details by unifying 3D correspondence modeling and a tailored attention mechanism.

Feed-Forward 3D Reconstruction. In contrast to traditional Structure from Motion (SfM)  [ 24 , 69 , 62 ] , Multi-View Stereo (MVS)  [ 100 , 26 , 108 , 11 ] , and learning-based SLAM  [ 52 , 38 ] , recent feed-forward 3D reconstruction approaches efficiently predict both camera poses, point clouds, and depth maps within a single forward pass. Dust3R  [ 85 ] pioneered learning two-view correspondence through powerful transformer models pre-trained by cross-view inpainting  [ 89 ] . Many follow-ups further investigated this realm, including multi-view-based  [ 96 , 96 ] , sequential-based  [ 19 , 83 ] , dynamic-based  [ 109 , 59 ] and conditional priors-based  [ 42 , 44 , 58 ] reconstructions. For impressive visual reconstruction, some works have focused on feed-forward 3D Gaussian Splatting (3DGS) generation  [ 18 ] , enabling fast, optimization-free 3DGS representations. However, 3D reconstruction methods follow “what you see is what you get”, i.e . , they require sufficient images to capture 3D structures, suffering from inferior results explored beyond observed viewpoints.

3D Scene Generation. One popular scene generation workflow involves iterative warp-and-inpaint , combining depth estimation, alignment, and image inpainting  [ 23 , 34 , 54 , 71 , 104 , 80 ] . While iterative scene generations perform well in novel view synthesis (NVS), they suffer from prohibitive per-scene optimization and inconsistent geometry across views. Another prominent paradigm can be summarized as generate first, then reconstruct , yielding various 3D outcomes from multi-view images  [ 25 , 12 , 118 ] , panoramas  [ 41 ] and camera-guided videos  [ 74 , 106 , 115 , 53 ] . However, both of them face distinct limitations: multi-view images cover adequate viewpoints but lack view consistency, while generated videos are insufficiently long to capture disparate viewpoints. Recent works have unified generation and reconstruction in end-to-end frameworks, simultaneously modeling depth  [ 39 , 120 , 17 ] , 3DGS  [ 97 , 2 , 51 ] , and point map  [ 75 , 112 ] rather than relying solely on RGB frames. Though these methods largely alleviate 3D consistency issues, they are highly data-hungry and require substantial training, potentially hindering the generalization of foundation models. Our method preserves the original output formulation of VDMs with retained generalization, while capturing more consistent viewpoints via a novel memory mechanism for 3D reconstruction.

3

Method

Figure 2 : Overview of WorldStereo. WorldStereo comprises two ControlNet branches. The camera branch ensures precise camera control and Global-Geometric Memory (GGM), depending on global point clouds; the Spatial-Stereo Memory (SSM) branch leverages retrieved reference frames and pointmap (3D correspondence) guidance obtained from the 3D cache to further preserve fine-grained consistency. We omit the diffusion noise part for simplicity.

Overview. Given a single image, WorldStereo aims to reconstruct a complete 3D scene following the “generate first, then reconstruct” paradigm. We summarize the generation pipeline in Figure   2 . We first introduce the camera-guided VDM with point cloud guidance and our basic memory storage in Section   3.1 . Next, the global-geometric ( Section   3.2 ) and spatial-stereo memories ( Section   3.3 ) are proposed to generate multiple video sequences with consistent 3D geometry and textures for subsequent 3D reconstruction. To improve inference efficiency, WorldStereo adopts a modified distribution matching distillation ( Section   3.4 ).

3.1

Preliminaries

Uni3C   [ 13 ] is used as the base camera-guided VDM in WorldStereo. Formally, Uni3C is built upon the pre-trained Wan-I2V  [ 78 ] , integrated with a lightweight ControlNet branch  [ 111 ] trained from scratch. Except for the camera Plücker rays, Uni3C incorporates point clouds extracted from the reference frame as 3D geometric priors, derived via back-projected monocular depth. Point clouds X p ​ c ​ d X_{pcd} of the reference image can be presented as:

    X   p  ​  c  ​  d    ​   (  x  )    ≃    R   c  →  w    ​  D  ​   (  x  )   ​   K   −  1    ​   x  ^     ,   X_{pcd}(x)\simeq R_{c\rightarrow w}D(x)K^{-1}\hat{x},    

(1)

where R c → w R_{c\rightarrow w} denotes the camera-to-world pose matrix; D ​ ( ⋅ ) D(\cdot) is the depth at pixel x x estimated by MoGe  [ 84 ] ; K − 1 K^{-1} is the inverse of the camera intrinsic matrix; x ^ \hat{x} is the homogeneous pixel coordinate. Point clouds serve as strong geometric guidance to facilitate fast convergence and precise camera control, avoiding compromising the generalization of frozen foundational VDMs.

Memory Bank & 3D Cache. WorldStereo incorporates two memory components: a 2D memory bank and a 3D cache, as in Figure   2 . Formally, generated video frames are first temporally downsampled and stored in the memory bank as { I m ​ e ​ m } m = 0 M {I_{mem}}{m=0}^{M} , serving to retrieve spatially similar reference views for the subsequent spatial-stereo memory. Note that the initial condition image and perspective views split from the 360 ∘ panorama are also included in the memory bank. The 3D cache saves global point cloud set X c ​ a ​ c ​ h ​ e X{cache} , reconstructed based on memory bank images using the feed-forward 3D reconstruction method: WorldMirror  [ 58 ] . Specifically, we incrementally reconstruct point clouds for each generated video. To tackle long sequences, disparate 3D caches are merged by aligning point clouds from overlapping views via Umeyama transformation  [ 76 ] .

3.2

Global-Geometric Memory

We propose Global-Geometric Memory (GGM) that iteratively updates point cloud conditions, serving as global 3D priors for generating multiple consistent videos, iterative video continuation, and even supporting panorama-based 3D generation, as verified in Section   4.4 . As discussed in  [ 13 ] , point clouds just provide camera guidance rather than forcing the VDM to overfit to these 3D presentations. While this phenomenon is reasonable for camera-guided VDMs to preserve overall generalization from the degradation caused by inferior monocular depth, it causes our model to ignore most geometric structures brought by point clouds, even when the point clouds themselves are perfectly reconstructed.

To overcome this, we fine-tune the original control branch of WorldStereo using extended global point clouds X p ​ c ​ d g X^{g}{pcd} beyond the reference points ( X p ​ c ​ d X{pcd} from Eq.  1 ) as:

   X   p  ​  c  ​  d   g   =   [   X   p  ​  c  ​  d    ,    X  ^    p  ​  c  ​  d    ]    ,   X^{g}_{pcd}=[X_{pcd},\hat{X}_{pcd}],    

(2)

where X ^ p ​ c ​ d \hat{X}{pcd} denotes point clouds from other views. To avoid overfitting novel views’ point clouds during training, we introduce a point cloud masking strategy: randomly dropping a subset of points from target views to confirm robustness to partially geometric missing. We detail the sampling strategy in supplementary. For inference, we use the incrementally updated 3D cache as X ^ p ​ c ​ d \hat{X}{pcd} , which is aligned to the same coordinate as X p ​ c ​ d X_{pcd} via the Umeyama transformation  [ 76 ] among the overlapping point clouds from monocular depth estimation and WorldMirror. Notably, GGM is also compatible with panorama-based 3D generation. Since panoramas capture 360 ∘ views, VDMs have to maintain visually consistent content during viewpoint transitions. To this end, we follow the MoGe panorama depth estimation  [ 84 ] to construct panoramic point clouds as the initialized 3D cache for our panoramic 3D generation.

Figure 3 : Spatial-Stereo Memory (SSM). Reference views are retrieved from the memory bank, while pointmaps for both target and reference views are constructed based on the 3D cache. In SSM attention, we horizontally stitch each target-reference pair and rearrange the tensor shape to make each target frame’s features focus on the specifically retrieved reference. B, F, H, W, C indicate dimensions of batch, frame, height, width, and channels.

3.3

Spatial-Stereo Memory

Although GGM maintains coarse structures using point clouds from 3D cache, it struggles to preserve fine-grained details, as shown in  Figure   5 . Many previous studies  [ 105 , 118 , 68 , 49 ] retrieve historical reference frames and jointly model all frames via full-attention. However, this formulation requires substantial post-training to enable VDMs to adapt to long sequences. Moreover, we cannot guarantee the continuity of retrieved frames ( e.g . , panoramic scenarios). These disparate, unordered reference views further hinder the VDM learning process. Thus, we get inspirations from the traditional stereo matching  [ 61 ] and the reference-based inpainting  [ 10 ] and propose the Spatial-Stereo Memory (SSM) as illustrated in Figure   3 . Formally, we discretely retrieve reference views and spatially stitch each with its corresponding target view. We then constrain the attention receptive field to each reference-target pair, facilitating enhanced fine-grained detail recovery.

In practice, given N N target poses, we first sample F = N / 4 F=N/4 poses uniformly and retrieve their nearest-neighbor frames from the memory bank as references. We extend the retrieval strategy of  [ 105 ] from 2D planes to 3D spaces, selecting views where the volumetric overlapping fields-of-view (FoV) between the target and reference camera frustums are maximized. To ensure that fine-grained details are preserved after 3D-VAE encoding, we separately encode each retrieved reference view as an independent image, yielding latent features { z r ​ e ​ f } i = 1 F {z_{ref}}{i=1}^{F} . Then we horizontally stitch target and reference latent features as z s ​ t ​ i ​ t ​ c ​ h = [ z t ​ a ​ r ; z r ​ e ​ f ] ∈ ℝ F × 2 ​ H ​ W × C z{stitch}=[z_{tar};z_{ref}]\in\mathbb{R}^{F\times 2HW\times C} , where H , W , C H,W,C denote the height, width, and channel dimension of latent features. To improve the geometry-aware perception, we incorporate a pointmap to each stitched target-reference pair z s ​ t ​ i ​ t ​ c ​ h z_{stitch} , which indicates 3D corresponding information derived from our 3D cache. Specifically, the pointmap records point cloud positions of the target-reference pair in the 3D world coordinate. We normalize and colorize this into an RGB format, which is then encoded into latent features by 3D-VAE—denoted as z ^ t ​ a ​ r \hat{z}{tar} and z ^ r ​ e ​ f \hat{z}{ref} for target and reference pointmaps, respectively. The stitched pointmap latents are thus formulated as z ^ p ​ m = [ z ^ t ​ a ​ r ; z ^ r ​ e ​ f ] ∈ ℝ F × 2 ​ H ​ W × C \hat{z}{pm}=[\hat{z}{tar};\hat{z}{ref}]\in\mathbb{R}^{F\times 2HW\times C} . Subsequently, we achieve the final inputs for the SSM branch as z s ​ s ​ m = z s ​ t ​ i ​ t ​ c ​ h + z ^ p ​ m z{ssm}=z_{stitch}+\hat{z}_{pm} . Our ablation studies in  Figure   5 (c)(d) confirm that this 3D correspondence information is critical to SSM’s performance. The model architecture of the SSM branch is similar to the camera branch, comprising 20-layer DiT blocks trained from scratch. A key distinction lies in the SSM attention constraint: instead of full attention, we limit the attention receptive field such that each target-reference pair attends exclusively to its own features—facilitating more precise fine-grained learning. As shown in Figure   3 , we rearrange the feature to [ B ​ F , H ∗ 2 ​ W , C ] [BF,H2W,C] and restrict SSM attention to operate solely along the H ∗ 2 ​ W H2W dimension. Then, only the target features will be added to the main VDM block.

Data Curation.

Intuitively, training SSM needs multi-view videos to construct reference-target pairs, which is difficult to achieve in the real world. To overcome this, we generate training pairs by temporally misaligned sampling existing multi-view data  [ 56 , 92 , 86 , 1 ] , ensuring that the reference video and target video have a temporal overlap of 30% to 90%. Then we randomly shuffle and apply masks to reference views to confirm the robustness, as well as simulate the disorder and discreteness of real-world retrieval scenarios. More details are discussed in the supplementary.

3.4

Acceleration via DMD

We apply the modified Distribution Matching Distillation (DMD)  [ 101 ] to accelerate the inference of WorldStereo. DMD extends the idea of Variational Score Distillation (VSD)  [ 87 ] , distilling a few-step diffusion student G θ G_{\theta} through the approximate Kullback-Liebler (KL) divergence built from the difference between the frozen real score function s r ​ e ​ a ​ l s_{real} and the trainable fake score function s f ​ a ​ k ​ e s_{fake} . The update gradient of DMD can be written as:

   ∇   ℒ  DMD    =   −    𝔼  𝑡   ​   (   ∫    (     s  real   ​   (   x  t   ,  t  )    −    s  fake   ​   (   x  t   ,  t  )     )   ​    d  ​   x  t     d  ​  θ    ​   𝑑  z     )      ,   \hskip-10.00002pt\nabla\mathcal{L}_{\text{DMD}}=-\underset{t}{\mathbb{E}}\left(\int\left(s_{\text{real}}(x_{t},t)-s_{\text{fake}}(x_{t},t)\right)\frac{dx_{t}}{d\theta}dz\right),    

(3)

where x = G θ ​ ( z ) x=G_{\theta}(z) denotes the student generation given random Gaussian noise z ∼ 𝒩 ​ ( 0 ; 𝐈 ) z\sim\mathcal{N}(0;\mathbf{I}) and t ∼ 𝒰 ​ ( 0 , 1 ) t\sim\mathcal{U}(0,1) , while x t ∼ q t ​ ( x t | x , t ) x_{t}\sim q_{t}(x_{t}|x,t) indicates the forward diffusion process.

The generator G θ G_{\theta} of WorldStereo is distilled into a 4-step DiT. G θ G_{\theta} , s r ​ e ​ a ​ l s_{real} , s f ​ a ​ k ​ e s_{fake} are all initialized from the camera-guided VDM (Uni3C): s r ​ e ​ a ​ l s_{real} is frozen, while G θ G_{\theta} and s f ​ a ​ k ​ e s_{fake} are trainable. Following  [ 101 ] , we train s f ​ a ​ k ​ e s_{fake} 5 times per generator update. The stochastic gradient truncation  [ 40 ] is employed to stabilize the training phase. We omit the GAN loss, as we found its impact to be insignificant while substantially slowing down training. Moreover, the DMD training is based on pure camera-guided video generation without any memory training. To decouple the control and the basic few-step generation capabilities, we freeze the camera-guided control branch of the generator G θ G_{\theta} , leaving only its main backbone trainable. Notably, both the camera-guided and memory-based control branches can be generalized to the distilled generator G θ G_{\theta} without any joint fine-tuning. This largely simplifies the DMD training process while preserving the generalization (annotated memory data requires well-aligned depth maps, which is less than the camera control data). Importantly, all parameters of the fake score function remain trainable, facilitating s f ​ a ​ k ​ e s_{fake} to track the generator’s distribution. Additionally, we empirically find that retaining high-quality and relatively easy trajectories is critical for the stable training of DMD. The student model prefers to attend to some artifacts ( e.g . , over-saturation and hallucination) from the teacher model, especially under challenging trajectories or low-quality reference images. As verified in Footnote   † , this data filter strategy does not degrade the camera controllability.

4

Experiments

4.1

Implementation Details

The frozen VDM of WorldStereo is built upon Wan2.1-14B-I2V  [ 78 ] , with the camera ControlNet retrained under the same setting as  [ 13 ] for 8,000 steps under a batch size of 32. For the GGM training in the second phase, we fine-tune the camera ControlNet with the global point cloud augmentation for 4,000 steps. For the third phase of SSM training, we train a new branch from scratch for 6,000 steps based on tailored memory-retrieval data as detailed in Section   3.3 . Training both memory mechanisms costs 60 hours on 64 NVIDIA H20 GPUs. For DMD training, we set the classifier-free guidance (CFG) scale to 5.0 for the real score function, leading to a CFG-free generator with 2x inference efficiency. Furthermore, we reduce the inference denoising steps from 40 to 4, achieving an overall speedup of 20×. We train DMD for 1,000 steps within 13 hours under the same resources. All training data are resized to 480p with flexible aspect ratios. We also verify that our method can be generalized to 720p inference in supplementary.

Datasets. The training data of our camera control includes DL3DV  [ 56 ] , Real10k  [ 119 ] , Tartanair  [ 86 ] , Map-Free-Reloc  [ 1 ] , WildRGBD  [ 92 ] , and UE5 rendering data. For memory training, we discard Real10k due to its low-quality images and overly simple camera trajectories. We further eliminate the Tartanair and narrow the frame interval of DL3DV for stable DMD training. More details are discussed in the supplementary.

Table 2 : Quantitative results of OOD benchmark with WorldScore  [ 21 ] images . ∗ indicates the baseline version of our method without any memory mechanism, while the ‘full’ version denotes adding both GGM and SSM. The ‘DMD’ version is based on ‘WorldStereo-full’. Inference times are all tested with 8 H20 GPUs. † † † All methods are evaluated under their pre-defined resolutions and frame numbers, while the settings of Uni3C and WorldStereo series are the same for fairness (512p and 81-frame).

RotErr ↓ \downarrow

TransErr ↓ \downarrow

ATE ↓ \downarrow

Q-Align-I ↑ \uparrow

Q-Align-V ↑ \uparrow

CLIP-I ↑ \uparrow

CLIP-T ↑ \uparrow

CLIP-IQA+ ↑ \uparrow

Laion-Aes ↑ \uparrow

Time (sec)

Voyager  [ 39 ]

0.678
0.630
1.343
3.279
0.664
82.63
11.399
0.414
4.994
343

SEVA  [ 118 ]

0.171
0.540
1.023
3.907
0.782
91.74
11.194
0.514
5.331
90

Gen3C  [ 66 ]

0.220
0.275
1.071
4.094
0.820
91.78
11.270
0.518
5.332
158

Uni3C  [ 13 ]

0.155
0.192
0.572
4.202
0.846
91.63
11.115
0.549
5.350
162

WorldStereo*
0.132
0.178
0.542
4.273
0.860
91.94
11.145
0.559
5.352
162

WorldStereo-GGM
0.129
0.162
0.706
4.339
0.875
92.24
11.189
0.572
5.408
162

WorldStereo-Full
0.145
0.253
0.667
4.287
0.866
91.80
11.169
0.561
5.369
173

WorldStereo-DMD
0.146
0.203
0.504
4.338
0.874
91.15
11.183
0.573
5.393
9

4.2

Results of Camera Control and Visual Quality

Settings and Metrics.

To fairly evaluate the camera controllability of camera-guided models trained on different datasets, we introduce a new out-of-distribution (OOD) benchmark as shown in Footnote   † . We select 100 images from the static subset of WorldScore  [ 21 ] as the initial frames of this benchmark, including high-quality samples across real-world, stylized, indoor, and outdoor scenarios. Next, we randomly combine translation, rotation, and panning to construct complex camera trajectories. To verify the camera precision, we employ WorldMirror  [ 58 ] on the generated videos to extract predicted cameras and compare Rotation Error (RotErr), Translation Error (TransErr), and Absolute Trajectory Error (ATE) against ground-truth camera guidance. We further evaluate various quality assessments (Q-Align-Image&Video  [ 90 ] , CLIP-Image&Text  [ 65 ] , CLIP-IQA+  [ 81 ] , Laion-Aes  [ 70 ] ) via IQA-PyTorch  [ 15 ] to verify the overall image and video quality.

Analysis.

From Footnote   † , the basic version WorldStereo ∗ outperforms other competitors with superior camera control and visual quality. Moreover, we provide ablation studies on our memory components (GGM, SSM), where the memory bank and 3D cache only store the first frame’s information. While the memory mechanism provides no benefit in single-image conditioned camera control, these results confirm that our memory-based training preserves the model’s generalization and visual quality. In particular, GGM even improves the overall quality of the generated videos, while SSM slightly degrades performance, but enables strong fine-grained details recovery, as verified in Figure   5 (d). We also include the result of DMD, replacing the VDM backbone of our full model with the distilled DiT, showcasing impressive camera control and consistent quality, and a rapidly reduced inference cost.

4.3

Single-View Reconstruction Benchmark

Table 3 : Quantitative results of 3D reconstruction based on Tanks-and-Temples  [ 45 ] and MipNeRF360  [ 4 ] . ∗ indicates the baseline version of our method without any memory mechanism.

Method

F1-Score  ↑ \uparrow

AUC  ↑ \uparrow

RotErr  ↓ \downarrow

TransErr  ↓ \downarrow

ATE  ↓ \downarrow

Tanks&Temples

Uni3C  [ 13 ]

0.424
0.378
0.362
0.1017
0.1572

Gen3C  [ 66 ]

0.416
0.380
0.342
0.0949
0.1704

SEVA  [ 118 ]

0.286
0.293
0.379
0.0949
0.1815

Lyra  [ 2 ]

0.227
0.193
–
–
–

VMem  [ 49 ]

0.386
0.375
0.533
0.1510
0.1922

WorldStereo ∗

0.447
0.389
0.377
0.0990
0.1545

WorldStereo-GGM
0.485
0.411
0.224
0.0885
0.1350

WorldStereo-Full
0.578
0.437
0.247
0.0927
0.1501

WorldStereo-DMD
0.534
0.410
0.291
0.1001
0.1547

MipNeRF360

Uni3C  [ 13 ]

0.352
0.347
0.112
0.0086
0.0104

Gen3C  [ 66 ]

0.356
0.340
0.349
0.0220
0.0318

SEVA  [ 118 ]

0.332
0.311
0.282
0.0138
0.0295

Lyra  [ 2 ]

0.203
0.263
–
–
–

VMem  [ 49 ]

0.256
0.245
0.403
0.0392
0.0752

WorldStereo ∗

0.350
0.342
0.097
0.0076
0.0099

WorldStereo-GGM
0.342
0.346
0.107
0.0079
0.0206

WorldStereo-Full
0.406
0.402
0.114
0.0080
0.0132

WorldStereo-DMD
0.390
0.387
0.159
0.0106
0.0267

Settings and Metrics.

To comprehensively evaluate the quality of video generation and multi-trajectory consistency for reconstruction, we propose a novel 3D reconstruction benchmark based on single-view generation. Our benchmark comprises the training split of Tanks-and-Temples  [ 45 ] and MipNeRF360  [ 4 ] . The training split of Tanks-and-Temples has ground-truth point clouds. For MipNeRF360, we first reconstruct the global point clouds scene by MVS  [ 11 ] and then crop the centric foreground areas as pseudo ground-truth point clouds. Each scene of both datasets only provides a single image as the initial frame. More details about the data curation of this benchmark are discussed in the supplementary. The evaluation workflow includes: 1) generating videos along 4 pre-defined trajectories of up, left, right rotations, and orbit as shown in Figure   4 (a); 2) reconstructing point clouds through WorldMirror  [ 58 ] ; 3) aligning the reconstructed point clouds to the ground-truth ones. For MipNeRF360, we can leverage the initial frame’s MVS depth as an anchor to align point-to-point matched WorldMirror points via Umeyama translation  [ 76 ] . For Tanks-and-Temples, we first align the rotation and translation via the first frame’s camera. Then, we apply ICP  [ 5 ] to optimize a refined scale between the reconstructed and ground-truth point clouds. Except for camera metrics, we include two widely used point cloud metrics. Specifically, we get the precision and recall of point cloud within scene-wise distance thresholds manually adjusted as  [ 45 ] , which evaluate the accuracy and completeness of reconstructed point clouds. Besides, we incorporate the point cloud AUC, calculated as the area under the ROC curve of precision and recall with varying thresholds.

Analysis.

From Table   3 , WorldStereo ∗ without any memory mechanisms already outperforms other methods, while our full model, enhanced with GGM and SSM, achieves substantial improvements on both reconstruction and camera precision. Moreover, WorldStereo-DMD still performs well in 3D reconstruction, showing impressive consistency with significant acceleration.

4.4

Extensive Qualitative Results

Qualitative Comparison.

Figure 4 : Results of 3D reconstruction benchmark. The column (a) shows input views and ground-truth point clouds with pre-defined four trajectories (up, left, right rotations, and orbit). We compare the qualitative results of reconstructed point clouds (left) and generated novel views (right) for each method.

We show qualitative results in Figure   4 . As shown in Figure   4 (a), our 3D reconstruction benchmark features high-quality ground-truth point clouds that focus on foreground objects. The compared methods must retain consistent outcomes with symmetrical and logically coherent structures to achieve good point cloud scores. Thus, this benchmark serves as a reliable evaluation for 3D scene generation. SEVA  [ 118 ] suffers from distorted structures and blurry backgrounds, while Gen3C  [ 66 ] faces challenges in producing consistent videos under different trajectories—both methods yield ambiguous, incomplete reconstructions. In contrast, point clouds reconstructed from WorldStereo’s outputs enjoy remarkable completeness and precision, while its novel views remain consistent across different trajectories and exhibit superior quality.

Memory Mechanisms.

Figure 5 : Ablation studies of memory components. Please see the red-framed regions to check the consistency compared to retrieved references. Baseline results are generated without any memory. GGM can capture coarse structures, but loses fine-grained details. Moreover, the incorporation of pointmap significantly enhances the consistency gained via the reference frames retrieved from the memory bank.

We show the qualitative ablation studies in Figure   5 to verify the individual effectiveness of each memory component. The baseline without any memory mechanism randomly hallucinates new objects in novel views, while GGM largely retains the structural consistency and improves camera control for disparate viewpoint changes (second row), benefiting from the incrementally updated 3D cache. The ability of SSM to preserve fine-grained consistency with the memory bank reference stems from its attention mechanism, which leverages images with 3D guidance instead of drawing information solely from coarse-grained point clouds. We clarify that the 3D correspondence-based pointmap guidance is critical for the SSM to focus on correct matching regions.

3D Panorama Generation.

Figure 6 : Results of 3D panorama generation. Please zoom-in for more details of reconstructed point clouds and novel views.

Thanks to memory capabilities, WorldStereo can be easily extended to 3D panorama generation as shown in Figure   6 . Different from panoramic video generation  [ 103 , 98 , 93 ] , our method largely preserves the generalization of the foundational VDM while producing high-resolution perspective views (all panorama inferences are performed at 576p). Both aforementioned points are critical for 3D reconstruction. Formally, we split the panorama into 27 frames under FoV 90 × 120 90\times 120 as the initial memory bank, and leverage the panoramic depth estimation of MoGe  [ 84 ] to build the 3D cache. We heuristically design wonder trajectories as used in our reconstruction benchmark ( Section   4.3 ) and generate videos for the middle three non-overlap images via WorldStereo.

5

Conclusion

In this paper, we present WorldStereo, a novel camera-guided video generation framework tailored for 3D reconstruction. To address the challenge of capturing long-sequence viewpoints while maintaining 3D consistency, WorldStereo generates multiple consistent videos via two memory mechanisms: Global-Geometry Memory (GGM) and Spatial-Stereo Memory (SSM). GGM incrementally updates a point cloud-based 3D cache to enhance the coherent 3D structures, while SSM learns coherence between generated and retrieved views through 3D correspondence to preserve fine-grained details. Moreover, we modify the distribution matching distillation (DMD) strategy to accelerate WorldStereo for fast inference with negligible performance drop. Additionally, we develop a new 3D reconstruction benchmark for evaluating video generation performance. WorldStereo demonstrates strong performance and generalizes effectively to diverse tasks, such as object-centric, face-forward, and 3D panorama generation.

References

[1] E. Arnold, J. Wynn, S. Vicente, G. Garcia-Hernando, A. Monszpart, V. Prisacariu, D. Turmukhambetov, and E. Brachmann (2022)

Map-free visual relocalization: metric pose relative to a single image .

In European Conference on Computer Vision ,

pp. 690–708 .

Cited by: Table 4 , Table 4 , Appendix B , §3.3 , §4.1 .

[2] S. Bahmani, T. Shen, J. Ren, J. Huang, Y. Jiang, H. Turki, A. Tagliasacchi, D. B. Lindell, Z. Gojcic, S. Fidler, et al. (2025)

Lyra: generative 3d scene reconstruction via video diffusion model self-distillation .

arXiv preprint arXiv:2509.19296 .

Cited by: §2 , Table 3 , Table 3 .

[3] S. Bahmani, I. Skorokhodov, G. Qian, A. Siarohin, W. Menapace, A. Tagliasacchi, D. B. Lindell, and S. Tulyakov (2025)

AC3D: analyzing and improving 3d camera control in video diffusion transformers .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

Cited by: §1 , §2 .

[4] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman (2022)

Mip-nerf 360: unbounded anti-aliased neural radiance fields .

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

pp. 5470–5479 .

Cited by: Appendix C , §1 , §4.3 , Table 3 , Table 3 .

[5] P. J. Besl and N. D. McKay (1992)

Method for registration of 3-d shapes .

In Sensor fusion IV: control paradigms and data structures ,

Vol. 1611 , pp. 586–606 .

Cited by: §4.3 .

[6] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. (2023)

Stable video diffusion: scaling latent video diffusion models to large datasets .

arXiv preprint arXiv:2311.15127 .

Cited by: §2 .

[7] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al. (2023)

Stable video diffusion: scaling latent video diffusion models to large datasets .

arXiv preprint arXiv:2311.15127 .

Cited by: §1 .

[8] T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh (2024)

Video generation models as world simulators .

External Links: Link

Cited by: §1 .

[9] R. Burgert, Y. Xu, W. Xian, O. Pilarski, P. Clausen, M. He, L. Ma, Y. Deng, L. Li, M. Mousavi, et al. (2025)

Go-with-the-flow: motion-controllable video diffusion models using real-time warped noise .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 13–23 .

Cited by: §1 , §2 .

[10] C. Cao, Y. Cai, Q. Dong, Y. Wang, and Y. Fu (2024)

Leftrefill: filling right canvas based on left reference through generalized text-to-image diffusion model .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

pp. 7705–7715 .

Cited by: §3.3 .

[11] C. Cao, X. Ren, and Y. Fu (2024)

MVSFormer++: revealing the devil in transformer’s details for multi-view stereo .

In International Conference on Learning Representations ,

Cited by: §2 , §4.3 .

[12] C. Cao, C. Yu, S. Liu, F. Wang, X. Xue, and Y. Fu (2025)

MVGenMaster: scaling multi-view generation from any image via 3d priors enhanced diffusion model .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 6045–6056 .

Cited by: §2 .

[13] C. Cao, J. Zhou, S. Li, J. Liang, C. Yu, F. Wang, X. Xue, and Y. Fu (2025)

Uni3C: unifying precisely 3d-enhanced camera and human motion controls for video generation .

Cited by: Appendix B , §1 , §1 , §2 , §3.1 , §3.2 , §4.1 , Table 2 , Table 3 , Table 3 .

[14] B. Chen, D. Martí Monsó, Y. Du, M. Simchowitz, R. Tedrake, and V. Sitzmann (2024)

Diffusion forcing: next-token prediction meets full-sequence diffusion .

Advances in Neural Information Processing Systems 37 , pp. 24081–24125 .

Cited by: §1 .

[15] C. Chen and J. Mo (2022)

IQA-PyTorch: pytorch toolbox for image quality assessment .

Note: [Online]. Available: https://github.com/chaofengc/IQA-PyTorch

Cited by: §4.2 .

[16] G. Chen, D. Lin, J. Yang, C. Lin, J. Zhu, M. Fan, H. Zhang, S. Chen, Z. Chen, C. Ma, et al. (2025)

Skyreels-v2: infinite-length film generative model .

arXiv preprint arXiv:2504.13074 .

Cited by: §2 .

[17] J. Chen, H. Zhu, X. He, Y. Wang, J. Zhou, W. Chang, Y. Zhou, Z. Li, Z. Fu, J. Pang, et al. (2025)

DeepVerse: 4d autoregressive video generation as a world model .

arXiv preprint arXiv:2506.01103 .

Cited by: §2 .

[18] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T. Cham, and J. Cai (2024)

Mvsplat: efficient 3d gaussian splatting from sparse multi-view images .

In European Conference on Computer Vision ,

pp. 370–386 .

Cited by: §2 .

[19] Z. Chen, M. Qin, T. Yuan, Z. Liu, and H. Zhao (2025)

Long3r: long sequence streaming 3d reconstruction .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 5273–5284 .

Cited by: §2 .

[20] E. Decart, Q. McIntyre, S. Campbell, X. Chen, and R. Wachen (2024)

Oasis: a universe in a transformer .

URL: https://oasis-model. github. io .

Cited by: §2 .

[21] H. Duan, H. Yu, S. Chen, L. Fei-Fei, and J. Wu (2025)

Worldscore: a unified evaluation benchmark for world generation .

arXiv preprint arXiv:2504.00983 .

Cited by: §4.2 , Table 2 , Table 2 .

[22] W. Feng, J. Liu, P. Tu, T. Qi, M. Sun, T. Ma, S. Zhao, S. Zhou, and Q. He (2025)

I2VControl-camera: precise video camera control with adjustable motion strength .

In International Conference on Learning Representations ,

Cited by: §1 , §2 .

[23] R. Fridman, A. Abecasis, Y. Kasten, and T. Dekel (2023)

Scenescape: text-driven consistent scene generation .

Advances in Neural Information Processing Systems 36 , pp. 39897–39914 .

Cited by: §2 .

[24] S. Galliani, K. Lasinger, and K. Schindler (2015)

Massively parallel multiview stereopsis by surface normal diffusion .

In Proceedings of the IEEE international conference on computer vision ,

pp. 873–881 .

Cited by: §2 .

[25] R. Gao, A. Holynski, P. Henzler, A. Brussee, R. M. Brualla, P. P. Srinivasan, J. T. Barron, and B. Poole (2024)

CAT3d: create anything in 3d with multi-view diffusion models .

In The Thirty-eighth Annual Conference on Neural Information Processing Systems ,

External Links: Link

Cited by: §2 .

[26] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan (2020)

Cascade cost volume for high-resolution multi-view stereo and stereo matching .

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

pp. 2495–2504 .

Cited by: §2 .

[27] Y. Gu, W. Mao, and M. Z. Shou (2025)

Long-context autoregressive video modeling with next-frame prediction .

arXiv preprint arXiv:2503.19325 .

Cited by: §2 .

[28] Z. Gu, R. Yan, J. Lu, P. Li, Z. Dou, C. Si, Z. Dong, Q. Liu, C. Lin, Z. Liu, et al. (2025)

Diffusion as shader: 3d-aware video diffusion for versatile video generation control .

In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers ,

pp. 1–12 .

Cited by: §1 , §2 , §2 .

[29] J. Guo, Y. Ye, T. He, H. Wu, Y. Jiang, T. Pearce, and J. Bian (2025)

Mineworld: a real-time and open-source interactive world model on minecraft .

arXiv preprint arXiv:2504.08388 .

Cited by: §2 .

[30] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai (2024)

AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning .

International Conference on Learning Representations .

Cited by: §2 .

[31] H. He, Y. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang (2025)

Cameractrl: enabling camera control for text-to-video generation .

In International Conference on Learning Representations ,

Cited by: §1 , §2 .

[32] H. He, C. Yang, S. Lin, Y. Xu, M. Wei, L. Gui, Q. Zhao, G. Wetzstein, L. Jiang, and H. Li (2025)

CameraCtrl ii: dynamic scene exploration via camera-controlled video diffusion models .

arXiv preprint arXiv:2503.10592 .

Cited by: §1 , §2 .

[33] X. He, C. Peng, Z. Liu, B. Wang, Y. Zhang, Q. Cui, F. Kang, B. Jiang, M. An, Y. Ren, et al. (2025)

Matrix-game 2.0: an open-source, real-time, and streaming interactive world model .

arXiv preprint arXiv:2508.13009 .

Cited by: §2 .

[34] L. Höllein, A. Cao, A. Owens, J. Johnson, and M. Nießner (2023)

Text2room: extracting textured 3d meshes from 2d text-to-image models .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 7909–7920 .

Cited by: §2 .

[35] C. Hou, G. Wei, Y. Zeng, and Z. Chen (2025)

Training-free camera control for video generation .

In International Conference on Learning Representations ,

Cited by: §2 .

[36] T. Hu, H. Peng, X. Liu, and Y. Ma (2025)

EX-4d: extreme viewpoint 4d video synthesis via depth watertight mesh .

arXiv preprint arXiv:2506.05554 .

Cited by: §2 .

[37] T. Hu, J. Zhang, R. Yi, Y. Wang, H. Huang, J. Weng, Y. Wang, and L. Ma (2024)

MotionMaster: training-free camera motion transfer for video generation .

arXiv preprint arXiv:2404.15789 .

Cited by: §2 .

[38] J. Huang, Q. Zhou, H. Rabeti, A. Korovko, H. Ling, X. Ren, T. Shen, J. Gao, D. Slepichev, C. Lin, et al. (2025)

Vipe: video pose engine for 3d geometric perception .

arXiv preprint arXiv:2508.10934 .

Cited by: §2 .

[39] T. Huang, W. Zheng, T. Wang, Y. Liu, Z. Wang, J. Wu, J. Jiang, H. Li, R. W. Lau, W. Zuo, et al. (2025)

Voyager: long-range and world-consistent video diffusion for explorable 3d scene generation .

arXiv preprint arXiv:2506.04225 .

Cited by: §2 , Table 2 .

[40] X. Huang, Z. Li, G. He, M. Zhou, and E. Shechtman (2025)

Self forcing: bridging the train-test gap in autoregressive video diffusion .

arXiv preprint arXiv:2506.08009 .

Cited by: §1 , §3.4 .

[41] T. HunyuanWorld (2025)

HunyuanWorld 1.0: generating immersive, explorable, and interactive 3d worlds from words or pixels .

arXiv preprint .

Cited by: §2 .

[42] W. Jang, P. Weinzaepfel, V. Leroy, L. Agapito, and J. Revaud (2025)

Pow3r: empowering unconstrained 3d reconstruction with camera and scene priors .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 1071–1081 .

Cited by: §2 .

[43] W. Jin, Q. Dai, C. Luo, S. Baek, and S. Cho (2025)

Flovd: optical flow meets video diffusion model for enhanced camera-controlled video synthesis .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 2040–2049 .

Cited by: §1 , §2 .

[44] N. Keetha, N. Müller, J. Schönberger, L. Porzi, Y. Zhang, T. Fischer, A. Knapitsch, D. Zauss, E. Weber, N. Antunes, et al. (2025)

MapAnything: universal feed-forward metric 3d reconstruction .

arXiv preprint arXiv:2509.13414 .

Cited by: §1 , §2 .

[45] A. Knapitsch, J. Park, Q. Zhou, and V. Koltun (2017)

Tanks and temples: benchmarking large-scale scene reconstruction .

ACM Transactions on Graphics (ToG) 36 ( 4 ), pp. 1–13 .

Cited by: §1 , §4.3 , Table 3 , Table 3 .

[46] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. (2024)

Hunyuanvideo: a systematic framework for large video generative models .

arXiv preprint arXiv:2412.03603 .

Cited by: §1 , §1 , §2 .

[47] Kuaishou (2024)

Kling .

https://klingai.kuaishou.com .

Cited by: §1 .

[48] J. Li, J. Tang, Z. Xu, L. Wu, Y. Zhou, S. Shao, T. Yu, Z. Cao, and Q. Lu (2025)

Hunyuan-gamecraft: high-dynamic interactive game video generation with hybrid history condition .

arXiv preprint arXiv:2506.17201 .

Cited by: §2 .

[49] R. Li, P. Torr, A. Vedaldi, and T. Jakab (2025)

VMem: consistent interactive video scene generation with surfel-indexed view memory .

arXiv preprint arXiv:2506.18903 .

Cited by: §2 , §3.3 , Table 3 , Table 3 .

[50] T. Li, G. Zheng, R. Jiang, T. Wu, Y. Lu, Y. Lin, X. Li, et al. (2025)

RealCam-i2v: real-world image-to-video generation with interactive complex camera control .

arXiv preprint arXiv:2502.10059 .

Cited by: §1 , §2 .

[51] X. Li, T. Wang, Z. Gu, S. Zhang, C. Guo, and L. Cao (2025)

FlashWorld: high-quality 3d scene generation within seconds .

arXiv preprint arXiv:2510.13678 .

Cited by: §2 .

[52] Z. Li, R. Tucker, F. Cole, Q. Wang, L. Jin, V. Ye, A. Kanazawa, A. Holynski, and N. Snavely (2025)

MegaSaM: accurate, fast and robust structure and motion from casual dynamic videos .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 10486–10496 .

Cited by: §2 .

[53] H. Liang, J. Cao, V. Goel, G. Qian, S. Korolev, D. Terzopoulos, K. N. Plataniotis, S. Tulyakov, and J. Ren (2024)

Wonderland: navigating 3d scenes from a single image .

arXiv preprint arXiv:2412.12091 .

Cited by: §1 , §2 .

[54] Y. Liang, X. Yang, J. Lin, H. Li, X. Xu, and Y. Chen (2024)

Luciddreamer: towards high-fidelity text-to-3d generation via interval score matching .

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

pp. 6517–6526 .

Cited by: §2 .

[55] B. Lin, Y. Ge, X. Cheng, Z. Li, B. Zhu, S. Wang, X. He, Y. Ye, S. Yuan, L. Chen, et al. (2024)

Open-sora plan: open-source large video generation model .

arXiv preprint arXiv:2412.00131 .

Cited by: §2 .

[56] L. Ling, Y. Sheng, Z. Tu, W. Zhao, C. Xin, K. Wan, L. Yu, Q. Guo, Z. Yu, Y. Lu, et al. (2024)

Dl3dv-10k: a large-scale scene dataset for deep learning-based 3d vision .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

pp. 22160–22169 .

Cited by: Table 4 , Table 4 , Appendix B , §3.3 , §4.1 .

[57] X. Liu, P. Tayal, J. Wang, J. Zarzar, T. Monnier, K. Tertikas, J. Duan, A. Toisoul, J. Y. Zhang, N. Neverova, et al. (2025)

Uncommon objects in 3d .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 14102–14113 .

Cited by: Table 4 , Table 4 , Appendix B .

[58] Y. Liu, Z. Min, Z. Wang, J. Wu, T. Wang, Y. Yuan, Y. Luo, and C. Guo (2025)

WorldMirror: universal 3d world reconstruction with any-prior prompting .

arXiv preprint arXiv:2510.10726 .

Cited by: Figure 1 , Figure 1 , §2 , §3.1 , §4.2 , §4.3 .

[59] J. Lu, T. Huang, P. Li, Z. Dou, C. Lin, Z. Cui, Z. Dong, S. Yeung, W. Wang, and Y. Liu (2025)

Align3r: aligned monocular depth estimation for dynamic videos .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 22820–22830 .

Cited by: §2 .

[60] B. Ma, H. Gao, H. Deng, Z. Luo, T. Huang, L. Tang, and X. Wang (2025)

You see it, you got it: learning 3d creation on pose-free videos at scale .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 2016–2029 .

Cited by: §1 , §2 , §2 .

[61] D. Marr and T. Poggio (1976)

Cooperative computation of stereo disparity: a cooperative algorithm is derived for extracting disparity information from stereo image pairs. .

Science 194 ( 4262 ), pp. 283–287 .

Cited by: §1 , §3.3 .

[62] L. Pan, D. Barath, M. Pollefeys, and J. L. Schönberger (2024)

Global Structure-from-Motion Revisited .

In European Conference on Computer Vision (ECCV) ,

Cited by: §2 .

[63] W. Peebles and S. Xie (2023)

Scalable diffusion models with transformers .

In Proceedings of the IEEE/CVF international conference on computer vision ,

pp. 4195–4205 .

Cited by: §1 .

[64] S. Popov, A. Raj, M. Krainin, Y. Li, W. T. Freeman, and M. Rubinstein (2025)

CamCtrl3D: single-image scene exploration with precise 3d camera control .

arXiv preprint arXiv:2501.06006 .

Cited by: §1 , §2 .

[65] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021)

Learning transferable visual models from natural language supervision .

In International conference on machine learning ,

pp. 8748–8763 .

Cited by: §4.2 .

[66] X. Ren, T. Shen, J. Huang, H. Ling, Y. Lu, M. Nimier-David, T. Müller, A. Keller, S. Fidler, and J. Gao (2025)

Gen3c: 3d-informed world-consistent video generation with precise camera control .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

Cited by: §1 , §2 , §2 , §4.4 , Table 2 , Table 3 , Table 3 .

[67] RunwayML (2024)

Gen-3 alpha .

https://runwayml.com/research/ introducing-gen-3-alpha .

Cited by: §1 .

[68] M. Schneider, L. Höllein, and M. Nießner (2025)

WorldExplorer: towards generating fully navigable 3d scenes .

arXiv preprint arXiv:2506.01799 .

Cited by: §2 , §3.3 .

[69] J. L. Schönberger, E. Zheng, M. Pollefeys, and J. Frahm (2016)

Pixelwise view selection for unstructured multi-view stereo .

In European conference on computer vision ,

Cited by: §2 .

[70] C. Schuhmann (2022)

Improved-aesthetic-predictor: clip+mlp aesthetic score predictor .

Note: https://github.com/christophschuhmann/improved-aesthetic-predictor

Cited by: §4.2 .

[71] J. Shriram, A. Trevithick, L. Liu, and R. Ramamoorthi (2024)

Realmdreamer: text-driven 3d scene generation with inpainting and depth diffusion .

arXiv preprint arXiv:2404.07199 .

Cited by: §2 .

[72] C. Song, M. Stary, B. Chen, G. Kopanas, and V. Sitzmann (2025)

Generative view stitching .

arXiv preprint arXiv:2510.24718 .

Cited by: §1 .

[73] K. Song, B. Chen, M. Simchowitz, Y. Du, R. Tedrake, and V. Sitzmann (2025)

History-guided video diffusion .

arXiv preprint arXiv:2502.06764 .

Cited by: §2 .

[74] W. Sun, S. Chen, F. Liu, Z. Chen, Y. Duan, J. Zhang, and Y. Wang (2025)

DimensionX: create any 3d and 4d scenes from a single image with controllable video diffusion .

In International Conference on Computer Vision (ICCV) ,

Cited by: §2 , §2 .

[75] S. Szymanowicz, J. Y. Zhang, P. Srinivasan, R. Gao, A. Brussee, A. Holynski, R. Martin-Brualla, J. T. Barron, and P. Henzler (2025)

Bolt3d: generating 3d scenes in seconds .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 24846–24857 .

Cited by: §2 .

[76] S. Umeyama (1991)

Least-squares estimation of transformation parameters between two point patterns .

IEEE Transactions on Pattern Analysis & Machine Intelligence 13 ( 04 ), pp. 376–380 .

Cited by: §3.1 , §3.2 , §4.3 .

[77] D. Valevski, Y. Leviathan, M. Arar, and S. Fruchter (2024)

Diffusion models are real-time game engines .

arXiv preprint arXiv:2408.14837 .

Cited by: §2 .

[78] A. Wang, B. Ai, B. Wen, C. Mao, C. Xie, D. Chen, F. Yu, H. Zhao, J. Yang, J. Zeng, et al. (2025)

Wan: open and advanced large-scale video generative models .

arXiv preprint arXiv:2503.20314 .

Cited by: Appendix D , Table 1 , Table 1 , §1 , §1 , §2 , §3.1 , §4.1 .

[79] A. Wang, H. Huang, J. Z. Fang, Y. Yang, and C. Ma (2025)

ATI: any trajectory instruction for controllable video generation .

arXiv preprint arXiv:2505.22944 .

Cited by: §1 , §2 .

[80] H. Wang, Y. Liu, Z. Liu, W. Wang, Z. Dong, and B. Yang (2025)

Vistadream: sampling multiview consistent images for single-view scene reconstruction .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 26772–26782 .

Cited by: §2 .

[81] J. Wang, K. C. Chan, and C. C. Loy (2023)

Exploring clip for assessing the look and feel of images .

In Proceedings of the AAAI conference on artificial intelligence ,

Vol. 37 , pp. 2555–2563 .

Cited by: §4.2 .

[82] J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny (2025)

VGGT: visual geometry grounded transformer .

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

Cited by: §1 .

[83] Q. Wang, Y. Zhang, A. Holynski, A. A. Efros, and A. Kanazawa (2025)

Continuous 3d perception model with persistent state .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 10510–10522 .

Cited by: §1 , §2 .

[84] R. Wang, S. Xu, Y. Dong, Y. Deng, J. Xiang, Z. Lv, G. Sun, X. Tong, and J. Yang (2025)

MoGe-2: accurate monocular geometry with metric scale and sharp details .

arXiv preprint arXiv:2507.02546 .

Cited by: Figure 1 , Figure 1 , §3.1 , §3.2 , §4.4 .

[85] S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud (2024)

Dust3r: geometric 3d vision made easy .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

pp. 20697–20709 .

Cited by: §2 .

[86] W. Wang, D. Zhu, X. Wang, Y. Hu, Y. Qiu, C. Wang, Y. Hu, A. Kapoor, and S. Scherer (2020)

TartanAir: a dataset to push the limits of visual slam .

Cited by: Table 4 , Table 4 , Appendix B , §3.3 , §4.1 .

[87] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu (2023)

Prolificdreamer: high-fidelity and diverse text-to-3d generation with variational score distillation .

Advances in neural information processing systems 36 , pp. 8406–8441 .

Cited by: §3.4 .

[88] Z. Wang, Z. Yuan, X. Wang, Y. Li, T. Chen, M. Xia, P. Luo, and Y. Shan (2024)

Motionctrl: a unified and flexible motion controller for video generation .

In ACM SIGGRAPH 2024 Conference Papers ,

pp. 1–11 .

Cited by: §2 .

[89] P. Weinzaepfel, T. Lucas, V. Leroy, Y. Cabon, V. Arora, R. Brégier, G. Csurka, L. Antsfeld, B. Chidlovskii, and J. Revaud (2023)

Croco v2: improved cross-view completion pre-training for stereo matching and optical flow .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 17969–17980 .

Cited by: §2 .

[90] H. Wu, Z. Zhang, W. Zhang, C. Chen, L. Liao, C. Li, Y. Gao, A. Wang, E. Zhang, W. Sun, et al. (2023)

Q-align: teaching lmms for visual scoring via discrete text-defined levels .

arXiv preprint arXiv:2312.17090 .

Cited by: §4.2 .

[91] T. Wu, S. Yang, R. Po, Y. Xu, Z. Liu, D. Lin, and G. Wetzstein (2025)

Video world models with long-term spatial memory .

arXiv preprint arXiv:2506.05284 .

Cited by: §2 .

[92] H. Xia, Y. Fu, S. Liu, and X. Wang (2024)

RGBD objects in the wild: scaling real-world 3d object learning from rgb-d videos .

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ,

pp. 22378–22389 .

Cited by: Table 4 , Table 4 , Appendix B , §3.3 , §4.1 .

[93] Y. Xia, S. Weng, S. Yang, J. Liu, C. Zhu, M. Teng, Z. Jia, H. Jiang, and B. Shi (2025)

PanoWan: lifting diffusion video generation models to 360 { { \ \backslash deg } } with latitude/longitude-aware mechanisms .

Advances in Neural Information Processing Systems .

Cited by: §4.4 .

[94] Z. Xiao, Y. Lan, Y. Zhou, W. Ouyang, S. Yang, Y. Zeng, and X. Pan (2025)

Worldmem: long-term consistent world simulation with memory .

arXiv preprint arXiv:2504.12369 .

Cited by: §2 .

[95] J. Xing, M. Xia, Y. Zhang, H. Chen, W. Yu, H. Liu, G. Liu, X. Wang, Y. Shan, and T. Wong (2024)

Dynamicrafter: animating open-domain images with video diffusion priors .

In European Conference on Computer Vision ,

pp. 399–417 .

Cited by: §2 .

[96] J. Yang, A. Sax, K. J. Liang, M. Henaff, H. Tang, A. Cao, J. Chai, F. Meier, and M. Feiszli (2025)

Fast3r: towards 3d reconstruction of 1000+ images in one forward pass .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 21924–21935 .

Cited by: §1 , §2 .

[97] S. Yang, L. Hou, H. Huang, C. Ma, P. Wan, D. Zhang, X. Chen, and J. Liao (2024)

Direct-a-video: customized video generation with user-directed camera movement and object motion .

In ACM SIGGRAPH 2024 Conference Papers ,

pp. 1–12 .

Cited by: §2 , §2 .

[98] Z. Yang, W. Ge, Y. Li, J. Chen, H. Li, M. An, F. Kang, H. Xue, B. Xu, Y. Yin, et al. (2025)

Matrix-3d: omnidirectional explorable 3d world generation .

arXiv preprint arXiv:2508.08086 .

Cited by: §2 , §4.4 .

[99] Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, et al. (2025)

Cogvideox: text-to-video diffusion models with an expert transformer .

In International Conference on Learning Representations ,

Cited by: §1 , §1 , §2 .

[100] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan (2018)

Mvsnet: depth inference for unstructured multi-view stereo .

In Proceedings of the European conference on computer vision (ECCV) ,

pp. 767–783 .

Cited by: §2 .

[101] T. Yin, M. Gharbi, T. Park, R. Zhang, E. Shechtman, F. Durand, and B. Freeman (2024)

Improved distribution matching distillation for fast image synthesis .

Advances in neural information processing systems 37 , pp. 47455–47487 .

Cited by: §1 , §3.4 , §3.4 .

[102] T. Yin, M. Gharbi, R. Zhang, E. Shechtman, F. Durand, W. T. Freeman, and T. Park (2024)

One-step diffusion with distribution matching distillation .

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition ,

pp. 6613–6623 .

Cited by: §1 .

[103] Y. Yin, H. Guo, F. Liu, M. Wang, H. Liang, E. Li, Y. Wang, X. Jin, Y. Zhao, and Y. Wei (2025)

PanoWorld-x: generating explorable panoramic worlds via sphere-aware video diffusion .

arXiv preprint arXiv:2509.24997 .

Cited by: §4.4 .

[104] H. Yu, H. Duan, C. Herrmann, W. T. Freeman, and J. Wu (2025)

Wonderworld: interactive 3d scene generation from a single image .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 5916–5926 .

Cited by: §2 .

[105] J. Yu, J. Bai, Y. Qin, Q. Liu, X. Wang, P. Wan, D. Zhang, and X. Liu (2025)

Context as memory: scene-consistent interactive long video generation with memory retrieval .

arXiv preprint arXiv:2506.03141 .

Cited by: §2 , §3.3 , §3.3 .

[106] W. Yu, J. Xing, L. Yuan, W. Hu, X. Li, Z. Huang, X. Gao, T. Wong, Y. Shan, and Y. Tian (2024)

Viewcrafter: taming video diffusion models for high-fidelity novel view synthesis .

arXiv preprint arXiv:2409.02048 .

Cited by: §1 , §2 , §2 , §2 .

[107] D. J. Zhang, R. Paiss, S. Zada, N. Karnad, D. E. Jacobs, Y. Pritch, I. Mosseri, M. Z. Shou, N. Wadhwa, and N. Ruiz (2025)

Recapture: generative video camera controls for user-provided videos using masked video fine-tuning .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 2050–2062 .

Cited by: §2 .

[108] J. Zhang, S. Li, Z. Luo, T. Fang, and Y. Yao (2023)

Vis-mvsnet: visibility-aware multi-view stereo network .

International Journal of Computer Vision 131 ( 1 ), pp. 199–214 .

Cited by: §2 .

[109] J. Zhang, C. Herrmann, J. Hur, V. Jampani, T. Darrell, F. Cole, D. Sun, and M. Yang (2024)

Monst3r: a simple approach for estimating geometry in the presence of motion .

arXiv preprint arXiv:2410.03825 .

Cited by: §2 .

[110] L. Zhang and M. Agrawala (2025)

Packing input frame context in next-frame prediction models for video generation .

arXiv preprint arXiv:2504.12626 .

Cited by: §2 .

[111] L. Zhang, A. Rao, and M. Agrawala (2023)

Adding conditional control to text-to-image diffusion models .

In Proceedings of the IEEE/CVF international conference on computer vision ,

pp. 3836–3847 .

Cited by: §3.1 .

[112] Q. Zhang, S. Zhai, M. A. B. Martin, K. Miao, A. Toshev, J. Susskind, and J. Gu (2025)

World-consistent video diffusion with explicit 3d modeling .

In Proceedings of the Computer Vision and Pattern Recognition Conference ,

pp. 21685–21695 .

Cited by: §2 .

[113] Y. Zhang, C. Peng, B. Wang, P. Wang, Q. Zhu, F. Kang, B. Jiang, Z. Gao, E. Li, Y. Liu, et al. (2025)

Matrix-game: interactive world foundation model .

arXiv preprint arXiv:2506.18701 .

Cited by: §2 .

[114] Y. Zhang, C. Cao, C. Yu, and J. Zhu (2025)

LiON-lora: rethinking lora fusion to unify controllable spatial and temporal generation for video diffusion .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 14569–14579 .

Cited by: §2 .

[115] Y. Zhao, C. Lin, K. Lin, Z. Yan, L. Li, Z. Yang, J. Wang, G. H. Lee, and L. Wang (2024)

Genxd: generating any 3d and 4d scenes .

arXiv preprint arXiv:2411.02319 .

Cited by: §2 .

[116] G. Zheng, T. Li, R. Jiang, Y. Lu, T. Wu, and X. Li (2025)

Cami2v: camera-controlled image-to-video diffusion model .

In International Conference on Learning Representations ,

Cited by: §1 .

[117] Z. Zheng, X. Peng, T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You (2024)

Open-sora: democratizing efficient video production for all .

arXiv preprint arXiv:2412.20404 .

Cited by: §2 .

[118] J. J. Zhou, H. Gao, V. Voleti, A. Vasishta, C. Yao, M. Boss, P. Torr, C. Rupprecht, and V. Jampani (2025)

STABLE virtual camera: generative view synthesis with diffusion models .

arXiv e-prints , pp. arXiv–2503 .

Cited by: §2 , §2 , §3.3 , §4.4 , Table 2 , Table 3 , Table 3 .

[119] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely (2018-07)

Stereo magnification: learning view synthesis using multiplane images .

ACM Trans. Graph. 37 ( 4 ).

External Links: ISSN 0730-0301 , Link , Document

Cited by: Table 4 , Table 4 , Appendix B , §4.1 .

[120] H. Zhu, Y. Wang, J. Zhou, W. Chang, Y. Zhou, Z. Li, J. Chen, C. Shen, J. Pang, and T. He (2025)

Aether: geometric-aware unified world modeling .

In Proceedings of the IEEE/CVF International Conference on Computer Vision ,

pp. 8535–8546 .

Cited by: §2 .

Appendix A

Datasets

We summarize the datasets of this work in Table   4 , including both training and evaluation dataset settings. Note that most of our training data is publicly available. We randomly sample a subset for each data for each epoch, according to its diversity and video, trajectory quality.

Table 4 : Dataset details of WorldStereo. The training datasets used for various training processes include DL3DV  [ 56 ] , Re10K  [ 119 ] , Tartainair  [ 86 ] , Map-Free-Reloc  [ 1 ] , WildRGBD  [ 92 ] , and UCo3D  [ 57 ] . We dynamically sample subsets for each dataset across training epochs.

Train Processes
Scene Number

Camera
GGM
SSM
DMD
Total
Epoch

DL3DV
✓ \checkmark
✓ \checkmark
✓ \checkmark
✓ \checkmark
9,174
19,868

Re10K
✓ \checkmark
✓ \checkmark

26,544
12,000

Tartainair
✓ \checkmark
✓ \checkmark
✓ \checkmark
✓ \checkmark
2,214
2,142

Map-Free-Reloc
✓ \checkmark
✓ \checkmark
✓ \checkmark
✓ \checkmark
960
910

WildRGBD
✓ \checkmark
✓ \checkmark
✓ \checkmark
✓ \checkmark
23,005
3,000

UCo3D
✓ \checkmark

✓  \checkmark    

164,455
10,000

UE-Render
✓ \checkmark
✓ \checkmark
✓ \checkmark
✓ \checkmark
5,939
4,000

Appendix B

More Quantitative Ablation Studies

Benchmark for Memory Components.

To quantitatively assess the contribution of various modules for video generation and memory capabilities, we have established a benchmark consisting of 100 scenes drawn from our diverse test set in Table   5 , containing data from DL3DV  [ 56 ] , Map-Free-Reloc  [ 1 ] , WildRGBD  [ 92 ] , Tartanair  [ 86 ] , and UE5-rendered scenes. It features a wide array of challenging situations, including real and virtual environments, indoor and outdoor settings, and varying complexities of camera motion. The construction of the memory bank involves selecting videos that have a temporal overlap of 30% to 90%. To increase the diversity and complexity of test samples, a random dropping strategy is applied to the reference frames. Specifically, there is a 10% chance that all reference frames are dropped, yielding an empty memory bank, and a 10% chance that no frames are dropped. For the remaining scenarios, each reference frame associated with a target view is randomly dropped with a probability of 40%. Since this benchmark contains ground-truth video views to evaluate the image fidelity, we can include PSNR, SSIM, and LPIPS to verify the effectiveness of memory components.

Table 5 : Quantitative ablation for memory components. ∗ indicates our baseline without any memory mechanism, while the ‘full’ version denotes adding both GGM and SSM. The ‘DMD’ version is based on ‘WorldStereo-full’.

RotErr ↓ \downarrow

TransErr ↓ \downarrow

ATE ↓ \downarrow

PSNR ↑ \uparrow

SSIM ↑ \uparrow

LPIPS ↓ \downarrow

WorldStereo* 1.300 0.112 0.237 14.64 0.443 0.412

WorldStereo-GGM 0.699
0.067
0.131
17.45 0.532 0.288

WorldStereo-Full 0.748
0.079 0.142 18.40
0.561
0.283

WorldStereo-DMD 0.772 0.069
0.130
18.04
0.546
0.289

Camera Control Evaluation.

We build a test set (DL3DV, Real10k  [ 119 ] , Map-Free-Reloc, WildRGBD, Tartanair, and UCo3D  [ 57 ] ) to evaluate the capability of camera control. Different from the aforementioned memory ablation benchmark, all samples in this benchmark only contain depth and point clouds extracted from the first frame without additional information from other views. A comparative analysis was conducted among Uni3C  [ 13 ] , WorldStereo, and its DMD-accelerated variant, WorldStereo-DMD. We did not apply memory components in this part. The primary metrics for comparison are camera trajectory accuracy and resulting image quality. The results presented in Table   6 , indicate that our proposed method yields substantial gains in camera control precision compared to Uni3C, while also offering improvements to video quality. Notably, the application of DMD acceleration results in a 20x increase in inference speed for the WorldStereo-DMD model, with no significant drop in qualitative output and camera control.

Table 6 : Quantitative results for camera control.

RotErr ↓ \downarrow

TransErr ↓ \downarrow

ATE ↓ \downarrow

PSNR ↑ \uparrow

SSIM ↑ \uparrow

LPIPS ↓ \downarrow

Uni3C 1.816 0.122 0.359 14.28
0.475
0.449

WorldStereo 1.384
0.115
0.347
14.27
0.473
0.441

WorldStereo-DMD 1.435
0.085
0.211
13.78 0.436 0.463

High-Resolution Inference.

As presented in Section 4.1, our model is capable of performing inference on high-resolution 720 × 1280 720\times 1280 videos, despite only training exclusively on 480p data. This capability highlights the success of our method in retaining the generalization ability of the base model, which is inherently capable of processing 720p videos. Furthermore, as shown in Figure   7 , the model can generate images with enhanced detail by directly performing inference at high resolutions, obviating the need for retraining. Nevertheless, to balance performance with computational efficiency, the experiments in this paper are carried out using 480p resolution.

Figure 7 : Qualitative Results of different resolutions. High-resolution (720 × \times 1280) yields sharper images with richer details than the low-resolution ones (480 × \times 768), as demonstrated in the red-boxed regions.

Appendix C

Trajectory Settings

Table 7 : Overlapping FoV scores of 3D panorama generation (trajectory order ablation) . Memory bank settings: ‘only panorama’ uses 24 views split from the panorama; others are incrementally updated with generations from different trajectory orders. Higher scores mean that more relevant frames are retrieved. ‘reference prop.’ indicates the proportion of retrieved frames belonging to panoramic (pano.) or generated (gen.) frames.

orbit
up
right
left
all
reference prop.

pano.
gen.

only panorama
46.0
43.4
38.4
38.5
167.2
100.0%
0.00%

up → \rightarrow right → \rightarrow left → \rightarrow orbit

50.7
43.4
39.6
41.3
175.0
65.17%
34.83%

orbit → \rightarrow up → \rightarrow right → \rightarrow left

46.9
46.2
41.3
42.8
177.1
54.89%
45.11%

orbit → \rightarrow right → \rightarrow left → \rightarrow up

46.9
46.9
41.1
42.8
177.6
53.66%
46.34%

up → \rightarrow orbit → \rightarrow right → \rightarrow left

49.2
43.4
41.3
42.8
176.7
55.55%
44.45%

right → \rightarrow left → \rightarrow up → \rightarrow orbit

50.7
45.2
38.4
41.2
175.6
60.89%
39.11%

right → \rightarrow left → \rightarrow orbit → \rightarrow up

49.3
46.9
38.4
41.2
176.3
62.43%
37.57%

To determine an optimal trajectory sequence for high-quality 3D reconstruction, we performed a series of ablation studies across 10 panoramic scenes, with results summarized in Table   7 . The memory bank was initialized with 24 views extracted from the panorama, while others are incrementally updated with generations from different trajectory orders (see   Figure   8 ). From Table   7 , the updated memory bank exhibits more reliable references compared to the baseline setting that solely relies on panoramic images. Notably, the orbit trajectory—with its information-rich viewing angles, should be prioritized as the initial trajectory. We finally use ‘orbit → \rightarrow up → \rightarrow right → \rightarrow left’ as our default trajectory sequence. This ordering is justified by the fact that left and right rotations are more critical to 3D reconstruction than other trajectories. Positioning these rotations last allows them to leverage a larger number of pre-accumulated memory frames, thereby enhancing their contribution to the final reconstruction.

Figure 8 : Illustration of the trajectory.

Specifically, we set the rotation angles for the upward, leftward, and rightward rotations to be 45 ∘ , 90 ∘ , 90 ∘ , respectively. For these rotational movements, the distance from the rotation center is configured as the median depth of the scene, while the radius of the orbital trajectory is set to 0.3 times the median depth. For some face-forwarding scenes ( e.g . , ‘room’ in MipNeRF360  [ 4 ] ), we manually reduce the rotation angles to avoid the collision.

Appendix D

Details of Data Curation

Sampling Strategy for Training GGM.

A global point cloud is constructed from multi-view images and their associated depth maps. Our GGM training process begins with the point cloud of the initial frame, X p ​ c ​ d X_{pcd} . We then augment this by randomly sampling 1 to 4 additional frames, generating their respective point clouds ( X ^ p ​ c ​ d \hat{X}{pcd} ) from depth information, and aligning them to the coordinate system of X p ​ c ​ d X{pcd} . To mitigate overfitting, we apply two data augmentation techniques to X ^ p ​ c ​ d \hat{X}_{pcd} . The first one is random masking, which nullifies 30% to 70% of randomly selected pixels in the depth map. The second one is contiguous masking, which applies a randomly positioned rectangular mask that occludes 20% to 70% of the depth map’s area. These augmentations allow the GGM module to effectively learn global geometric features without being misled by spurious or incorrect global point cloud conditions.

Sampling Strategy for Training SSM.

Following the procedure for SSM training outlined in Section 3.3, we construct training pairs from our multi-view dataset, ensuring each reference-target pair has a temporal overlap of 30% to 90% and the same number of frames. To enhance model robustness, we employ a reference dropout strategy. Specifically, for any given training sample, the entire reference condition is omitted with a 10% probability. Otherwise, for each target frame, its corresponding reference frame is randomly dropped with a 30% probability, thereby nullifying the reference condition for that frame. This process generates an unordered reference set for SSM training, which is closer to real-world application scenarios. Consequently, instead of encoding them as a coherent video, we process each remaining reference frame independently, encoding it into a latent representation using a pre-trained VAE  [ 78 ] .

BETA