Knowledge Vault 5 /89 - CVPR 2023
DynIBaR: Neural Dynamic Image-Based Rendering
Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, Noah Snavely
< Resume Image >

Concept Graph & Resume using Claude 3 Opus | Chat GPT4o | Llama 3:

graph LR classDef main fill:#f9d4d4,stroke:#333,stroke-width:2px,font-weight:bold,font-size:14px classDef dynabar fill:#d4f9d4,stroke:#333,stroke-width:2px,font-weight:bold,font-size:14px classDef prior fill:#d4d4f9,stroke:#333,stroke-width:2px,font-weight:bold,font-size:14px classDef render fill:#f9f9d4,stroke:#333,stroke-width:2px,font-weight:bold,font-size:14px classDef optimize fill:#f9d4f9,stroke:#333,stroke-width:2px,font-weight:bold,font-size:14px A[DynIBaR: Neural Dynamic
Image-Based Rendering] --> B[DynaBar: View synthesis
from moving camera 1] A --> C[Prior methods struggle
with complex videos 2] B --> D[DynaBar insight: Blend
pixels at render 3] B --> E[Builds on IBRNet
for static scenes 4] B --> F[DynaBar optimizes MLP
for motion trajectories 5] F --> G[Renders by advecting samples
and projecting 6] F --> H[Stores motion, images
store appearance data 7] B --> I[Optimized per-video using
reconstruction loss 8] I --> J[Cross-time rendering improves
generalization 9] B --> K[Decomposes static and
dynamic components 10] B --> L[Enables effects like
Hitchcock zoom, bokeh 11] B --> M[Outperforms recent dynamic
NeRF methods 12] B --> N[Limitations: Less movement,
per-video optimization 13] B --> O[Key insight: Global motion
more efficient 14] A --> P[Promising approach for
scaling view synthesis 15] class A main class B,D,E,G,H,K,L,M,N,O dynabar class C prior class F,I,J optimize class P render

Resume:

1.- DynaBar: Dynamic view synthesis from single moving camera video, rendering new views in space and time.

2.- Prior methods (DVS, NSFF) struggle with long, complex videos due to reliance on high-capacity MLPs.

3.- DynaBar insight: synthesize target image by stealing and blending pixels from nearby source frames at render time.

4.- Builds on IBRNet, a recent image-based rendering (IBR) method for static scenes using epipolar constraints.

5.- DynaBar accounts for scene motion by optimizing an MLP to describe 3D motion trajectories.

6.- Renders a ray at time t by advecting samples according to learned motion and projecting to other views.

7.- Stores motion instead of full 4D scene, letting source images store high-res appearance data for sharp outputs.

8.- Optimized per-video by rendering rays, comparing to ground truth, and adjusting motion based on reconstruction loss.

9.- Cross-time rendering: optimizing to render frames from different times improves generalization when freezing time and moving camera.

10.- Decomposes scenes into static and dynamic components, recompositing them to render new views.

11.- Enables effects like Hitchcock zoom, bullet time, video stabilization, synthetic aperture (bokeh), and adjustable focus.

12.- Outperforms recent dynamic NeRF methods (HyperNeRF, NSFF) in photorealism.

13.- Limitations: less camera movement than static scenes, requires camera poses, per-video optimization (potential for offline generalized training).

14.- Key insight: optimizing global motion model is more efficient than full scene geometry and appearance.

15.- Promising approach for scaling view synthesis to arbitrary videos, with room for further research and improvement.

Knowledge Vault built byDavid Vivancos 2024