View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume

Authors

Kai-En Lin, University of California, San Diego
Guowei Yang
Lei Xiao, Reality Labs Research, Meta
Feng Liu, Portland State UniversityFollow
Ravi Ramamoorthi, University of California, San Diego

Published In

ArXiv preprint

Document Type

Pre-Print

Publication Date

2021

Subjects

Computer vision, Image processing -- Methodology, Neural networks (Computer science), Machine learning

Abstract

Image view synthesis has seen great success in reconstructing photorealistic visuals, thanks to deep learning and various novel representations. The next key step in immersive virtual experiences is view synthesis of dynamic scenes. However, several challenges exist due to the lack of high-quality training datasets, and the additional time dimension for videos of dynamic scenes. To address this issue, we introduce a multi-view video dataset, captured with a custom 10-camera rig in 120FPS. The dataset contains 96 high-quality scenes showing various visual effects and human interactions in outdoor scenes. We develop a new algorithm, Deep 3D Mask Volume, which enables temporally-stable view extrapolation from binocular videos of dynamic scenes, captured by static cameras. Our algorithm addresses the temporal inconsistency of disocclusions by identifying the error-prone areas with a 3D mask volume, and replaces them with static background observed throughout the video. Our method enables manipulation in 3D space as opposed to simple 2D masks, We demonstrate better temporal stability than frame-by-frame static view synthesis methods, or those that use 2D masks. The resulting view synthesis videos show minimal flickering artifacts and allow for larger translational movements.

Rights

© Copyright the author(s)

Description

This article has been accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/TPAMI.2023.3289333

Locate the Document

10.1109/TPAMI.2023.3289333

DOI

10.1109/TPAMI.2023.3289333

Persistent Identifier

https://archives.pdx.edu/ds/psu/40458

Citation Details

Lin, K. E., Yang, G., Xiao, L., Liu, F., & Ramamoorthi, R. (2021). View Synthesis of Dynamic Scenes based on Deep 3D Mask Volume. arXiv preprint arXiv:2108.13408.