Image Stitching: Handling Parallax, Stereopsis, and Video
Abstract
Panorama stitching increases the field of view in an image by assembling multiple views together. Traditional stitching techniques are proven to be effective only when dealing with parallax-free monocular images. Many challenges that remain unsolved in the stitching research area include how to stitch monocular images with large parallax, how to stitch stereoscopic images to maintain their stereoscopic consistency and original disparity distribution, and how to create panoramic videos with temporally coherent content. To provide more powerful stitching techniques with more universality, we first develop a parallax-tolerant image stitching technique. With the help of it, we then effectively extend the stitching techniques into the stereoscopic image and the video domain to assist users easily making stereoscopic panoramas and video panoramas.
In this dissertation, we first introduce a parallax-tolerant stitching method, which is a local stitching method to stitch monocular images with large parallax. This method is based on the observation that input images do not need to be perfectly aligned over the whole overlapping region for stitching. Instead, they only need to be aligned in a way that there exists a local region where they can be seamlessly blended together. We develop a randomized algorithm to search for a local homography, which, combined with content-preserving warping, allows for optimal stitching. Our experiments show that our method can effectively stitch images with large parallax that are difficult for existing methods.
After studying the problem of regular 2D image stitching, we continue to research 3D image stitching in this dissertation. In particular, we develop a technique for stitching stereoscopic panoramas from stereo images casually taken using a stereo camera. Stereoscopic image stitching needs to address three challenges: how to deal with parallax, how to stitch the left- and right-view panorama consistently, and how to take care of disparity during stitching. We address these challenges by first stitching the left images with the parallax-tolerant image stitching method to create an artifact-free left view panorama, then stitching the disparity maps with disparity optimization, finally warping and stitching the right images according to the stitched disparity map and the left view panorama. Experiment results show that our technique allows for easy production of high quality stereoscopic panoramas that deliver a pleasant stereoscopic 3D viewing experience.
With the 3D image stitching problem addressed, we further study a more complex and challenging task of video stitching. We contribute two video stitching techniques, namely the motion map guided video stitching and the feature trajectory guided video stitching. Our techniques stitch pre-synchronized videos captured from a fixed or hand-held camera array which contains multiple cameras with fixed inter-camera configurations. One unique challenge for video stitching is how to maintain temporal coherence. To address this problem, we propose to consistently stitch frames with the guidance of the target camera motion path. In particular, we develop two techniques using dense motion maps and sparse motion vectors to compute the target camera motion path. Afterwards, we warp and stitch frames according to the target camera motion path to create panoramic videos with temporal coherence. Experiments show that our methods can improve the overall panoramic video stitching quality compared with existing methods.