Robust and Accurate Structure from Motion of Rigid and Nonrigid Objects

Abstract

As a central theme in computer vision, the problem of 3D structure and motion recovery from image sequences has been widely studied during the past three decades, and considerable progress has been made in theory, as well as in prac- tice. However, there are still several challenges remaining, including algorithm robustness and accuracy, especially for nonrigid modeling. The thesis focuses on solving these challenges and several new robust and accurate algorithms have been proposed.

The first part of the thesis reviews the state-of-the-art techniques of structure and motion factorization. First, an introduction of structure from motion and some mathematical background of the technique is presented. Then, the general idea and different formulations of structure from motion for rigid and nonrigid objects are discussed.

The second part covers the proposed quasi-perspective projection model and its application to structure and motion factorization. Previous algorithms are based on either a simplified affine assumption or a complicated full perspective projection model. The affine model is widely adopted due to its simplicity, whereas the extension to full perspective suffers from recovering projective depths. A quasi-perspective model is proposed to fill the gap between the two models. It is more accurate than the affine model from both theoretical analysis and experimental studies. More geometric properties of the model are investigated in the context of one- and two-view geometry. Finally, the model was applied to structure from motion and a framework of rigid and nonrigid factorization under quasi-perspective assumption is established.

The last part of the thesis is focused on the robustness and three new al- gorithms are proposed. First, a spatial-and-temporal-weighted factorization algorithm is proposed to handle significant image noise, where the uncertainty of image measurement is estimated from a new perspective by virtue of repro- jection residuals. Second, a rank-4 affine factorization algorithm is proposed to avoid the difficulty of image alignment with erroneous data, followed by a robust factorization scheme that can work with missing and outlying data. Third, the robust algorithm is extended to nonrigid scenarios and a new augmented nonrigid factorization algorithm is proposed to handle imperfect tracking data.

The main contributions of the thesis are as follows: The proposed quasi- perspective projection model fills the gap between the simplicity of the affine model and the accuracy of the perspective model. Its application to structure and motion factorization greatly increases the efficiency and accuracy of the algorithm. The proposed robust algorithms do not require prior information of image measurement and greatly improve the overall accuracy and robustness of previous approaches. Moreover, the algorithms can also be applied directly to structure from motion of nonrigid objects.