Research on Multiview Differential Geometry of Curves and Surfaces

Ricardo Fabbri and Benjamin Kimia

This project proposes a paradigm shift for 3D reconstruction from multiple perspective projections, based on differential geometry. We have been developing a new framework to model curved structures on both space and time, including general non-planar curves, surfaces, shading, curvilinear camera trajectories, and nonrigid motion.

State-of-the-art camera calibration and 3D reconstruction systems are based on very sparse point features, such as SIFT, and projective geometry, which can only model points and lines or simple curves such as circles and other conic sections. These systems suffer from many of the following limitations: sparsity, requirements of simple scene, controlled acquisition, difficulty with non-planar objects, requirement of strong calibration, abundant texture, short baselines, and lack of geometric consistency. We believe these systems are useful but form only a module within a greater structure from motion system.

Given two or more views of a fixed space curve, in 2004 we have shown how the torsion of the curve can be reconstructed from image measurements (pdf). Two views are necessary and sufficient, and more than 3 views provide an over-constrained solution. This initial study has opened an entire new line of research, where we have extended these results to most types of contours (rigid, occluding, nonrigid), and have been modeling many other problems involving curvilinear phenomena in the geometry of multiple views, including camera auto-calibration from tracked curves, the use of surface patches and their shading under different illumination models, and fields of both short and long curve fragments for multiview applications.

Multiview geometry of a space curve

We are also working on a practical and comprehensive application for automatically reconstructing complex 3D objects from a sequence of images, and have completed many modules. One of our systems is based on image curve fragments as obtained from a subpixel edge linker. A simple step in corresponding curve fragments in two views is shown in the following figure, where the epipolar lines are in green:

Curve correspondence

Check out the website for Multiview 3D Drawings for a recent implementation presented at ECCV'16 and CVPR'17. See also CVPR'20 paper on the trifocal pose estimation modules.

A curve-based reconstruction has many advantages over point features:

We tackled the challenges of using linked curve fragments, namely the instabilities in linking and correspondence in the 3D Drawing system. The practicalities around nonrigid curves (e.g. occluding and other nonrigid), the trifocal bootstrapping of the system, and the surfacing of 3D Drawings are currently active topics.

Publications

(updated list available at Fabbri's home page)