Non-Rigid Photometric Stereo with Colored Lights

Example reconstruction

Example input image from dancing video

Example 3D reconstruction

Abstract

We present an algorithm and the associated capture methodology to acquire and track the detailed 3D shape, bends, and wrinkles of deforming surfaces. Moving 3D data has been difficult to obtain by methods that rely on known surface features, structured light, or silhouettes. Multispectral photometric stereo is an attractive alternative because it can recover a dense normal field from an un-textured surface. We show how to capture such data and register it over time to generate a single deforming surface.

Experiments were performed on video sequences of untextured cloth, filmed under spatially separated red, green, and blue light sources. Our first finding is that using zero-depth-silhouettes as the initial boundary condition already produces rather smoothly varying per-frame reconstructions with high detail. Second, when these 3D reconstructions are augmented with 2D optical flow, one can register the first frame’s reconstruction to every subsequent frame.

Papers

	Video Normals from Colored Lights Gabriel J. Brostow¹, Carlos Hernández², George Vogiatzis³, Björn Stenger⁴, Roberto Cipolla⁵ ¹University College London, ²Google Inc., ³Aston University, ⁴Toshiba Research Europe, ⁵University of Cambridge Published in IEEE TPAMI, Vol. 33, No. 10, pages 2104-2114, October 2011. [paper pdf, 12.7 MB]
	Non-Rigid Photometric Stereo with Colored Lights Carlos Hernández¹, George Vogiatzis¹, Gabriel J. Brostow², Björn Stenger¹, Roberto Cipolla² ¹Toshiba Research Europe, ²University of Cambridge Published in ICCV 2007 [paper pdf, 5.5 MB] [poster pdf, 1.1MB]

BibTeX

@ARTICLE{BrostowPAMI11,
  author = 	 {Brostow, G. J. and Hern\'andez, C. and Vogiatzis, G. and Stenger, B. and Cipolla, R.},
  title = 	 {Video Normals from Colored Lights},
  journal = 	 {TPAMI},
  volume =       {33},
  number =       {10},
  pages =        {2104--2114},
  month =        {October},
  year = 	 {2011}
}

@INPROCEEDINGS{HernandezICCV07,
  author =       {Hern\'andez, C. and Vogiatzis, G. and Brostow, G. J. and Stenger, B. and Cipolla, R.},
  title =        {Non-Rigid Photometric Stereo with Colored Lights},
  booktitle =    {ICCV},
  location =     {Rio de Janeiro, Brazil},
  month =        {October},
  year =         {2007}
}

Downloadable Videos

Old + New [41 Mb mpeg4 avi]
Short [23 Mb mpeg4 avi]
Short Small [5.7 Mb mpg1]

Results (ICCV07)

New Results

Data

Input sequences and light calibration files
DirectShowParser (a.k.a. mxf2avi) to extract frames from mxf files
Link to codecs + utility for extracting frames [alternative]

Mesh data for "sweater" and "pirate shirt" sequence
TexMesh3D to view tri files
Meshlab to view obj files

Contacts

Gabriel Brostow
Carlos Hernández
George Vogiatzis
Bjorn Stenger
Roberto Cipolla