Collecting 3D Dynamic Face Data

We propose to implement a synchronized video/audio data capture system (camera/microphone array). The main work includes camera/microphone synchronization and multi-camera calibration.

Documents:

1. Technical Report (1MB)

2. Presentation (10MB)

Data:

1. 3D foam head scanned by CyberWare, used as the ground truth for reconstruction (5MB) [download]

2. Checker board for camera calibration (4.7MB) [download]

3. Sample raw avi video (Bayer Pattern) (28MB) [download]

4. Sample rgb24 color avi video (12MB) [download]

5. Demo videos for facial marker tracking [too big, provided upon request]

6. etc.

Codes:

1. Multi-camera Data Collection in C++ [download]

Input: Multiple Point Grey Dragon Fly Cameras

Output: raw (Bayer Pattern) avi files

Feature: synchronization, compression (Microsoft RLE8)

2. Color Conversion in C++ [download]

Input: raw (Bayer Pattern) avi file

Output: rgb24 color avi file (not compressed)

Feature: rawàrgb, select different de-mosaic algorithms, same the raw frame

3. 3D ground truth labeling tool in Matlab[download]

Input: texture and range data from Cyberware Scanner,

Output: 3D coordinates of the points labeled by the user

Feature: Interactive, remove the scanning noise

4. 2D facial marker tracking in Matlab [download]

5. 3D reconstruction by stereo triangulation in Matlab [download]

6. etc.

Useful Links:

Point Grey Research Inc. http://www.ptgrey.com/

About Bayer Pattern: http://en.wikipedia.org/wiki/Bayer_pattern

Camera Calibration Toolbox from CalTech http://www.vision.caltech.edu/bouguetj/calib_doc/

References:

1. Y.X. Hu, H. Tang, T.S. Huang, Camera and Microphone Array for 3D Audio/Visual Face Data Collection, The 33rd International Conference on Acoustics, Speech, and Signal Processing (ICASSP2008)

2. Luhong Liang, Yu Luo, Feiyue Huang, and Ara V Nefian, "A multi-stream audio-video large-vocabulary Mandarin Chinese speech database", IEEE International Conference on Multimedia and Expo (ICME 2004) June, 2004, Taiwan, China

3. Rodney Goh, Lihao Liu, Xiaoming Liu, and Tsuhan Chen,"The CMU Face In Action (FIA) Database", in the Proceedings of the 2005 IEEE International Workshop on Analysis and Modeling of Face and Gestures, in conjunction with the 2005 International Conference of Computer Vision (ICCV '05), Beijing, China, October 2005.

Contacts:

Yuxiao Hu, Hao Tang

{Hu3,HaoTang2}@ifp.uiuc.edu

IFP Group, Beckman Institute, UIUC