Hybrid Facial Capture and Animation
High quality acquisition and realistic animation/rendering of the human face is an important task that is studied intensively in literature. We are developing a framework for realistic facial capture and animation from captured multiview data. In this framework we are using geometric as well as textural data to store the facial performance of a subject. The geometry is represented as a parametric model (linear and non-linear), which allows interpolating smoothly on the manifold of facial expressions using a small set of parameters. Optionally, we compute a vertex refinement vector to adapt the model geometry to the ture geometry of the captured person. In order to allow for realistic renderings, we are planning to use texture based animation for a more realistic animation and rendering of the human face model.
The tasks in this projects are:
- creation of compact parametric models for facial geometry (identity) and deformation (expression)
- creation of dynamic textures from a calibrated multi view video stream
- development of efficient and robust tracking algorithms from stereo video
- development of efficient algorithms for realistic rendering using geometric animation and image based rendering techniques
Capture: In order to increase the expressive power of the geometric model we are exploring different linear and non-linear representations of the facial expression manifold. This includes for example an automatic partitioning of the face template in regions that are used to describe the facial deformation of a set of training examples using a method called archetype analysis. On the non-linear side, we explore the generation of low dimensional expression manifolds from small sets of training data as well as robust tracking algorithms for the extraction of tracking parameters from stereo video.
Automatically detected facial regions using archetype analysis
In order to further extend the realism of the created face models, we will use image-based rendering techniques to capture fine grain movements and light variations, which cannot be explained with geometry alone.
Videos
Publications
W. Paier, A. Hilsmann, P. Eisert,
Hierarchische Rekonstruktion und Modellierung von 3D Gesichtsgeometrie, Proc. innteract 2016, Chemnitz, Germany, June 2016.