4D : AUGMENTATION
The quality of the retargeting system did not reach a high level of believable performance due to several reasons. The facial expressions of the character animation system were not anatomically correct by the time of the retargeting. Secondly, the daily neutral photogrammetry session was not available on the day of the performance. This lead to an experimental approach of creating a three dimensional model of the human face with the matrix of the allocated markers on the facial surface.
PHOTOGRAMMETRY
During the construction stage of the digital character, there were discovered technologies that allowed the creation of anatomically correct facial surfaces in three dimensional space. This was done by multiple view camera photography of synchronised capture. Similar to the workflow of integrating computer generated imagery, the photogrammetry method of reconstructing geometry surface of the human face is employed. This is particularly realised in the form of capturing a neutral face of the character to obtain colour information that can be projected from photographs onto the polygonal model.
Closed eyes are used to extract the patch of the upper eyelid that covers the unrevealed area of the human face. Capture of dentures is performed for both lower and upper teeth of the subject. This includes additional capture of the oral cavity interior and the shape of the tongue.
Six basic emotions are captured to obtain the colour difference in blood flow of the face and the creation of the wrinkle map.
After the colour driven session is accomplished the variety of facial expressions undergo the photogrammetry capture. This is done in conjunction with colour dot pattern placement on the surface of the actor’s face to measure skin deformation. The capture of the facial expressions is driven by the motivation to create a highly expressive character performance system that is capable of speech animation and expressing a full range of emotional states.
Core selection of the Facial Action Coding System (FACS) (Paul Ekmann, 1979) expressions are captured to access the anatomical possibilities of the human face.
FACE
After the alignment of the photographs it is significant to undergo the procedure of constructing a digital identity of the character using the geometry with the deformation ready topology. This process is sometimes referred to as re-topologising or re-meshing. The basic principle is to morph the real-time capable mesh of the topology of the human face into the scan data. This is done by radio basis function, driven with the guide of manual landmark constraints. The markers are placed on both geometric surfaces with the knowledge application about the anatomy of the human face. This includes regions of the nasojugal and nasolabial groves and corners of the lips and eyes in order to communicate constraint differences between two surfaces.
ACTION UNITS
To create a performance-compliant animation system for the character the facial expression of FACS and speech shapes has to be created based on the translation or wrapping procedure. Based on the example of the advanced facial character system, the generation of the facial expressions can be simulated based on the similar character. Depending on the topology structure this could be done directly as a delta shape or as a wrap deformation that would constrain one surface to the other. Depending on the result of the transfer method the fine sculpting of each facial expression has to take place. The cleanup and the sculpting time would depend on the quality of the facial geometry and the complexity of the individual expression.
The blend between the two methods is the approach for creation of the final height map that communicates the elevation and reflectivity information to the rendering engine. The specularity map can be achieved by the process of artistic painting over the highly detailed skin pore level surface to obtain a break-up of the specularity effect constrained with the cavity masking.
ARTIFICIAL VISUAL APPARATUS
[ A V A ]
AVA or artificial visual apparatus refers to a method of approaching the creation of eyes in virtual agents by digital technology. The design of a digital eye has two angles of the investigation. One is concerned with the high quality data acquisition and construction of the scanner device for capturing macro details of the human eye. The second angle is based on the anatomical knowledge about the human eye structure prior to the practice of making digital doubles of actors. The shape of the eyeball is approximated based on the MRI data.
THE SCANNER
Following the action research methodology, a number of actions were taken in order to obtain high quality data of the human eye. The problem of capturing the human eye is in its size and occlusion of the orbital bone that creates shadow. Furthermore, the eye has a very reflective and wet spherical surface that brings a complexity for multiple view reconstruction. The main target of creating the scanner was to obtain 4D data in order to have the ability to reconstruct a time-based reference of the human eye. The plan of the construction came from the idea of making a helmet- based solution for the eye capture and eventually moved on to the static scanner solution.
The 3D model was created as a prototype to understand the position of the human skull in relation to the scanner frame and the mounted cameras.
The macro lenses were used to capture the high quality result of the human eye in conjunction with the strong continuous light source. Flash photography was used separately in order to capture iris structure and colour in high detail. The scanner was designed to have the ability to capture the eye from four multiple angles for further 3D and 4D reconstruction.
After planning the structure, an experiment with a manikin took place in order to understand the process of capturing an extreme close-up of the human eye. The remote trigger device was used in order to release the shutter without physical contact with the several cameras simultaneously.
A light source was provided by a table lamp pointing its view to the area of interest. Once the test images of eye capture were created, the scanner was used on set with the actress to obtain the extreme close-up of the human eye, both in static and video reference.
4D experimentation with the device had no success due to the low number of cameras, poor lighting conditions and distance between the camera and the eye. The stability of the human skull in physical space was the key problem in reconstructing 4D data as the surface is mobile in relation to the change of its landscape.
Furthermore, the human eye’s high reflectivity and refraction characteristics including its spherical egg-like shape brought additional complications during the 4D processing. However, the scanner is capable of producing high quality reference of the human eye’s appearance and sequential behaviour.
