1. INTRODUCTION

Face recognition has become a key tool for unobtrusive human identification. Various government entities across the globe, including U.S. DHS, have proposed using face recognition at entry/exit ports and for border security [1,2]. In 2014, the FBI fully deployed the next generation identification (NGI) system [3] to broaden the biometric modalities at their disposal for fighting crime. One of the modalities being exploited is face biometrics. With the widespread proliferation of face-recognition systems for access control and surveillance, much focus has been placed on developing visible-spectrum systems robust to various conditions, such as illumination, pose, and expression [4–7]. While a plethora of techniques have been developed to address these issues, a relatively overlooked challenge for visible-spectrum face recognition is the application of makeup or cosmetics, which can alter the facial signature.

In the visible spectrum (wavelengths 400—750 nm), the perceived color of an object is a result of the spectral response of various color pigments that make up the surface material reflecting visible light. Recognizing faces using visible-spectrum imaging relies heavily on the key edges of the face, namely, edges of the eyes, nose, and mouth. Since light captured in visible-spectrum images is predominantly reflected light, these edges are dependent upon the source of illumination and the reflectivity of the facial tissue or any material applied to the face.

In contrast, electromagnetic radiation in the long-wavelength infrared (LWIR) thermal band (8–14 μm) is typically emission dominant. Thermal radiation is naturally emitted from facial skin tissue and can be acquired passively even if makeup or cosmetics have been applied to the surface of the skin. As a new modality recently applied to face recognition, polarimetric-thermal imaging [8–10] allows for the collection of geometric facial information from thermal imagery and may provide advantages over conventional thermal imaging in the presence of makeup or cosmetics. Figure 1 provides an example of the visual effects of face paint that are observed in visible, thermal, and polarimetric-thermal images.

 

Fig. 1. (a) Visible image taken of subject without oil-based paint (left), partial coverage (middle), and full coverage (right). Simultaneously acquired (b) thermal and (c) polarimetric-thermal images.

Download Full Size | PDF

Applying artificial materials (e.g., cosmetics, paint) to the surface of an object changes the perceived spectral characteristics of the object and can introduce false edges or masking of true edges when viewing images acquired in the visible spectrum. In this paper, we investigate the thermal properties of materials commonly applied to the face and their effects on LWIR imaging. We offer a brief discussion of polarimetric-thermal properties under various environmental conditions. We then provide an experimental study examining the spectral properties of materials when applied to the surface of a metallic sphere with Krylon flat black paint. Finally, we provide a practical application by investigating the feasibility of acquiring and recognizing a polarimetric-thermal facial signature in the presence of these materials.

2. BACKGROUND

Polarimetric imaging in the LWIR part of the spectrum is often dominated by thermal emission and not reflection. To understand under which conditions the imaging scene is dominated by thermal emission, let us consider a hypothetical scene depicted in Fig. 2. The three temperatures of interest are the temperatures of the apparent optical background $T_o$, of the target $T_t$, and of the image background $T_i$. For example, let us consider an outside scene on a warm sunny day with $T_i=25°\mathrm{C}$ and $T_o\le 0°\mathrm{C}$ for the upper (sky) portion of the target. The bottom (ground) portion of the optical background would, in general, be different from the upper portion of the optical background. If $T_i\approx T_t$ and $T_t\gg T_o$, then the polarization signature of the target will be dominated by the thermally emitted waves. However, if $T_i\approx T_t$ and $T_t\ll T_o$, then the polarization signal will be dominated by the optical background that has been reflected from the target. In both scenarios, there will be some image background waves that will reach the camera. However, these waves can only be measured by the focal plane array (FPA) after undergoing diffraction or lens aberrations, so their contribution to the polarization of the image is usually weak. The above scenarios are discussed in greater detail and have been experimentally confirmed in [11].

 

Fig. 2. Typical imaging scene is shown. In this scene, waves produced by the optical background are denoted by a dashed blue line, waves emitted by the target are denoted by a solid green line, and waves emitted by the image background are denoted by a dotted red line.

Download Full Size | PDF

A common case of practical interest occurs when $T_t\gtrsim T_o$. In such cases, both emission from the target and reflection of the optical background must be considered. From the Fresnel equations, the waves that have been emitted and reflected from the target tend to be orthogonally polarized with respect to each other, causing a reduction in the polarization state. If the optical background temperature is sufficient, there can be a complete cancellation and the polarization signature of the target vanishes, even though there is a temperature difference between the target and the image and optical backgrounds. In practice, one can infer from the degree of linear polarization (DoLP) if the scene may be treated as an emission phenomenon. Even for targets with high nominal emissivity, say, ${ϵ}_{\perp ,\parallel }\gtrsim 0.8$, the temperature alone will not determine whether the scene will be emission or reflection dominant. There are many parameters such as concaveness of the surface, opaqueness, and surface roughness that can influence this. We will concentrate on surface roughness because it has a strong influence on emissivity and reflection [12,13].

For example, for a surface with wavelength-order roughness, the measured real and imaginary parts of the effective refractive index can be 30% and 70% lower than for a smooth surface, respectively [12]. Moreover, for a surface with subwavelength-order roughness ($\lesssim \frac{\lambda }{10}$), both parts of the measured effective refractive index can be over 85% lower than its smooth counterpart [12]. This behavior is expected because Rayleigh scattering becomes more significant as surface roughness decreases beyond the wavelength of interrogation.

Determining the emissivity of human skin is an active area of research. Steketee [14] found that human skin has a thermal emissivity of $0.98\pm 0.01$, making it a strong emitter of thermal radiation; however, Togawa [15] argued that this value may be incorrect because it does not account for the temperature gradient across the surface of the skin, resulting in an underestimation of skin temperature. Recently, Sanchez-Marin et al. [16] developed a novel, noncontact approach to estimate the emissivity of human skin that does not rely on directly measuring the temperature of the skin. They project a ${\mathrm{CO}}_2$ laser beam on the surface of the skin and calculate the difference in radiation before and after the projection. Using Kirchhoff’s law and defining the sum of emissivity and reflectivity of an opaque body to be one, they provide a measure of skin thermal emissivity around 0.996 at a wavelength of 10.6 μm. This high emissivity enables polarimetric-thermal imaging to be useful for acquiring human facial signature for biometric recognition.

3. THEORETICAL ANALYSIS

Makeup may be used to conceal blemishes or alter various visible features of the face for cosmetic purposes. These materials, when applied to the surface of the face, have been found to compromise face-recognition algorithm performance in the visible spectrum [17–19]. Various studies have been conducted to develop algorithms to overcome the challenges posed by the presence of cosmetics with limited success [20–22]. To investigate the robustness of polarimetric-thermal-based face recognition to the application of cosmetics and other materials, we conduct an initial experiment using a spherical blackbody target, both with and without materials applied to the surface. Next, we collect corresponding visible and polarimetric-thermal face images for two scenarios: application of everyday cosmetic makeup and an exaggerated application of face paint.

A. Selection of Materials

The emissivity of a surface is affected by its morphology and can be altered by the application of additional materials (e.g., cosmetics and paint) to induce apparent surface temperature differences. We consider a stationary target with surface temperature $T$ and nominal emissivity $ϵ$. To find the expected radiance for our target, we find the within-band radiance by integrating the Planck’s radiation function following the method of Widger and Woodall [23]

To find the within-band radiance, we find the difference between two one-sided integrals

Using Eq. (3) and considering a thermal blackbody with absolute temperature of 308 Kelvin (corresponding to skin temperature surface of approximately 35°C), skin surface patch with ${ϵ}_{\text{skin}}=0.99$, and a neighboring skin surface patch concealed with cosmetic material, where emissivity ${ϵ}_{\text{material}}=0.97$, we can expect a reduction in radiance of approximately $0.08\times {10}^{-3}{\mathrm{Wcm}}^{-2}{\mathrm{sr}}^{-1}$, corresponding to an apparent temperature difference of approximately 1°C between the two patches. While factors such as sensor noise tolerance (i.e., noise-equivalent temperature difference), sensitivity, and material thickness and layering will ultimately determine the magnitude of reduction and whether this reduction is measurable or not, for materials applied to the surface of the face with much lower emissivity, say, $ϵ=0.75$, it is expected that discontinuities will be observed at the boundary of the skin and covered patches. In contrast, polarimetric-LWIR imaging is invariant to these changes and expected to produce consistent results in the presence of smoothly applied materials. Eight different products are used to induce various effects in the visible spectrum. In general, these products consist of three core materials: oil, wax, or powder, having an emissivity greater than 0.75.

B. Metallic Sphere Study

The background discussion suggests that if conditions are such that thermal emission dominates, then the polarization signature of the target will be independent of illumination conditions and the signature will be sensitive to surface topology and roughness. To demonstrate, we consider a metal ball painted with Krylon black paint onto which three different makeup materials have been applied in different quartiles (see Fig. 3). The ball is heated to 45°C and a division-of-time spinning achromatic retarder, LWIR polarimetric sensor with Stirling-cooled mercury cadmium telluride (HgCdTe) FPA, described in further detail by Gurton et al. [8], is used to obtain a DoLP image of the ball. The ball is allowed to cool down to 30°C while DoLP images are recorded (see Fig. 4).

 

Fig. 3. Metal ball painted with Krylon flat black paint and three different makeup materials is shown. The following makeup paints have been applied to the ball: upper-left painted with gel makeup, lower-left painted with black makeup paint, lower-right painted with green makeup paint, and upper-right is makeup free.

Download Full Size | PDF

 

Fig. 4. (a) DoLP and (b) retrieved mapping surface normals $\theta$ for a ball heated to (top) 45°C, (middle) 35°C, and (bottom) 30°C.

Download Full Size | PDF

It is important to note that the measurements are made indoors where additional heat sources (e.g., human bodies and fluorescent ballasts) contribute to the optical background, thereby adding noise to the polarimentric emission signature of the target. By expressing DoLP in terms of ${ϵ}_{\perp ,\parallel }$ and using Fresnel relations, we can obtain the angle $\theta$ between the surface normal $N$ and the line-of-sight of the camera [24]. The Fresnel relations depend on the effective refractive index $n_{\text{eff}}$, which also depends on the surface roughness. Since we know the shape of the ball and its corresponding surface slopes, we can select an effective refractive index that will provide reasonable results. We choose $n_{\text{eff}}=1.2+0.1$, which also is within reasonable values of the nominal refractive index for Krylon paint. Because we are interested in the distribution of the angles and not their absolute values, we use the same $n_{\text{eff}}$ for the other three portions of the ball; see Fig. 4(b).

C. Results and Discussion

The makeup materials had different consistencies and their application to the metal ball resulted in significantly different surface morphologies. For instance, the gel makeup in the upper-left portion of the ball went on thickly and relatively nonuniformly, resulting in the greatest amount of surface roughness. Random surface roughness tends to decrease the polarization signature. Its presence on a sloped surface can be considered through the introduction of individual surface facets. The total DoLP retrieved is the superposition of the signals coming from all the individual surface facets. When surface roughness is introduced, facets having smaller emission angles will have a greater projected area and hence a greater contribution to the signal. Because of this, the gel makeup in the upper-left quartile has the greatest surface roughness, and consequently the greatest reduction in DoLP and the retrieved surface normals. This can be seen in Fig. 4. The bare region in the upper-right portion is the smoothest and has the greatest DoLP and retrieved surface normals. The black and green makeup regions of the lower hemisphere have comparable surface roughness and their signals are reduced and almost indistinguishable from each other. The gel makeup in the upper-left quartile has the greatest surface roughness and the greatest reduction in DoLP and retrieved surface normals. While the makeup paints have a dramatic effect in the visible spectrum (Fig. 3), the effect is rather minimal in the LWIR and polarimetric LWIR images, which is largely due to the surface roughness in the application process.

One other effect to note is the specular reflection of the fluorescent lights in Fig. 3. We note that the reconstructed surface normals in Fig. 4(b) appear offset toward the upper hemisphere, which would result from the contribution of the wave emanating from the fluorescent ballast. In this study, the effect of surface morphology on the polarimetric measurements (i.e., DoLP) dominated the effect that the material composition had on radiometric measurements (i.e., total LWIR intensity). As the temperature is reduced to 35°C, we see only minor changes in the images. However, when the temperature is reduced further to 30°C, the temperature is now comparable to that of the optical background, and the reflected contributions play a significant role. At the edges of the ball, where the emitted DoLP is greatest, the background significantly decreases the DoLP. However, at the center of the ball, where the emission angles, and resulting DoLP, approached zero, the signal has increased. This is because the reflected contributions from the dominant components of the optical background are to the side and strike the surface at oblique incidence. The total effect of the background is to wash out the measured image.

4. APPLICATION TO FACE RECOGNITION

In this section we assess the effect of makeup application on visible, LWIR, and polarimetric-thermal images acquired from faces both with and without makeup application. We perform an analysis of the collected data by examining the match scores generated through comparisons. To compare the face images, we use a face-recognition framework to cross-modally match the visible and polarimetric images, as in [9]. It is expected that the performance of visible-to-visible matching will suffer when the makeup and paint are applied due to the alteration/deterioration of the face signature. Conversely, the polarimetric-thermal-to-visible matching is expected to remain relatively consistent, due to the invariance/stability of the polarimetric-thermal signature to these surface materials.

A. Polarimetric-Thermal Face Acquisition

Simultaneous visible and polarimetric LWIR face imagery is collected using the LWIR polarimeter described in Section 3.B and as in [9]. We assess the utility of polarimetric LWIR for robustness to cosmetic makeup and paint by analyzing similarity score results against a dataset of known visible identities. We use the same approach as in [9] for preprocessing and classification. Figure 5 shows example imagery from this experiment and summarizes the regions of the face and types of material applied. Two scenarios are collected: one representing light application of makeup and the other examining an exaggerated case where makeup paint is applied to completely mask the structure of the face. Visualization 1 and Visualization 2 contain videos acquired in visible, thermal, and polarimetric-thermal dynamic facial signatures. These videos further illustrate the impact of cosmetic materials obscuring facial texture and geometry in the visible spectrum.

 

Fig. 5. Figure illustrating face signature differences in visible before (left) and after (right) the application of makeup for two scenarios. Annotations indicate the type of material applied to each portion of the face. Visualization 1 illustrates the signature differences of a dynamic face with cosmetics acquired by visible, thermal, and polarimetric-thermal imagers. Visualization 2 illustrates the same for the exaggerated makeup example.

Download Full Size | PDF

B. Facial Signature Qualitative Analysis

Here, we present sample imagery acquired using both visible-band and polarimetric-thermal sensors. In Fig. 6, the signature differences between visible and polarimetric-thermal images are illustrated (top and middle rows). Looking at Fig. 6(a), we see significant differences in the visible-spectrum facial signature, before and after the application of cosmetics, specifically around the mouth region. Most notably, we see an enlargement of the mouth and sharpening and thickening of the edges in the eyebrow region. We see little difference in the acquired signature from the thermal (decrease in radiance around the lips) and no observable differences from the polarimetric-thermal images in the columns of Figs. 6(b) and 6(c). This alteration is even more substantial when considering the same columns of Fig. 6(d) for the extreme scenario. The polarimetric images show only minor changes after the application of makeup.

 

Fig. 6. Figure illustrating the cropped face signature used for recognition before (top) and after (middle) the application of cosmetic material in (a), (d) visible; (b), (e) polarimetric-LWIR; and (c), (f) conventional LWIR. Bottom row illustrates local structural similarity measures of pixel neighborhoods in the collected imagery.

Download Full Size | PDF

We provide quantitative analysis of the raw pixel-intensity measurements from the sensor before and after the application of makeup by comparing local structures of pixel neighborhoods using the structural similarity (SSIM) metric [25]

Visible and polarimetric images used in this analysis were acquired simultaneously, before and after the application of facial cosmetics. Images are aligned by manually extracting a large set of corresponding fiducial points in and around the interior of the face, jawline, and hairline. A multipoint registration is performed using a piecewise linear geometric transformation. Before and after corresponding images from the two test cases are registered using a piecewise linear transformation found from a large number of corresponding control points covering the core region of the face. The SSIM is calculated over corresponding $8\times 8$ pixel regions across the registered images.

The results are shown in the bottom row of Figs. 6(a)–6(f) as a heat map, where large values (red color) represent high strucural similarity and negative values (blue color) indicate regions that are strucurally dissimilar. The range of similarity values is $-1$ to 1. A global SSIM is calculated over the entire image by averaging values across the local SSIM map. Of note in Figs. 6(a)–6(c) is the region around the mouth, where the impact of the cosmetic material drastically distorts the perceived facial structure in the visible spectrum. Additionally, we notice a low similiarty measure in the thermal imagery within this region. Investigating the pixel values corresponding to the LWIR radiometric data, an increase in radiance is observed after the application of makeup, which is most likely a result of the increase in emissivity due to the material applied to this region being primarily oil-based. Changes to the polarimetric image are minimal before and after the application of cosmetic materials.

For the more extreme makeup scenario, we notice a much greater impact on the structural similarity of the registered visible images. The SSIM between the before and after makeup visible face images was only 0.3106, while the SSIM between the before and after makeup conditions in the thermal and polarimetric-thermal images was 0.8251 and 0.6683, respectively. This indicates that cosmetics had significantly less impact on the LWIR and P-LWIR facial signatures than the visible facial signatures. The local SSIM maps indicate regions of significant change due to the appliation of cosmetics. The high SSIM measured between the LWIR imagery before and after makeup application is most likely a result of the materials (oil-based) having similar properties to human skin.

C. Facial Comparison Performance

To assess the performance and robustness of polarimetric-thermal imaging to cosmetics and makeup, we use a face-recognition framework to cross-modally match the visible and polarimetric images described in [9], which builds support vector machine models in a high dimensional space using visible-spectrum features. Subsequent “on-versus-rest” classification is performed using only thermal probe feature data. This type of matching is referred to as cross-spectrum or heterogeneous matching, where we seek to recognize a polarimetric-thermal probe face image in a biometric database containing only visible-spectrum facial imagery of individuals. It is expected that the performance of visible-to-visible matching will suffer when the makeup or paint is applied, due to the alteration/deterioration of the face signature. Conversely, the polarimetric-thermal-to-visible matching is expected to remain relatively consistent, due to the invariance of the signature to these surface materials in this domain.

The probe data contains multiple samples of two subjects from the two test cases: cosmetics and exaggerated paint. Seven samples from each subject are extracted before and after the application of the materials. Subjects were asked to speak a phrase to introduce variability to the dataset through a variety of facial expressions (as shown in Visualization 1 and Visualization 2). A total of 248 visible samples, four each from 62 subjects are used to construct the gallery set. In total, we perform 56 within class and 3416 out of class comparisons. All images are geometrically normalized using manually extracted fiducial points, corresponding to the centers of the eyes and registered as in [26].

Figure 7 (top) shows the results of the first experiment matching visible-to-visible and polarimetric-to-visible face images before and after the application of normal cosmetics. We compare visible and polarimetric samples to the genuine target subject and additional imposter, or nonmatch, subjects, where the genuine target subject is always from a visible image and expected to match more strongly with another visible image than a LWIR or polarimetric-LWIR image. In the left plot, the average scores for the “target subject,” or genuine, comparisons are indicated. Here, it can be seen that the similarity score is reduced for the visible image probe and not for the polarimetric image probe in the presence of makeup. In the right plot, the score difference is shown for both visible and polarimetric images. Note the decrease, albeit modest, in the average similarity measure of 0.8% across the visible probe images before and after normal makeup application. For the polarimetric-thermal case, we observe an increase from 0.78 to 0.94, which may be a result from increased SNR due to “smoothing” of surface structure inherent to the cosmetics. The scores range from $-1$ to 1.

 

Fig. 7. (Top-Left) Average match score resulting from comparison of probe images with target and nontarget data for the no makeup/makeup case using visible and polarimetric-thermal images. (Top-Right) Difference in match scores of target and nontarget/match subjects resulting from the application of makeup. (Bottom) Same charts using results from the exaggerated cosmetics scenario.

Download Full Size | PDF

In Fig. 7 (bottom), the results from testing the exaggerated paint scenario are shown. Here, we see a larger drop in similarity measure when comparing visible probe images before and after the application of paint. We record a 62.7% drop, from 0.8821 to 0.3284, in the average similarity score across the visible exaggerated paint samples. This effect is most likely due to the extreme change in edge information introduced by the diferent color pigmentations and rerflectance properties of the paint in the visible spectrum. We observe only a slight increase in similarity score for the polarimetric-thermal images, most likely attributable, again, to the “smoothing” of the surface structure. These results demonstrate that polarimetric-thermal imaging is advantageous for biometric face recognition, as the polarimetric-thermal facial signature is relatively unaffected by cosmetics/makeup.

We measure the statistical significance of our findings using the Wilcoxon signed rank test [27]. This test produces p-values of 0.0469, 0.0156, 0.0156, and 0.1563 for the before and after: makeup from visible, exaggerated paint from visible, makeup from polarimetric, and exaggerated paint from polarimetric, respectively. This indicates the reduction in scores from the visible before/after makeup and exaggerated paint are validated at a 95% and 98% confidence level, respectively. Also of note, the p-value from the score changes from polarimetric images, in the exaggerated paint scenario, 0.1563, indicating no statistically signficant changes in scores due to the applied paint.

While the application of makeup materials to the metallic sphere did impact its polarimetric signature in Section 3, the effect on the facial signature is relatively less, as demonstrated in Section 4. This is likely because the surface roughness of the metallic sphere increased substantially with makeup application, especially the makeup gel; whereas, the change in surface roughness was minimal when the makeup was applied to human skin (i.e., makeup can be applied much more uniformly on human skin than on the metallic sphere).

5. CONCLUSIONS

In this paper, we have presented an approach demonstrating the efficacy of polarimetric-thermal imaging for recognizing faces in the presence of cosmetics and makeup. The approach utilizes polarimetric-thermal imaging, which depends most strongly on the surface morphology and textural details of the face, rather than the chemical composition of the outer materials. While the nominal emissivity of common materials used in cosmetics and face paint were identified, the polarimetric-thermal signatures were relatively invariant to these materials, due to the similarity in thermal properties of the materials and human skin. Additionally, we quantified the robustness of polarimetric-thermal-based face recognition to cosmetics by showing the unaltered feature representation and match score distributions of the polarimetric images acquired before and after the application of facial obscurants.