Open Access
Add to BibSonomy
Abstract
We developed a method for visualization of makeup finishing with structured lighting. By analyzing images with a sequence of projection patterns, reflectance and transmittance of the cosmetic foundation (FD) layer were extracted as spatial maps using the difference between the light spread of bare skin and made-up skin. The spatial maps reflect conditions and distribution of applied FD under real situations. By calibrating the relationship between optical properties and the amount of FD applied, the application amount distribution was also estimated. Additionally, we proposed approximation formulae to estimate the above values without images of bare skin. These formulae provide good agreement with the original formula for reflectance.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Among the various types of cosmetic products, cosmetic foundation (FD) is used to give skin a good tone and realize a good complexion [1]. Although its main purpose is to diminish flaws like uneven skin color, natural-looking skin is also important. Acceptable results depend on the technique of application, which influences the amount and distribution of FD and its condition, in addition to the physical and chemical characteristics of FD itself. To achieve better results regarding those factors, many studies have been performed in the field of cosmetics [2–9]. The scope of assessment is not limited to just after application, but also after a certain amount of time, after which degradation can become an issue [3,4]. Although it is a matter of serious concern how to achieve and maintain the desired appearance of makeup, it is often difficult to see how FD is distributed on skin and how it can affect skin appearance. To judge the “finish” of makeup, visual evaluation is usually performed. In this evaluation, it is necessary to distinguish FD from the skin to separately examine the condition of FD; however, this often poses a challenge, even for a skillful evaluator. A visualization method to differentiate them easily is desired.
The main reason for the difficulty is the similarity between the color and spectra of bare skin and skin with applied FD. Typically, the color of FD is selected to be a similar to the color of the skin to which it is applied. This is favorable for easily achieving natural-looking makeup, and for minimizing flaws and artificiality. However, this characteristic makes the evaluation of applied FD difficult. As a result, the number of reports about the issue is small despite its importance. It seems to make sense that color unevenness may not be important if skin and FD are not distinguishable. However, because the optical properties of skin and FD are not exactly the same [5,6], even if facial flaws seem to be concealed well at the time of makeup application, such flaws can become evident under different lighting environments, such as under the sun, because of the difference in the spectrum, directionality, and intensity of the lighting environment under which FD is applied.
Nonetheless, some studies have attempted to determine the optical condition of makeup, although most of them are not sufficient for the evaluation of makeup quality. As a quantitative treatment of FD condition, measurement of the reflectance spectrum, L*a*b*, or expression with other color spaces are conceivable at first glance [3]; however, such indices are mixed with the color of bare skin, and then often become difficult to interpret. Some studies attempted to extract the spectrum of skin with applied FD from the spectrum of bare skin and from pre-calculated parameters following the Kubelka-Munk theory [7,8]. Although the spectra could be expected to have high reproducibility, the FD layer for the measurement was applied to an optimized substrate and not to skin. Another study examined the reflectivity of FD applied to skin based on light propagation using a Monte Carlo simulation [10]. However, the model tended to be oversimplified for the calculation, and the complexity of the actual phenomenon as a whole is not described well; thus, the application range is limited to the aspects the model covered. In addition, the studies described thus far are basically for a single point or average, and the spatial distribution of the parameters is difficult to evaluate. Two-dimensional spatial information is quite important in evaluating the quality of makeup. Another study attempted to image the spatial distribution of FD by applying a customized multi-bandpass filter to enhance the spectral difference between bare skin and skin with FD applied [5,6]. Although the spectra were similar, small differences were emphasized by the filter. Although this approach visualized the spatial distribution of FD on skin, which is vital for FD evaluation, it assumes specific spectra for bare skin and FD-applied skin to optimize the filter spectrum and analyze images of the FD distribution. As a consequence, the determined difference between the assumed spectra and the actual spectra can contain errors, which degrades the accuracy and detail of the results.
To distinguish FD and skin accurately, among the optical properties, we focused on scattering power instead of reflectance. A very thin layer of FD is used to modify skin appearance. This demands a large scattering power for the FD, and most FD products are designed this way. In contrast, bare skin is translucent. This means that the scattering power increases and the lateral spread under the skin surface decreases when FD is applied to skin; therefore, if this change can be visualized, the distribution of applied FD will be shown clearly. We previously invented a method to estimate the optical properties of an FD layer using this difference [9]; however, it was for single-point measurement, and not sufficient for evaluating the quality of applied makeup. To improve the evaluation, we used a method of structured light [11,12], which associates reflected images with modulated lighting to estimate the properties of subjects. The method is commonly used for small areas; however, potentially it can be applied to large areas, such as a whole face [13].
2. Materials and methods
2.1 Reflection model
Light reflection from FD applied to skin was modeled as follows (Fig. 1). Because human skin is translucent, the incident light spreads laterally under the surface and is partly absorbed; the remainder is then emitted from the surface. From the requirement that the FD should change the skin appearance with a thin layer, typically, FD should have a high scattering power, which means that inevitably the degree of lateral spread in the FD layer is much smaller than in skin. Based on the premise that reflection from the surface is removed in advance, we extracted two components according to the distance between the incident point and the exit point; in this paper, they are referred to as close and distant, respectively. The words close and distant by themselves are relative and ambiguous; however, their definition here is associated with the measuring device described in Sections 2.3 and 2.4. The ratio of reflected light to incident light is represented as R, and the reflections from the close area and the distant area are expressed by superscripts “C” and “D,” respectively. Subscripts “S” and “S + F” indicate bare skin and made-up skin, respectively, with “F” representing the FD layer. Additionally, an upper case R is used when incident light is measurable, and a lower case r, t is used when it is theoretical.
Fig. 1. Reflection model focusing on distance between the incident and exit points. (a) FD applied to skin; (b) bare skin; component elements at the close area in the (c) FD layer and (d) skin layer; (e) breakdown to multiple light paths at the close area in FD-applied skin.
We considered the light behavior at the close area based on the Kubelka-Munk theory [14] (Fig. 1). Only the longitudinal direction of light flow was considered. Two optical components of the FD layer were defined: the light that is reflected by the entry side, rF, and the light that penetrates to the opposite side, tF [Fig. 1(c)]. Here, we did not distinguish between the upward and downward direction and assumed both have the same values. At the close area, we assumed that the light travels back and forth between the FD and skin layer before it exits, and the total reflectance is the sum for these paths [Fig. 1(e)]. RS+FC is expressed as the sum along each reflection path, which is a geometric series except for the first term, and is expressed as
To estimate the reflection from the distant area, we first formulated the transmission of the FD layer on skin, including the paths going back and forth between the FD and the skin layer tS+FC. In the same fashion as RS+FC, tS+FC becomes a geometric series as
In the formula, the component going back and forth between the skin and FD at locations other than the close and distant areas is ignored. In the modeling, the effect of the interface is not considered. The correction of its effect is formulated in Section 2.5.
To estimate the unknown components rF and tF in Eqs. (1) and (3), we examined three types of formulae with different degrees of approximation. First, we derived precise solutions of Eqs. (1) and (3). A quadratic equation for rF was derived by eliminating tF from Eqs. (1) and (3); therefore, rF can be solved by a quadratic formula. Then, tF can be calculated by substituting the derived rF into Eq. (1) or (3). The precise solutions are represented as rF(p) and tF(p). Specifically, when
rF(p) and tF(p) are derived asNext, we derived approximate solutions for rF(a) and tF(a), in which terms higher than the second term in Eq. (1) and those higher than the first term in Eq. (2) were eliminated. The specific formulae become
Moreover, other approximate solutions rF(r) and tF(r) were also examined, which are basically the same as rF(a) and tF(a), respectively; however, averages were used for the values of bare skin instead. This approximation is practically meaningful, because when the values for bare skin are derived in advance, measurement of a subject’s bare skin is not required. This means that they can be applied even when it is difficult or impossible to acquire the corresponding values of bare skin against FD applied to skin point-by-point, which is a very common situation.
2.2 Measuring device
The configuration of the measuring system is shown in Fig. 2. As a light source, an M403H projector (NEC Corp., Tokyo, Japan) was used. For photography, a D500 camera (Nikon Corp., Tokyo, Japan) with an AF-S Micro NIKKOR 60mm f/2.8G ED lens (Nikon) was used. Polarizing plates were set in front of the projector and the camera in such a way that the polarization directions were perpendicular to each other to eliminate surface reflection from the acquired images. For calibration of the white balance and illuminance of images, a white chart was set under the chin rest in the field of view. Photographs were saved as RAW data, and then converted to linear gamma images. To execute sequences, software was developed with Visual Studio C# (Microsoft, Redmond, WA, USA). The output to the projector was duplicated and displayed on a monitor for verification.
The resolution of the projection patterns was 1920 × 1080 pixels, and that of the acquired images was 3827 × 5599 pixels. The distance between the front surface of the projector and the chin rest was about 50 cm, that between the front surface of the camera lens and the chin rest was about 54 cm, and the angle between the camera and the projector from the chin rest was about 17°. With respect to actual lengths on a subject, the resolution of the projection patterns was 5.7 pixels/mm, and that of photographs was 27.6 pixels/mm. In other words, the distance between adjacent pixels in the projected pattern on a photograph was 4.8 pixels. The full width at half maximum was about 7.1 pixels in the red channel, when a single pixel spot was projected onto a Spectralon white diffuse reflectance standard (99%) (Edmund Optics, Barrington, NJ, USA) at the chin rest after focus adjustment. Although the cause of the spreading contained the effect of the optical properties of the Spectralon standard, it was assumed to represent predominantly the resolution of the system as a whole.
2.3 Projection pattern
To estimate the optical properties of a subject, some studies used a single-point [15] or ring-shaped [16] light projection. However, the information acquired at one time is only for one point, and numerous acquisitions are required to obtain images. As another option, unidirectional stripe-like modulation is also often used, which realizes rapid acquisition [12]; however, some directions have low resolution. We chose intermediate patterns, which give good resolution within an acquisition time tolerable for human subjects.
The projection plane was divided using grids, and each grid was assigned one of the patterns in a sequence. Either every second or every third grid was selected in the horizontal and vertical directions for each pattern (Fig. 3(a)). Patterns were serially projected (Fig. 3(b)). By doing this, every point on a subject was illuminated by one pattern but not by any other. By gathering reflected light from the illuminated area, the reflection image from the close area was synthesized, and by also gathering light from the area farthest from the illuminated area, the reflection from the distant area was synthesized. The basic concept was as above; however, overlapping was permitted to improve the quality of synthesized images. By enlarging each grid, light intensity degradation near the boundary of the grids was reduced. The adverse effect of subject movement during measurement was also reduced to some extent. In the enlargement, every point on a subject should not be illuminated by at least one pattern.
Fig. 3. (a) Basic concept of projection pattern construction by space segmentation (in the case of 3 × 3); (b) Sequence of projections.
Several parameters are conceivable for optimization, including the size of grids, the distance between adjacent selected grids in a pattern, and the size of the overlap area. These values should be examined under the limitations that pixel size is the minimum unit, and the regularity [i.e., the interval and overlapping between A – B, B – C, and C – A in Fig. 3(a) must be the same] is required. The four types of patterns we examined in the experiment are shown in Fig. 4. Taking Sequence II [Fig. 4(b)] as a base, overlap is reduced in Sequence I [Fig. 4(a)], the size of the grids is enlarged in Sequence III [Fig. 4(c)], and both the size of grids and overlap are enlarged in Sequence IV [Fig. 4(d)].
Fig. 4. Projection patterns for each sequence (magnification of the left top corner). Each image represents a sequence: (a) I, (b) II, (c) III, (d) IV. Each square in black solid grids represents one pixel of the projected patterns. The colored squares represent projection patterns, and the same color means projection at the same time. Sequence I (a) is a set of 2 × 2 = 4 patterns, and (b), (c), (d) are sets of 3 × 3 = 9 patterns.
2.4 Preprocessing
For a series of pictures in a sequence, the pixel value for each RGB channel at each coordinate in each image, after subtracting the pixel values of a blank image [a picture under the condition black (zero pixel values) was projected in whole [13]], is expressed as V(n, x, y, c). Here, x and y represent horizontal and vertical coordinates, respectively; c represents one of the RGB channels; and n represents the index of projection patterns (n = 1–N; in the case 3×3, N = 9). Additionally, the sum of the RGB values for specific x, y, and n values is defined as Vsum(n, x, y). When Vsum(nmax, x, y) is the largest among Vsum(n, x, y) where n = 1–N, V(nmax, x, y, c) is represented as MAX(x, y, c). In contrast, when Vsum(nmin, x, y) is the smallest among Vsum(n, x, y) where n = 1–N, V(nmin, x, y, c) is represented as MIN(x, y, c). It follows that MAX(x, y, c) and MIN(x, y, c) represent reflected light from the illuminated area and that from the location farthest from the illuminated area, respectively. They are hereafter referred to as MAX and MIN images, respectively. Although a sophisticated method was proposed in a previous study [13] to calculate MAX and MIN from pictures using complementary illumination, it was not applicable in this study because the areas of MAX and MIN were not complementary in a single picture.
We assigned MAX to RC and MIN to RD in Section 2.1. The wavelength dependence of the RGB channels was the convolution of the spectrum of the projected light and the spectral sensitivity of each channel of the camera. By definition, concrete range scales for close and distant depend on the sequence; however, when the same sequence is used, the calculation of Eqs. (4) to (6) with the derived RC and RD maintain consistency. Calibration of the white balance and illuminance was applied to composed MAX and MIN images by multiplying red, green, and blue channels independently by constants so that the reflectance of the white chart under the chin rest became 0.9 for all channels, because the reflectance of the white chart was 0.9 in the specification. Because the results include the effect of the interface, RC and RD were transformed with the formulae described in Section 2.5 to reduce it, and then the results were used for the calculation in Section 2.1.
2.5 Reduction of the effect of the air interface
To reduce the effect of the interface, Kubelka’s approach [14] was used (Fig. 5) again. For simplicity, the refractive indices of bare skin and FD were assumed to both be 1.5, and then only the air interface was taken into account. For the close area, surface reflection was removed by polarizers in the system; therefore, the reflection for I0D can be considered to be zero [Fig. 5(b)], which results in the following relationships:
Because acquired images include the effect of the air interface, RC = I0U / I0D. To exclude this interface, RC´ = I1U / I1D should be used instead of RC. The relationship between RC and RC´ can be derived from Eq. (7) as
Fig. 5. Balance of light intensity at the air interface. (a) Flow of light to and from the interface. Elements of reflection and transmission are shown for (b) close area and (c) distant area.
For the distant area, only the lowest order was considered [Fig. 5(c)] as
The contribution of I1D to I1U is difficult to estimate from measurable values; however, rI becomes 0.04 when the reflective index is 1.5 for normal incidence; therefore, the contribution of I1D to I0U is less than 4%. This simplification is considered to be reasonable.
In summary, we used RC´ and RD´ instead of RC and RD. We assumed 1.5 as the refractive index for both bare skin and FD and assumed normal incidence. Thus, 0.96 and 0.04 were used as rI and tI, respectively.
2.6 Illustrative example
Examples of images of liquid FD (LFD) applied to cheek skin are shown in Fig. 6. Figure 6(a) is a single photograph in a sequence; then, from 3 × 3 images, including Fig. 6(a), MAX and MIN images were synthesized as Fig. 6(b) and Fig. 6(c), respectively. The area with LFD applied (center of each image) became brighter than the bare skin area (periphery of the images) in the MAX image and darker in the MIN image, because light was less dispersed in the area with LFD applied.
Fig. 6. Examples of (a) single picture in a sequence; (b) MAX image; (c) MIN image of the right cheek in the case of Sequence III. For clarity, (c) is displayed three times brighter than (a) and (b).
For qualitative understanding of the effect of projected patterns, the acquired MAX images for Sequences I to IV are shown with an image under normal illumination in Fig. 7. LFD was applied to the cheek. Figure 7(d) is the same image as Fig. 6(b). For comparison, applied LFD was rubbed off with a finger after application [white broken circle in Fig. 7(a)]. For normal light [Fig. 7(a)], the difference between the applied area (center of the image) and non-applied area (periphery of the image) is small and difficult to distinguish visually. In the case of MAX images [Fig. 7(b) – (e)], the applied and non-applied areas are distinguishable, and the difference depends on the sequence. The darker the bare skin appears, the more recognizable the unevenness of the applied LFD becomes, which is remarkable in Sequence III.
Fig. 7. Examples of images with normal illumination. (a) normal lighting image and MAX images of sequences: (b) I, (c) II, (d) III, (e) IV. In (b) – (e), the bottom right shows the magnification of an area measuring 7 mm × 7 mm. For comparison, after application, LFD was partly rubbed off [white broken circle in (a)]. For clarity, (b) – (e) are displayed 25% brighter than (a).
Magnifications of MAX images of the sequences are also shown in Fig. 7. A dark lattice-shaped line pattern is seen, especially in Sequence I. The lines in Sequence IV are the least recognizable between sequences. However, the unevenness of LFD application also became the least recognizable.
2.7 Light transmitted via undesirable paths
In the method, we assumed that MAX is the light component whose incident point and exit point are within identical squares, and then MIN is the light component whose exit point is located in the middle of four adjacent squares, one of which is the incident point. However, several paths are conceivable in reality, including indirect illumination from reflection at other surfaces and subsurface scattering from adjacent (for MAX) and non-adjacent (for MIN) incident points. These are undesirable components in the model. The effects of these components were estimated by the patterns shown in Fig. 8 adding to a sequence with normal patterns and were named E1 and E2.
Fig. 8. Patterns for the estimation of components from undesirable paths at a specific location: (a) reflection from the white open square is the undesirable component for MAX, which is named E1; (b) reflection from the red open square is that for MIN, which is named E2. White solid squares were illuminated. Other areas, including white open squares, were not illuminated.
The skin on an arm was measured with and without LFD. The LFD described in Section 2.8 was used, and the application amount was 1.31 mg/cm2. Table 1 shows mean values of MAX, MIN, E1, and E2 at the corresponding areas, and their ratios. Table 2 shows rF(a) and tF(a) calculated from MAX and MIN (“undesirable paths included”), those calculated from (MAX – E1) and (MIN – E2) (“undesirable paths excluded”), and the ratio of the difference between “included” and “excluded” to “excluded” (“error ratio”). E2 / MIN values were larger than E1 / MAX values for LFD applied to skin, while they were comparable for bare skin. The error ratios were significantly small as compared to E1 / MAX and E2 / MIN.
Table 1. MAX, MIN, E1, E2, and their ratios at a certain location on bare skin and skin with applied LFD.
Table 2. rF(a) and tF(a) with and without reduction of components from undesirable paths.
2.8 Cosmetic foundation
LFD and powder FD (PFD) were used for the experiment. As LFD, “est, the glowing cream makeup OC202” (Kao Corp., Tokyo, Japan) was used, and as PFD, “KATE, skin cover filter foundation 02” (Kanebo Cosmetics Inc., Tokyo, Japan) was used. They are average LFD and PFD in terms of their optical properties.
3. Experiment
This study was reviewed and approved by the Human Research Ethics Committee of Kao Corporation. The participants provided written informed consent to participate in this study. Healthy Japanese male (n = 1) and females (n = 8) in their 20s to 50s (Types III and IV on the Fitzpatrick scale [17]) participated.
A region of interest (ROI) measuring 8 cm × 4 cm was set on the inner side of the left forearm of each participant, and each ROI was measured as bare skin. Next, a small amount of LFD was applied uniformly to the ROI and measured. Then, LFD was applied to the ROI again and measured; the application procedure was gradually repeated until no more LFD could be added. By doing so, data for more than one application amount were acquired for each person. The application quantity was calculated as the decrease in the sum of the weights of the FD container and applicator before and after the application, divided by the application area (= 32 cm2). After the series of measurements with LFD was finished, the ROI was cleansed with a makeup remover and facial wash. Then, the same procedure was performed with LFD switched to PFD. For each FD and for each participant, FD applied to skin was measured three times except for one participant, for whom it was measured just two times for PFD. The amount of applied FD was in a range of 1 to 6 mg/cm2 for LFD, and 0.1 to 0.7 mg/cm2 for PFD.
For four participants, whole faces were also measured for a qualitative evaluation of applicability. After a facial wash, each face was measured as bare skin, after which LFD was applied to the whole face and measured. Next, the face was cleansed again and PFD was applied to the face and measured. In the application, a practitioner applied FD to the faces of participants uniformly as far as possible as ordinary makeup.
4. Results
For each FD, plots of MAX [ Fig. 9(a), (b)] and MIN [Fig. 9(c), (d)] are shown for the red channel for Sequence III as examples. The values were derived by averaging the ROI of 300 × 300 pixels (11 × 11 mm in actual size). The average (standard deviation) of MAX of bare skin for all participants was 0.131 (0.017), and that of MIN was 0.071 (0.010), which reflect the plots of zero FD amount in Fig. 9. The values except those for bare skin fall approximately on a straight line in each graph, and the results of a linear regression (excluding bare skin) are displayed. The correlation between MAX and FD amount is positive, that for MIN is negative, and that for LFD is higher than that for PFD.
Fig. 9. Measured values for (a), (b) MAX; (c), (d) MIN for the red channel for Sequence III, against the application amount of (a), (c) LFD; (b), (d) PFD. The regression analyses were performed excluding the data for bare skin (FD amount = 0). Each line represents a regression line, while broken lines represent the 95% confidence interval and dotted lines represent the 95% prediction interval. The same symbol indicates data for the same participant.
Next, from MAX and MIN values, after the reduction of surface reflection described in Section 2.5, rF and tF with different levels of approximation were calculated, and then the values were derived to average the same ROI as those for MAX and MIN. For the red channel for Sequence III, plots of rF(p) [Fig. 10(a), (b)], tF(p) [Fig. 10(c), (d)], rF(r) [Fig. 10(e), (f)], and tF(r) [Fig. 10(g), (h)] are shown for each type of FD. In the same manner as in Fig. 9, the results of linear regression excluding zero application are displayed in Fig. 10. The average (standard deviation) for RSC, RSD, and RSC / RSD for all values was 0.136 (0.017), 0.071 (0.010), and 1.932 (0.113), respectively, the second and the third of which were used in the calculation of tF(r) and rF(r). The corresponding signal-to-noise ratio, the average value divided by the standard deviation, was 7.78, 6.89, and 17.13. The correlation between rF and FD amount was positive, that for tF was negative, and that for LFD was higher than that of PFD. According to the linear regression, rF(r) gives good agreement with rF(p) (Fig. 10); however, the correlation between tF(r) and the applied amount was worse than that for tF(p).
Fig. 10. Estimated values of the red channel for Sequence III: (a), (b) rF(p); (c), (d) tF(p); (e), (f) rF(r); (g), (h) tF(r). (a), (c), (e), (g) LFD; (b), (d), (f), (h) PFD. The regression analyses were made excluding data for bare skin (FD amount = 0). Each solid line represents a regression line, broken lines represent 95% confidence intervals, and dotted lines represent 95% prediction intervals. The same symbol indicates data for the same participant.
For the red channel, correlation coefficients for rF and tF for each level of approximation for each FD and each sequence are shown in Table 3. Because rF(p) and rF(a) were exactly equal up to the third decimal place, the correlation coefficients were also equal and they are listed in the same row. The absolute correlation coefficients for tF, especially those for PFD, were lower than those for rF. With respect to tF(p), correlations for tF(a) were lower, and those for tF(r) worsened markedly. Comparison between sequences showed that Sequence III had the highest correlation. Generally, for rF, the order of correlation strength was III > IV > II > I, and for tF, the order was III > II > I > IV. For all sequences, the strength of the correlation for PFD was lower than that for LFD. For Sequence III, the correlation coefficients for rF and tF for each level of approximation for each FD are shown for channels of green and blue in Table 3. Generally, the order of the correlation strength was R > G > B for rF of Sequence III; however, this comparison was not applied to all parameters.
Table 3. Correlation coefficients for rF and tF for the red channel with Sequences I to IV and for the green and blue channels with Sequence III against the amount of application.
With the data set of Sequence III, the amount of FD application was estimated using the relationship against rF(r). To make the amount zero at rF(r) = 0, the data were fitted with a quadratic function with a zero constant term for LFD [ Fig. 11(a)]. For PFD, the data were fitted with a linear formula [Fig. 11(b)] because the p-value for the second-order term became larger than 0.1. With these results, we examined the prediction of the amount of applied FD from the images of skin with makeup. Representative results for skin with applied LFD and PFD are shown in Fig. 12 with images under normal light and MAX images. The application amounts were 0.85 mg/cm2 for LFD and 0.42 mg/cm2 for PFD. The unevenness of FD is not recognizable in the images under normal light [Fig. 12(a), (b)]; however, it is clearly visible in the MAX images, and is also reflected in the estimated rF(r) values. We can see that the bright areas in Fig. 12(c), (d) coincide with the thick areas in Fig. 12(e), (f).
Fig. 11. (a) Estimation line for LFD, whose specific function is y = 12.241 × x + 41.503 × x2; (b) Estimation line for PFD, whose specific function is y = 5.204 × x. The plots in (a) and (b) are the same as those in Fig. 10(e) and Fig. 10(f), respectively; however, the x- and y-axes are reversed.
5. Discussion
Analysis of photographs with a sequence of projection patterns allowed the rF and tF for the FD layer to be estimated as images following the reflection model. The MAX image represents the less dispersed component under the surface, and MIN represents the more dispersed component. Thus, they responded sensitively to the amount of FD application, which is the key to visualizing the FD distribution clearly. Including the case in which unevenness was intentionally introduced (Fig. 7), in the range of usual makeup, the application condition was well visualized; for example, we could easily identify an area of poorly applied FD [upper right of mouth in Fig. 12(c), (e)] and traces of the applicator [around the right cheek in Fig. 12(d), (f)]. Because the FD color is usually chosen to resemble that of skin, unevenness could not be visualized with such clarity with normal lighting [Fig. 12(a), (b)]. However, the combination of the structured lighting and image synthesis made visualization possible.
The rF and tF for the FD layer were correlated with the amount of FD applied (Fig. 10). Figure 10 also indicates that rF approached zero when the applied amount approached zero, which is reasonable. When we consider that the participants had a certain range of skin color variation, the influence of bare skin seems to be well removed because the plots for all participants are nearly on the same line. Actually, while the MAX and MIN of some participants were obviously lower than the average (e.g., pink circle in Fig. 9), such disparity cannot be seen in rF(p) and tF(p) (Fig. 10). The low-order approximation rF(a) fitted rF(p) almost completely, which means the low-order term in Eq. (1) was dominant. Even at the level of MAX and MIN images, the values had a positive relationship with the application amount (Fig. 9); in particular, MAX is sensitive to the application amount [Fig. 12(c), (d)]. This means that a MAX image itself can be a good visual representation of actual FD application.
Fig. 12. Representative example of visualized application condition for (a), (c), (e) LFD; (b), (d), (f) PFD. (a), (b) images with normal lighting; (c), (d) MAX images; (e), (f) estimation of application amount from rF(r). The units of the color bars in (e) and (f) are mg/cm2.
The rF and tF can be calculated from the images of FD applied to skin and images of the corresponding point on bare skin; however, it is usually difficult to attain a point-by-point correspondence. If the parameters could be derived only from images of FD applied to skin without those of bare skin, this would be highly beneficial in practical terms. We found that using the average value of bare skin for rF(r) gives a good approximation. In Eq. (6), the measured value for bare skin appears in the form RSC / RSD. Each RSC and RSD value for bare skin showed a certain degree of variation among individuals; however, the variation decreased when the ratio was used. In the case of Sequence III, the averaged values divided by the standard deviations were 7.78, 6.89, and 17.13 for RSC, RSD, and RSC / RSD, respectively. This indicates that the errors in RSC and RSD for skin color were synchronized and that the error in the ratio converged to a lower value. In contrast, for tF(r), the correlation decreased. This is because RSD is used alone as the value for bare skin instead of a ratio; therefore, the variation of RSD among individuals directly reflected the error in tF(r). The use of representative values of RSC / RSD and RSD for each person and each site separately may provide an effective method of improvement.
The patterns for projection affected the quality of synthesized images and the strength of correlation between rF and the applied amount. If the illuminated area between patterns did not overlap [Sequence I in Fig. 7(b)], the pixel values near the border of each illuminated square decreased compared to those at the center of the square [Fig. 7(a)], which resulted in reduced image quality. Then the error propagated to the map of rF. In contrast, if the overlap was too large [Sequence IV in Fig. 7(e)], the sensitivity against translucency became too small, which then reduced the information about the condition of applied FD. Among the sequences we used, Sequence III shows the highest correlation (Table 3), which means the discrimination between close and distant in Sequence III was the most successful among all the sequences. The precision of the result is affected by the balance between light spreading under the skin surface and the size of the projected pattern. We should also consider that both the MAX and MIN images are convolutions over a certain range of distances between the incident point and the exit point, and this could be a factor that diminishes the resolution. In the system we used, the resolution of the projector was the bottleneck for optimization of the resolution, which imposes a limitation on available patterns.
If the accuracy is positively correlated with the difference in lateral spread under the surface between bare skin and skin with FD applied, the magnitude relation for the correlation coefficient for R/G/B channels should be R > G > B. This is because the scattering power of skin decreases as the wavelength increases [18], meaning a greater amount of light spreads beneath the skin surface. The behavior of rF for Sequence III may be explained from that perspective (Table 3); however, other sequences and for tF did not follow it in some cases. The variation between the translucency of bare skin and made-up skin is not the only element for defining the correlation; for example, the balance between the size of a square and the distance between adjacent squares may have some impact. As another factor, hemoglobin concentration oscillations due to the effect of pulsatile blood circulation could produce a wavelength dependence with a cycle length of about 1 s [19]; therefore, blood circulation could be a cause of skin color fluctuation during a measurement.
In the experiment, the upper bound of the application amount was extended as far as possible, but the lower bound depended on the feasibility of uniform application and reliable determination of the application amount, which should be considered. In the experiment, the range of application amount for the face was lower than the range used for the arm for LFD, while the face and arm application amount was largely the same for PFD. For example, for the participant shown in Fig. 12, the application amount on the face was about 0.85 mg/cm2 for LFD and 0.42 mg/cm2 for PFD. Because the face applications were done as ordinary makeup, presumably, the real-use dose on average is lower than the range in the experiment for LFD, while being generally the same for PFD. It is difficult to compare average application amounts with those in other studies; nonetheless, for example, one experiment [20] used 0.7 mg/cm2 for LFD and 0.5 mg/cm2 for PFD as real-use doses, which support the estimation in the experiment. When we observe the plots, including the points of zero application (bare skin), rF versus application amount resulted in curves for LFD, rather than straight lines, which may reflect non-linear effects. However, from the consideration mentioned above, the relationship for LFD may be close to linear under the range of actual usage. Although the correlation is higher for LFD than for PFD in the experiment, for the same reason, the correlation may not differ so much between LFD and PFD for the range of actual use.
In an optical system, the effect of nonuniform light intensity and the presence of shadows (e.g., of a nose) are problems. As can be seen from Eq. (6), the estimation error for illuminance proportionally reflects the estimated rF. If an area has an illumination that is stronger than the estimate, the rF estimate will be larger. This does not only depend on the performance of the light projector, but also the shape of the subject, and the variation in distance between the projector and each point on the subject. As is evident, non-illuminated areas cannot be analyzed. To address this issue, it is preferable to form a collinear arrangement by minimizing the angle from a subject to a projector and camera. The collinear arrangement is effective for minimizing non-illuminated areas viewed from the camera, and for minimizing the range of distance from the light source to the area of interest. In Fig. 12(e), the amount of LFD is more than 4 mg/cm2 for the right cheek, which is much thicker than the expected range and is probably due to the difference in illuminance (because of the difference in distance from the projector).
In the model, we assumed that the MAX signal is due to light directly irradiating a square and exiting from the same square, and the MIN signal is due to light irradiating one of four adjacent squares and exiting from their center after subsurface scattering. However, other pathways also are possible. As briefly discussed in Section 2.7, for bare skin, about one-third was due to components from other paths for both MAX and MIN, which seemingly were not small; however, the effect on rF and tF was extremely small in comparison. This is probably because the values of MAX and MIN move synchronously with the error, which cancels the variations in the calculation of rF and tF. Here, for the estimation, we used a forearm, which is smooth and flat; therefore, indirect irradiation through other skin may be suppressed. In the case of a face, which has more contours than a forearm, the amount of irradiation due to reflection from other areas could be larger, thereby increasing the error.
One of the advantages of the model is that prior information about skin optical characteristics is not needed except for the refractive index and existence of a certain degree of translucency, even though skin is known to have complex structure [21]. The parameters for skin were obtainable from acquired images and removable to extract the optical characteristics of FD in the model. The estimation error of the refractive index will affect rF and tF. Generally, multiplication by a certain amount, when the refractive indices of skin and FD are the same, will have the same effect on the degree of unevenness. However, if there are differences between the indices of skin and FD, the treatment of the interfaces should be reconsidered. Here, from the near field view, the refractive index at the interface was defined by the area approximately shallower than the wavelength of light (<1 µm), which can be recognized as “surface.” The refractive index of the skin surface can be regarded as constant against the wavelength in the visual light range, as shown in a study for porcine skin [22]; however, that of FD can be varied against the wavelength in general. As another aspect of skin that was not considered in the model but could affect the results, we ignored the surface roughness of skin, which changes the incident angle of incoming and emitted light at the interface.
In the experiment, the skin types of the subjects were III and IV on the Fitzpatrick scale [17]; however, the applicability of our model should be reconsidered for other skin types. Especially, for darker skin, dispersion under the skin surface is smaller, and the difference between the values of FD and bare skin will be smaller. In such cases, discriminating between FD and skin will become more difficult. Additionally, in the experiment, RSC / RSD differed little between persons, but this should be checked for darker skin types.
6. Conclusion
We succeeded in visualizing the optical properties of an FD layer and makeup finishing using the difference in scattering power between bare skin and FD applied to skin. In addition, we proposed approximation formulae, in which the values can be derived only for FD applied to skin without measurement of corresponding bare skin, for ease of application of the method. It will be a useful tool for the development of cosmetic products and testing procedures. Moreover, the concept may be applicable for the examination of a thin film with high scattering power on translucent material, where the condition of the thin film is of concern, such as in the field of surface modification.
Acknowledgment
We wish to express our gratitude to Ken Nishino for valuable advice, in particular for the average application amount of cosmetic foundation.
Disclosures
KY, NO, TI: Kao Corporation (E, P).
Data availability
The data used to obtain the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. R. Korichi and J.-F. Tranchant, “Decorative Products,” in Hanbook of Cosmetic Science and Technology, A. O. Barel, M. Paye, and H. I. Maibach, eds., 4th (CRC, 2014), pp. 439–451.
2. S. Y. Lee, J. H. Baek, M. K. Shin, and J. S. Koh, “The quantitative analysis of spreadability, coverage, and adhesion effect after application of the base make-up product,” Skin Res Technol 20(3), 341–346 (2014). [CrossRef]
3. Y. Yan, J. Lee, J. Hong, and H. Suk, “Measuring and describing the discoloration of liquid foundation,” Color Res. Appl. col. 46(4), 362–375 (2021). [CrossRef]
4. T. Nagaoka and Y. Kimura, “Quantitative cosmetic evaluation of long-lasting foundation using multispectral imaging,” Skin Res Technol 25(3), 318–324 (2019). [CrossRef]
5. K. Nishino, M. Nakamura, M. Matsumoto, O. Tanno, and S. Nakauchi, “Optical filter for highlighting spectral features part I: design and development of the filter for discrimination of human skin with and without an application of cosmetic foundation,” Opt. Express 19(7), 6020 (2011). [CrossRef]
6. K. Nishino, M. Nakamura, M. Matsumoto, O. Tanno, and S. Nakauchi, “Optical filter highlighting spectral features part II: quantitative measurements of cosmetic foundation and assessment of their spatial distributions under realistic facial conditions,” Opt. Express 19(7), 6031 (2011). [CrossRef]
7. M. Doi, R. Ohtsuki, and S. Tominaga, “Spectral estimation of made-up skin color under various conditions,” Proc. SPIE 6062, 606204 (2006). [CrossRef]
8. R. Ohtsuki, S. Tominaga, and R. Hikima, “Appearance analysis of human skin with cosmetic foundation,” Proc. SPIE 8292, 82920Q (2012). [CrossRef]
9. K. Yoshida and N. Okiyama, “Estimation of reflectance, transmittance, and absorbance of cosmetic foundation layer on skin using translucency of skin,” Opt. Express 29(24), 40038 (2021). [CrossRef]
10. T. Okamoto, T. Kumagawa, M. Motoda, T. Igarashi, and K. Nakao, “Monte Carlo simulation of light reflection from cosmetic powder particles near the human skin surface,” J. Biomed. Opt. 18(6), 061232 (2013). [CrossRef]
11. S. Gioux, A. Mazhar, and D. J. Cuccia, “Spatial frequency domain imaging in 2019: principles, applications, and perspectives,” J. Biomed. Opt. 24(07), 1 (2019). [CrossRef]
12. J. P. Angelo, S.-J. Chen, M. Ochoa, U. Sunar, S. Gioux, and X. Intes, “Review of structured light in diffuse optical imaging,” J. Biomed. Opt. 24(07), 1 (2018). [CrossRef]
13. S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar, “Fast separation of direct and global components of a scene using high frequency illumination,” ACM Trans. Graph. 25(3), 935–944 (2006). [CrossRef]
14. P. Kubelka, “New Contributions to the Optics of Intensely Light-Scattering Materials Part I,” J. Opt. Soc. Am. 38(5), 448 (1948). [CrossRef]
15. S. Jacques, “Video reflectometry to specify optical properties of tissue in vivo,” Proc. SPIE 10311, 103110D (1993). [CrossRef]
16. K. Nishino, A. Subpa-asa, Y. Asano, M. Shimano, and I. Sato, “Variable Ring Light Imaging: Capturing Transient Subsurface Scattering with an Ordinary Camera,” European Conference on Computer Vision (2018), pp. 624–639.
17. T. B. Fitzpatrick, “The Validity and Practicality of Sun-Reactive Skin Types I Through VI,” Arch. Dermatol. 124(6), 869–871 (1988). [CrossRef]
18. S. L. Jacques, “Optical properties of biological tissues: a review,” Phys. Med. Biol. 58(11), R37–61 (2013). [CrossRef]
19. K. Yoshida and I. Nishidate, “Phase Velocity of Facial Blood Volume Oscillation at a Frequency of 0.1 Hz,” Front. Physiol. 12, 627354 (2021). [CrossRef]
20. M. A. Kim, Y. C. Jung, J. Bae, J. Ha, and E. Kim, “Layering sunscreen with facial makeup enhances its sun protection factor under real-use conditions,” Skin Res Technol 27(5), 751–757 (2021). [CrossRef]
21. R. R. Anderson and J. A. Parrish, “The optics of human skin,” J. Invest. Dermatol. 77(1), 13–19 (1981). [CrossRef]
22. H. Ding, J. Q. Lu, K. M. Jacobs, and X.-H. Hu, “Determination of refractive indices of porcine skin tissues and Intralipid at eight wavelengths between 325 and 1557 nm,” J. Opt. Soc. Am. A 22(6), 1151–1157 (2005). [CrossRef]
