1. Introduction

Skin colors and spectra reflect the concentrations of pigments and the scattering property and provide important information in diagnosing dermatological conditions [1–7]. In addition, skin color is important from an aesthetic viewpoint because skin color is directly associated with appearance [8,9]. The field of computer graphics also uses skin color to make photorealistic images [10–13]. One concern is the relationship between chromophore concentrations and skin color. Therefore, even limiting the conversation to point measurement, many studies have been conducted [7,14–19]. The basic concept of most of these studies was to use the difference between the absorption spectrum of each chromophore. Mostly, reflectance spectra measurement by a spectral reflectometer was used [7,17–19], or, more simply, only a few wavelengths were measured, from which the chromophore concentrations were estimated [14–16].

Since the surface distributions of chromophores are also important for diagnosing skin condition and appearance, many attempts have been made to obtain this information [1–6,20]. Using multispectral or hyperspectral imaging enables a simple extension of single-point measurement with spectral analysis to chromophore imaging [5]. A disadvantage of multispectral and hyperspectral imaging is the high cost of the apparatus and the large quantities of data that are required. In order to avoid these issues, many studies have been conducted using common red/green/blue (RGB) cameras [3,6,12,13,20,21], which reduces both cost and data size. The advantages of common RGB cameras are the short acquisition time (just one shot in the case of a still image) and the availability of common photographic equipment and software. On the other hand, a disadvantage of common RGB cameras is that only three values are acquired for one point, and the degree of freedom is small. Therefore, we should use limited information to achieve reliable estimation.

With respect to analysis, many studies are based on the modified Lambert-Beer law [14,18,19], which is an extension of the Lambert-Beer law. The modified Lambert-Beer law is used for light reflection from an opaque material, whereas the original Lambert-Beer law is used for light transmission. In the method, the logarithm of the inverse of the RGB pixel values is assumed to be the linear sum of vectors corresponding to respective chromophores. The coefficient of the vectors can be associated with the respective chromophore concentration. The key point of the method is how to obtain the vectors corresponding to the respective chromophores. Some studies have derived vectors with the absorption coefficient spectrum and the spectral sensitivity of a camera [3] or with an experimental approach using a phantom [19]. These methods are sensible but require effort and time. Therefore, application to a new photographic instrument is troublesome. In addition, disagreement between the conditions of the models and those of the actual skin, including the lighting conditions, can cause systematic errors and artifacts [6].

In order to avoid these problems, image-based methods [11–13], in which parameters are acquired from images, have often been used. With an image-based method, using actual photographs taken from a certain photography system, parameters are determined in order to make chromophore images the most reasonable. If we set the degree of artifact as an evaluation function in a method, the degree of artifact can be minimized. In general, image-based methods are easier to apply than other methods because assumptions of the scattering coefficient, the thickness of each layer, and other properties to determine light propagation are not usually required. As an image-based method, a method with independent component analysis (ICA) has been proposed [12,13]. The method assumed the independence of the melanin distribution and the hemoglobin distribution and decided the vectors so as to maximize the independence of the resulting melanin and hemoglobin patterns of the test images. As a problem, for some frequency, the vectors converge far from appropriate values with certain test images, and such values should be removed manually in vector estimation. Although the method is very useful for a certain purpose, the criterion of elimination is ambiguous and makes the method problematic from the viewpoint of objectivity.

In order to deal with the problem, we developed an optimization method of chromophore vectors in which vectors are set to specific values in order to minimize artifacts from other chromophores. In the proposed method, by preparing test images following a prearranged procedure, the vectors can be adjusted stably, focusing on the angles made by vectors. In addition, the protocol contains the optimization of shading images, which makes use of the shading images possible. In the present paper, we first validated the performance of each step of the proposed protocol and then compared the proposed protocol with the method with ICA. Next, we evaluated the validity of the proposed protocol as an estimation method of chromophore concentration, as compared with the results obtained by the estimated formulae, which were described in a previous paper [17].

2. Materials and method

2.1 Modified Lambert-Beer law in imaging [12]

In pictures of skin, we assume that the surface reflection is reduced, and that the RGB values of a white object (e.g., white chart) whose reflectance spectrum is constant in visible wavelength are equal to each other; if not, normalize the RGB channels of the white object in the pictures to satisfy the condition. First, we consider the ${\left( {\begin{array}{{ccc}} r&g&b \end{array}} \right)^T}$ (T represents the transpose) values derived from the following relationship:

In the modified Lambert-Beer law, an approximation was introduced whereby the path length in the layer of each chromophore in skin can be regarded as constant for each wavelength. The specific value of the length cannot be determined without additional assumptions, and the product of the path length and the absorbance is measureable. In our skin model, we assumed that only melanin and hemoglobin are the absorbers, as was the case in many studies [3,6,12,13] and that the attenuation of light can be explained by their linear sum. In the case of a large region of interest (ROI), e.g., an entire face, the illuminance of each point in the ROI is difficult to determine. However, the uncertainty can be included in the model by treating the model as a virtual chromophore, which is called shading in the following [12]. In the model, the following formula holds [12]:

2.2 Pattern mixing in chromophore images from the variations of vectors

Assuming each vector has been set to the correct value and each component is separated perfectly from a picture with the vectors, we determine what happens if each of the vectors moves. With vectorial representation, ${\left( {\begin{array}{{ccc}} M&H&S \end{array}} \right)^T}$ are derived using the following formula, which is equivalent to Eq. (3):

For conciseness, let us describe the plane that contains both $\overrightarrow a $ and $\overrightarrow b $ as $\overline {ab} $. In Eq. (4), each of the denominators is a scalar constant, irrespective of $\overrightarrow p $. Therefore, it acts uniformly across the entire image and does not change the pattern. The cross product in the numerator does not affect the pattern if its direction is not changed. Only in the case in which the direction of the cross product in the numerator is changed will the pattern of the chromophore image change. In particular, if only $\overrightarrow m $ is changed to $\overrightarrow {m^{\prime}} $, the pattern of M will not change, although the intensity of the entire image may be changed. If $\overrightarrow {m^{\prime}} $ is on $\overline {ms} $, then the normals of $\overline {ms} $ and $\overline {m^{\prime}s} $ are identifiable. Therefore, the pattern of H will not change either, while the pattern of S will change because the normals of $\overline {mh} $ and $\overline {m^{\prime}h} $ will become different in this case. Considering the difference among the $\overrightarrow m $’s on the fixed plane $\overline {ms} $ results in the difference in the manner of splitting components on the $\overline {ms} $, the change in the pattern of S arises from the mixing of M. In the same manner, if $\overrightarrow {m^{\prime}} $ is on $\overline {mh} $, then the pattern of M and S will not change, but the pattern of H will change due to the mixing of M. The same discussion holds when $\overrightarrow h $ moves.

Conversely, if the pattern of M is mixed into S, $\overrightarrow m $ should be moved so that the angle between $\overrightarrow s $ and $\overrightarrow m $ changes on the plane $\overline {ms} $, and if the pattern of M is mixed into H, then $\overrightarrow m $ should be moved so that the angle between $\overrightarrow h $ and $\overrightarrow m $ changes on $\overline {mh} $. The same discussion holds in the case of mixing H into M and S.

2.3 Successive minimization of intermixture between vectors (SMI)

From the observation in the previous section, we developed the following sequence to optimize $\overrightarrow m $ and $\overrightarrow h $. First, we prepared tentative vectors. Then, under the spherical-coordinate system in which the direction of $\overrightarrow s $ is set as a pole, step A: move $\overrightarrow m $ latitudinally and search for a point to minimize the mixing of M to S; step B: move $\overrightarrow m $ longitudinally and search for a point to minimize the mixing of M to H, and then exchange the roles of $\overrightarrow m $ and $\overrightarrow h $, step C: move $\overrightarrow h $ latitudinally and search for a point to minimize the mixing of H to S; and step D: move $\overrightarrow h $ longitudinally and search for a point to minimize the mixing of H to M (Table 1, Fig. 1). Although steps B and D do not strictly move on $\overline {mh} $, these steps assure the orthogonal move to steps A and C, which makes the optimization more stable. In order to minimize the mixing, the Nelder-Mead simplex method was used, and the square of the correlation coefficient between the two components was used as the evaluation function to be minimized. The algorithm was implemented as a plug-in of ImageJ (open source software; National Institute of Mental Health, Bethesda, MD, USA) with Java as a programing language and Apache Commons Math (open source software; Apache Software Foundation) as a library to implement the Nelder-Mead simplex method.

 

Fig. 1. (a): Schematic view of procedure of SMI, and (b): visualization of the direction of adjustment. The symbol O in (b) represents the origin. The symbols “a” to “d” in (a) and (b) correspond to each other. The image (b) does not reflect the actual angles between the vectors.

Download Full Size | PDF

Table 1. Summary of the characteristics of each step in the sequence.

View Table | View all tables in this article

In order to demonstrate the effect of each step, we initially prepared an appropriate $\overrightarrow h $ and insufficient $\overrightarrow m $, which intentionally makes components mixed, and then derived M, H, and S from a sample image (Fig. 2a) using these values (Condition I; Fig. 2b-1d). Next, $\overrightarrow m $ was optimized by step A from the initial vectors, and M, H, and S were recalculated (Condition II; Fig. 2e-1 g). Then, the optimization was performed by step B from the initial vectors, and M, H, and S were recalculated (Condition III; Fig. 2h-1j). With the initial $\overrightarrow m $, a pattern of melanin spots (Fig. 2b) appeared in H (Fig. 2c) and S (Fig. 2d). By step A, the reflection of M in S has been decreased (from Fig. 2d to Fig. 2 g), and, by step B, that of M in H has been decreased (from Fig. 2c to Fig. 2i). By step B, the pattern of S has been improved (from Fig. 2d to Fig. 2j), which can be explained by the fact that $\overline {mh} $ also changed in the longitudinal move of $\overrightarrow m $ collaterally. Although the pattern of M can be still observed in Fig. 2i, this pattern will disappear by iteration of the entire sequence. The same holds for the case in which the roles of $\overrightarrow m $ and $\overrightarrow h $ are reversed. The actual impact for the improvement of the patterns depends on the adequacy of the initial vectors.

 

Fig. 2. Demonstration of the effects of steps A and B. (a): Original RGB image. Under condition I: derived images of (b) melanin, (c) hemoglobin, and (d) shading. Under condition II: derived images of (e) melanin, (f) hemoglobin, and (g) shading. Under condition III: derived images of (h) melanin, (i) hemoglobin, and (j) shading. In condition I, the initial vectors (vectors determined to display the artifacts) are used as the vectors. In condition II, the vectors are optimized with step A. In condition III, the vectors are optimized with step B. For visibility, the pixel values of the results have been multiplied by (b) 0.75, (c) 3.0, (e) 0.5, (f) 3.0, and (h) 0.75.

Download Full Size | PDF

In the optimization, different sets of test images should be prepared for the melanin vector and the hemoglobin vector because the variation of the focused chromophore should be large and the variation of the other chromophores should be small for high accuracy. In this case, steps A and B should be executed with melanin test images of large melanin variation, and steps C and D should be executed with hemoglobin test images of large hemoglobin variation. The vectors may be converged without the use of different sets, but this is preferred for applicability. In terms of lighting condition, bright (i.e., well-lit, facing the camera) areas are preferable as test images. Although the effect of shading is well eliminated with the method, the less the signal from a chromophore is, the smaller the signal to noise ratio becomes; therefore, the convergence of the vectors can worsen.

Conducting each step sequentially, $\overrightarrow m $ and $\overrightarrow h $ will be optimized for all directions. Although the sequence requires arbitrary initial $\overrightarrow m $ and $\overrightarrow h $, by repeating steps A through D several times, $\overrightarrow m $ and $\overrightarrow h $ will converge. The series of steps from step A to step D was repeated five times in order to make the vectors converge in the experiment. The number of repetitions is determined in order that the distance between the estimated values just before the series and estimated values (root mean square (RMS) of the difference of r/g/b) becomes smaller than 0.5×10⁻⁵, which is comparable to the quantizing error of pixel values of the 14-bit images we used. The exchangeability of those steps is discussed later.

2.4 Independent component analysis

We examined the method with ICA [12,13] as a conventional method for vector determination in order to compare the results with SMI. Independent component analysis was applied to a skin image assuming that the histograms of M and H are not Gaussian distributions and assuming that M and H are independent of each other. Although the plane of $\overline {mh} $ can be chosen arbitrarily, $\overline {mh} $ was determined with a small ROI of skin for which the shading can be recognized as constant and was around the ROI of the test image [13]. In the small ROI, ${\left( {\begin{array}{{ccc}} r&g&b \end{array}} \right)^T}$ is distributed on one plane that is parallel to $\overline {mh} $ because S is constant. In particular, principal component analysis is applied to ${\left( {\begin{array}{{ccc}} r&g&b \end{array}} \right)^T}$ of the small area, and the plane that includes both the first and second components is recognized as $\overline {mh} $[13]. Independent component analysis was applied to the projection of the estimated $\overline {mh} $. For the calculation, MATLAB (Mathworks, Natick, MA, USA) was used as a platform, and fastica [22] was used for ICA implementation.

With ICA, two vectors are obtained with ambiguity of the sign of elements (direction of a vector) and that of choice, which is $\overrightarrow m $ and $\overrightarrow h $. Therefore, if the elements of a vector are negative, the signs of all elements were flipped to be positive. In addition, if $r\;<\;g\;<\;b$, we consider the vector as $\overrightarrow m $, and, if $r\;<\;g$ and $g\;>\;b$, we consider the vector as $\overrightarrow h $, which is the use of prior knowledge. We used these restraint conditions just because no other simple rule occurred to us; however, the results were feasible as far as we could determine by our examination.

2.5 Rejection criteria

In SMI and ICA, two rejection criteria were set. First, if any elements of vectors were negative, the pair of $\overrightarrow m $ and $\overrightarrow h $ was rejected (rejection criterion 1), since a negative value means that the larger the chromophore concentration becomes, the larger the reflection becomes, which is irrational. Second, if the magnitude relations of elements were not such that $r\;<\;g\;<\;b$ for one vector, and $r\;<\;g$ and $g\;>\;b$ for the other vector, then the pair of $\overrightarrow m $ and $\overrightarrow h $ was rejected (rejection criterion 2). The appropriateness of the magnitude relationship of $\overrightarrow h $ can be explained by the fact that the pixel values of G are smaller than those of R and B (more specifically, R>B>G) in photographs of blood under moderate (not colored) lighting. This indicates that the convolutions of the spectral sensitivity of a camera against the absorption spectrum of hemoglobin are in reverse order; therefore, the elements of $\overrightarrow h $ in rgb become in the reverse order of those of RGB (r<b<g). The criterion can be relaxed; however, more values need to be rejected by subjective judgement or the derived vectors will become inappropriate.

2.6 Photographing and preprocessing

We used the following apparatus to acquire photographs of a whole face (Fig. 3). A face is set on a chin rest and illuminated by light. As the light source, fluorescent light tubes (FL20S·N-EDL-NU; HITACHI, Tokyo, Japan) were arranged in a square shape (side length: 80 cm), where three tubes were aligned on each edge, and twelve tubes were used in total. The light square faced the chin rest. A camera (D500; Nikon, Tokyo, Japan) with a lens (AF-S Micro Nikkor 105 mm 1:28G ED; Nikon, Japan) was placed on the line from the center of the chin rest to the center of the fluorescent light tubes. The distance between the center of the light source and the chin rest was 70 cm, and the distance between the chin rest and the camera was 100 cm. Polarization plates were placed between the light source and the chin rest and between the camera and the chin rest. The plate of the light source side is transparent to vertically polarized light and that of the camera side is transparent to horizontally polarized light. In order to reduce external light, the apparatus was closed off from the environment with boards. The internal surface of the apparatus was covered with black cloth. A white chart (ColorChecker; X-rite, Grand Rapids, MI, USA) was placed on the chin rest to correct the fluctuation of light intensity and white balance of photographs.

 

Fig. 3. Schematic diagram of photographic instrument

Download Full Size | PDF

The acquired images were saved as raw data (the filename extension is .nef) once and converted into linear images, the pixel values of which are proportional to the light intensity of the respective wavelength. The image size of the whole-face picture was 5,599×3,728 pixels (approximately 31 pixels/mm for one subject) at this point. The pixel values of R, G, and B were multiplied individually, so that the respective values in the white chart all become a certain value, which adjusts the brightness and white balance simultaneously. Actually, the color of white is relative and depends on characteristics of a camera and light in usual photography. However, in the analysis, the important thing is just to make pictures from the same photo booth comparable; therefore, as far as the RGB values of a specific material (white chart) at a specific position in a scene coincide, the discussions in the previous sections apply without change. The resolution of images was reduced by one fourth to 1,399×932 (approximately 7.8 pixels/mm for one subject); then the reduced images were used as the analysis object.

2.7 Estimation of chromophore concentration with multiple regression analysis aided by Monte Carlo simulation (spectrum analysis)

As a non-image-based method, multiple regression analysis aided by Monte Carlo simulation [17] was compared with SMI. The reflectance spectra of the skin subject were measured using a spectrophotometer (CM-2600d, Konica Minolta, Tokyo, Japan). Using the reflectance spectra from 500 nm to 600 nm at 10-nm intervals, chromophore concentrations were estimated following the reference [17]. The use of this spectral range is reasonable because there is a characteristic absorption peak of hemoglobin at around 550 nm which is useful for discriminating between hemoglobin and melanin. Briefly, first, using the logarithms of the inverse of the reflectance spectrum as objective variables, and using absorption spectra of melanin, oxygenated hemoglobin, and deoxygenated hemoglobin as explanatory variables, multiple linear regression analysis was performed. Next, expressing the multiple regression coefficients as ${a_m}$ (melanin), ${a_{oh}}$ (oxygenated hemoglobin), ${a_{dh}}$ (deoxygenated hemoglobin), ${a_0}$ (constant term), and ${a_{th}} \equiv {a_{oh}} + {a_{dh}}$, melanin density ${C_m}$ and hemoglobin density ${C_{th}}$ were derived using cubic formulae of ${a_m}$, ${a_{th}}$, and ${a_0}$. Concrete expressions were defined based on a phantom study [17], and the formulae were described in [17] explicitly as:

3. Experiments

Photographs of whole faces of adult Japanese females (n = 69; Types III and IV of the Fitzpatrick scale) in their 30s (n = 39) and 50s (n = 40) were taken, and the reflectance spectra of the left cheeks of these subjects were measured.

Next, the chromophore vectors were estimated by SMI with test images. From the original images, rectangular areas that contain typical pigmented spots (Fig. 4b, ROI 4) were selected and trimmed by an expert evaluator who is accustomed to evaluating skin condition from photographs and were used as test images for the estimation of the melanin vector. Pictures of whole faces with transient redness after pressing the push button of a ballpoint pen on the cheeks of the face for some time were taken, and rectangular areas including the redness (Fig. 4a, ROI 1) were then trimmed and used as test images for the estimation of the hemoglobin vector. The resulted melanin images and hemoglobin images were visually evaluated by expert evaluators. For each vector, ten trimmed images were prepared. The sizes of the test images were arbitrary but were defined such that melanin concentrations vary sufficiently within the images for the melanin vector and hemoglobin concentrations vary sufficiently within the images for the hemoglobin vector.

 

Fig. 4. Examples of pictures (trimmed) and ROI of test images. A representative example of (a) transient redness and (b) a melanin spot. The ROIs are displayed with numbers that represent 1) a test image for the hemoglobin vector in SMI, 2) a test image for ICA, 3) an image for shading removal in ICA, and 4) a test image for the melanin vector in SMI.

Download Full Size | PDF

The chromophore vectors were also estimated by ICA. The same images of transient redness for SMI were used as the test images for ICA because keeping the hemoglobin variation large is crucial. However, the selected ROIs were different from those for SMI because the ROI for SMI was too small for ICA to acquire appropriate vectors. The image sizes of all test images were 100×100 pixels (approximately 12 mm × 12 mm for a subject; Fig. 4a, ROI 2). From the neighborhood of the test images, 25×25-pixel regions (approximately 3 mm × 3 mm; Fig. 4a, ROI 3), the shading of which can be regarded as approximately constant, were selected and used as images for shading removal.

From the images of whole faces, melanin, hemoglobin, and shading images were obtained with vectors derived by SMI and those derived by ICA. The average values within the 150×150-pixel (19 mm × 19 mm) area of the left cheek were derived and used as the values of melanin and hemoglobin for each subject.

On the other hand, melanin and hemoglobin concentrations were estimated from the reflectance spectra of the left cheeks with the method in Sec. 2.7.

4. Results

By SMI, vectors were estimated for each test image, and then the average and standard deviation (SD) of melanin vectors $\overrightarrow m $ and hemoglobin vectors $\overrightarrow h $ were calculated as shown in Table 2. To evaluate the interchangeability between optimization steps, the result with a different optimization sequence was also examined, one in which longitudinal optimization was done prior to latitudinal optimization (Table 2). Here, note that the numbers do not represent the RGB values themselves, nor are they proportional to them, but rather they are the values in the space converted from RGB by Eq. (1). The SDs of $\overrightarrow m $ were in the range of 0.01 to 0.02, and the SDs of $\overrightarrow h $ were approximately three times those of $\overrightarrow m $ (Table 2). For each series of steps A through D, the distance between the initial (estimated values just before the series) and estimated values (RMS of the difference of r/g/b) were the first: 3.8×10⁻², second: 1.9×10⁻³, third: 8.3×10⁻⁴, fourth: 1.3×10⁻⁴, and fifth: 1.8×10⁻⁵, for $\overrightarrow m $. Those for $\overrightarrow h $ were first: 9.6×10⁻², second: 1.0×10⁻², third: 1.1×10⁻³, fourth: 2.0×10⁻⁴, and fifth: 1.2×10⁻⁵. No result was rejected for rejection criteria 1 and 2.

Table 2. Averages and standard deviations (in parentheses) of vector values derived from test images by SMI with normal order (A→B→C→D), interchanged order (B→A→D→C), and ICA, and the RMS of SDs of r/g/b.

View Table | View all tables in this article

By ICA, vectors from three of the ten test images were rejected according to rejection criterion 1, and two out of seven (ten minus three) test images were rejected according to rejection criterion 2. The averages and SDs were derived from the vectors from the remaining five images (Table 2). Regarding the average, $\overrightarrow m $ from ICA was relatively close to that of SMI. However, the discrepancy between $\overrightarrow h $ from ICA and that from SMI was large, especially in the r element (Table 2). The RMSs of the SDs of ICA were larger than those of SMI. The SDs of $\overrightarrow m $ were larger than those of $\overrightarrow h $ in ICA, whereas the opposite trend is exhibited for SMI.

Representative examples of the chromophore concentrations derived with SMI and with ICA are shown in Fig. 5. The expert evaluators confirmed visually that the images of melanin and hemoglobin with SMI and with ICA were reasonable. With respect to M and H, the patterns were visually much the same, except for a certain factor for whole images. On the other hand, the pattern of S obtained with ICA contains the pattern of H, while that obtained with SMI does not contain the pattern of H and is smooth.

 

Fig. 5. Representative example of (a) the original image and (b) images of components M, (c) H, and (d) S calculated with the vectors derived by SMI, and (e) M, (f) H, and (g) S obtained by ICA.

Download Full Size | PDF

The M and H for SMI were plotted with respect to those estimated from reflectance spectra (Fig. 6). With respect to M, the relationship was close to proportional (y-intercept was close to zero), and the correlation coefficient was large. However, with respect to H, the correlation was worse, and the regression line did not pass near the origin. Performing these calculations for the M and H for ICA gave essentially the same results as for SMI. For ICA, the regression formulas for M and H were ${C_m} = 3.80M + 0.39$ and ${C_{th}} = 0.57M + 0.35$, respectively, and the correlation coefficients were 0.862 for M and 0.724 for H.

5. Discussion

With SMI, by the use of images that were prepared following the structured routine, the vectors were derived with high stability and repeatability (Table 2). On the other hand, with ICA, five out of ten test images resulted in inappropriate vectors, such that the signs of the elements were not the same and the dispersions of the remaining elements were larger than that from SMI (Table 2). The assumption of ICA may be reasonable. However, statistical error is inevitable and probably resulted from the instability and weak convergence associated with this application. Since the process only needs to be performed at the beginning of the use of the photographic instrument, more appropriate vectors can be acquired than in the results of this experiment with adequate data and consideration. However, this requires time and effort, and prompt action is difficult for newly introduced instruments, which is practically important. In addition, a subjective judgment is unavoidable in choosing reasonable results, and, therefore, it is difficult to evaluate the accuracy and certainty of the derived vectors.

The convergence speed was fast enough so that the difference between the initial value and the result of one series (steps A through D) became smaller than the variation between test images after two or three repetitions, starting from decent values. The stepwise optimization of melanin/hemoglobin for latitudinal/longitudinal movement of vectors makes the convergence stable. As a preliminary study, we tested the optimization of vectors for two chromophores and two directions at once. However, the convergence became less stable. The successive optimization appears to be the key point of the stability. The order of the steps did not affect the results seriously as far as we could determine by our examination. The longitudinal move and latitudinal move were exchangeable in the experiment (Table 2). In addition, it stands to reason that the movements of $\overrightarrow m $ and $\overrightarrow h $ (i.e., A→B→C→D and C→D→A→B) are exchangeable because the difference between them is just the difference of the start point and the stop point in the iterative process. Since the optimization is executed for all chromophores and all directions, the initial value is not so important.

However, note that there still is a possibility of converging to a local solution and that the acceptable range has a certain limit. For example, if the initial $\overrightarrow m $ and $\overrightarrow h $ are close to each other or become close at some time during the optimization, then they will probably not separate and converge to preferable values. Therefore, the appropriateness of the resulting M and H images has to be evaluated by expert evaluators once after the determination; and if the images are not sufficiently separated, the initial values of the vectors or the selection of test images should be reconsidered. Estimation of the correlation between the results from SMI and other methods such as the spectrum analysis that we used would also be useful to evaluate the results.

With regard to the output images, the difference between SMI and ICA was prominently observed in shading images (Fig. 5). On the other hand, the outcomes of M and H were visually similar, except for overall multiplication, which is not important because the unit was originally arbitrary. Therefore, using M and H from ICA may provide the same results as those from SMI for some purpose. Considering the large difference of $\overrightarrow h $ between SMI and ICA (Table 2), the similarity of the resulting H images indicates the robustness of the framework of the modified Lambert-Beer law. However, in the image of S by ICA, even after the use of a technique for shading removal [13], the patterns of chromophores overlapped (Fig. 5g). In this experiment, the pattern of H was dominantly observed in S, while sometimes the pattern of M was dominantly observed in our preliminary experiments and depended on the situation. It mathematically depends on the choice of the ROI for the shading removal. While a small ROI is preferable in order to minimize the unevenness of shading, the ROI should be large in order to minimize the statistical error. These two requirements are contradictory, and preparing images that satisfy both requirements is difficult. On the other hand, in SMI, patterns of chromophores are not observed in S (Fig. 5d) due to the optimization for minimizing the mixture, which worked efficiently. In fact, the pattern of S images by ICA may be improved by optimizing the size of ROI and the actual selection of ROI; however, the optimal conditions have to be selected by trial and error while monitoring the derived images.

The value of M obtained from SMI had a good correlation with the M obtained from the reflectance spectrum (Fig. 6a), which indicates the linearity assumption giving a good approximation, probably because melanin exists in a limited shallow layer and the non-linear effect of scattering is small. The optical system we used to acquire the reflectance spectrum had an incident angle/measurement angle of d/8°, where d represents the diffusion light, which is different from 0°/d, which is used in the reference [17]. The difference can be a source of error. In addition, the difference between the optical condition of the skin model in the reference and that of actual skin also can be a source of error. However, these matters act in the direction of worsening the correlation. Therefore, the good correlation indicates that the linear model provides a good approximation for M. The method can be used for estimation of melanin concentration using the correlation.

 

Fig. 6. Values from SMI vs. values from spectrum analysis of (a) M (${C_m}$) and (b) H (${C_{th}}$). Solid line: regression line, broken line: 95% confidence interval, dotted line: 95% prediction interval.

Download Full Size | PDF

On the other hand, for H, the correlation was lower than that for M, and the regression line significantly deviated from the origin (Fig. 6b). This is likely due to the broad existence in depth enhancing the non-linearity effect of absorption by hemoglobin. In addition, the difference among individuals of the depth profile of hemoglobin existence and scattering properties also contributed to the low correlation. Therefore, although H may represent the apparent concentration, it is not appropriate for estimation of the actual H concentration. The large SDs of $\overrightarrow h $ compared to those of $\overrightarrow m $ may also reflect the non-linearity of $\overrightarrow h $. The linearity of M and non-linearity of H match the results reported in previous studies [18,21]. Since the limitation came from the divergence of the linear assumption in the modified Lambert-Beer law from the actual phenomenon, the same may apply widely to the linear method based on the modified Lambert-Beer law, no matter how the vectors are derived or how many wavelength bands are used.

In terms of the correlation with the value from a reflectance spectrum, we cannot declare the superiority of SMI to ICA, but rather the correlation coefficients for ICA were larger than those for SMI to some extent in this experiment. We do not think SMI and ICA show a significant difference on this point as far as the framework (Eqs. (1)–(3)) used. The repeatability of the magnitude relationship should be evaluated through applications to several types of photographic equipment in the future.

In the preparation of the test images for optimization of a certain chromophore, keeping the variations of r, g, and b with respect to the variation of the chromophore large is important for high-accuracy estimation in the images. For hemoglobin especially, due to the deviation from the linearity assumption, care should be taken because the convergence tends to be unstable. On the other hand, if the test images are prepared as in the present study, the derived values will converge well with high probability. One of the benefits of the SMI is that we can point out what kinds of images are appropriate and whether a specific image is suitable as a test image, which improves the visibility of the process. In addition, separate image sets selected according to the operation step can be used, which makes convergence more stable. Although distinguishing test image sets is not mandatory, the labor to prepare different sets is appropriate because stability is quite important practically, which is a characteristic that ICA cannot achieve in principle. Viewed from the opposite side of the condition, we should mention that the method is still subjective in terms of the way of choosing the test images, although it eliminates some types of subjectivity. As a more objective way to prepare test images and to evaluate the derived vectors, utilizing the change of skin color after ultraviolet irradiation or application of a blood circulation promoter such as methyl nicotinate are expected to be effective [12]. The changes of the concentrations of chromophores are controllable to some extent with external stimuli. Monte Carlo simulations and phantom studies are also useful for evaluating the validity of SMI (or ICA). We used the results from the reference [17] without changing them for our purpose; however, Monte Carlo simulations and phantom studies can be customized for a specific photographic condition.

The accuracy improvement of S by SMI opened up the possibility of the use of S, which has not yet been fully considered, except by a few studies [23]. For non-invasive and low-impact evaluation of skin condition, photography is a good solution since it is contactless and quick. Therefore, extracting as much information as possible is a good objective. As indicated by Eq. (2), S shows the reflection when the skin chromophore concentrations are zero under the condition of $\overrightarrow \varepsilon = {\left( {\begin{array}{{ccc}} 0&0&0 \end{array}} \right)^T}$, which primarily reflects illuminance [23]. In addition, absorption with other chromophores and angular dependency of reflection may be contributors. The non-linear effect of actual skin is also included. In order to extract useful information from S, we intend to continue our examination.

One advantage of the proposed method is the artifact minimization in imaging, while the pattern is more important than concrete values in some cases, such as visual inspection of the processed images. On the other hand, due to the image based characteristic, the unit of the value is arbitrary, which is a disadvantage. However, the value from this method can be correlated with the values from other methods, as in the experiment. If the same optical system is used to acquire the spectra of the skin model and the actual skin, the estimated concentrations will become more reasonable. Thus, the quantitative treatment and artifact minimization will be combined.

The developed method is suitable for our specific target: measurements in field surveys and product use tests in the field of cosmetics. For this aim, many photographic instruments should be used according to where the experiments are executed and what evaluation items are the focus. We have to adapt the method to newly introduced instruments frequently; therefore, deriving acceptable parameters quickly with certainty is crucial. To spread the method, the vectors should be derived by the users who are introducing the instruments; therefore, the explicitness of operation is important. In addition, since visual evaluation is often used in this field, the strategy of minimizing correlation between images is effective for removing artifacts from 2D images and fits with the purpose of use.

6. Conclusion

Successive minimization of intermixture between vectors enables chromophore vectors to be estimated appropriately and stably, robust to the choice of test images. In practice, SMI advances the capability of prompt action at the time of installation of new equipment. In addition, SMI eliminates arbitrary judgment of acceptance or rejection of vectors for each test image, which improves objectivity. Therefore, the evaluation of accuracy was enabled and is useful as a measurement method.