Open Access
Add to BibSonomy
Abstract
Illumination estimation is a fundamental prerequisite for many computer vision applications. Various statistics and deep learning-based estimation methods have been proposed, and further studies are ongoing. In this study, we first perform a comparative analysis of representative statistics and deep learning-based methods and subsequently investigate combining them to improve the illumination estimation accuracy. We use hyperspectral images as the training data and support vector regression to combine the methods. Based on the results, we confirm that their combination enhances their accuracy.
© 2021 Optical Society of America under the terms of the OSA Open Access Publishing Agreement
1. Introduction
Illumination estimation is a fundamental prerequisite in many research fields. For example, the accuracy of object recognition and object tracking in images can be improved by illuminant color estimation [1,2]. When combining real and virtual worlds in augmented reality, it is necessary to estimate the illuminant color to render three-dimensional models. Therefore, various illumination estimation methods have been proposed. Typical methods include the gray world assumption [3] and the gray edge hypothesis [4,5], which assume that the average reflectance and the average reflectance difference in a scene are achromatic, respectively. Other examples are the shades of gray method [6], which unifies the max-RGB, gray world, and gray edge with the Minkowski $p$-norm. Principal component analysis (PCA)-based method [7] estimates the illuminant color vector in the color space by detecting the lightest and darkest colors in the image by applying PCA. These methods statistically process all pixel values in an image. In addition, techniques that combine multiple conventional methods have been proposed [8]. For example, some methods compute the final illumination color by simply averaging or calculating the median of the results of several existing methods [9,10]. Li et al. combined estimation results using a more advanced learner, support vector regression(SVR) [11]. Concurrently, in recent years, some deep learning-based approaches have been proposed. For example, Hu et al. proposed a method called FC4, which uses a fully convolutional network [12] . FC4 employs raw images as the training data to learn which local regions in an image are useful for illuminant color estimation and determines how to combine these information. This method is very accurate for raw images. The source code is available online [13], and it takes sRGB images as input for practical use. However, as stated on the website of FC4 [13], its accuracy depends on the similarity between the training and test data. Therefore, although it can estimate the illuminant color with very high accuracy for raw images, the errors for other image types are unknown. For example, most illuminant colors in raw images are greenish, whereas the colors of natural illumination in sRGB images vary from red to blue.
In this study, we use hyperspectral images rendered with various illuminant colors and compare the illumination estimation accuracy of conventional methods. Subsequently, we investigate the possibility of combining them using a meta learner to improve their estimation accuracy for illumination colors.
2. Comparative study of conventional methods using a hyperspectral image dataset
In this section, we present the investigation of the performance of conventional methods and the relation among them. The methods used for comparison are summarized in Table 1. The top five methods are statistics-based, and the last one is a deep learning-based method. Hereafter, each method is denoted by the abbreviation specified in Table 1.
First, we define the error. As the error metric, we used the Euclidean distance in the CIE 1976 UCS diagram because it can be converted from simply XYZ values without considering any white point settings; however, it is more perceptually agreeable than $xy$ values. The estimation error, $E_M(I)$, of method $M$ for an image $I$ is in Eq. (1).
where $\hat {u'}$ and $\hat {v'}$ are the estimated $u'$ and $v'$ values, respectively, and $u'_g$ and $v'_g$ represent $u'$ and $v'$ components of the ground truth illuminant color of image $I$, respectively.
Table 1. Methods used in this study
In the following, we explain the test data. Many papers have already reported that deep learning-based methods are superior to statistical methods [12]. However, the illuminant colors of a raw image dataset are not diverse, and the number of images varies among them. Therefore, we render hyperspectral images [14,15] with various illuminant colors and quantitatively investigate the performance of each method under these colors. The number of hyperspectral images is 16. Each hyperspectral image consists of 33 grayscale images of various sizes, from 682 $\times$ 418 to 1,021 $\times$ 1,338. They are sampled at wavelengths from 400 nm to 720 nm. Examples of the hyperspectral images are shown in Fig. 1.
We use Eq. (2) to compute the $XYZ$ values from the spectral distribution.
For $P$, we use black body radiation. The sun is an approximate black body. The colors of artificial lighting used in daily life are also standardized to be near the black body radiation locus. Therefore, in this experiment, we use Planck’s black body radiation equation to generate virtual light radiation, which is in Eq. (3).
where $\lambda$ is the wavelength, $T$ is the temperature, $h$ is Planck’s constant, $c$ is the speed of light, and $k$ is Boltzmann’s constant.We set the color temperatures from 2,000 K to 12,000 K at 500 K intervals. Therefore, the total number of light sources is 21. The colors of light radiation used in this experiment are shown in Fig. 2. Figs. 2 (a)–(c) show the $xy$ values of each illuminant color, spectral data, and illuminant colors for reference, respectively.
Therefore, the total number of test images is 336 (16 hyperspectral images $\times$ 21 light sources). In Fig. 3, the estimation errors are visualized. The error, $E_M(I)$, as shown in Eq. (1), is used to generate boxplots. A boxplot has a box and a set of whiskers. The whiskers show the minimum and maximum data. The box is drawn from the first quartile (Q1) to the third quartile (Q3), with a horizontal line drawn inside to denote the median. The boxes and the median errors show that in this experiment, FC4 and wGE have similar performance and their maximum error values are significant.
On examining the images with significant errors, we find that FC4 has significant errors when the illuminant color is reddish. In contrast, errors in the statistical-based methods vary depending on the scene images. The estimation error for each test data image is visually presented in Fig. 4.
Fig. 4 visualizes all error values, $E_M(I)$, for 336 test images. It shows six rectangles representing the error values of PCA, GW, SoG, GE, wGE, and FC4, respectively. The errors are represented with colors, and the color bar is shown in the right of Fig. 4. Blue color implies a small error, whereas red implies a significant error. Each rectangle is composed of narrow rectangles, which are aligned in 21 $\times$ 16. These narrow rectangles are arranged with different color temperatures in the horizontal direction (21 color temperatures) and different hyperspectral images in the vertical direction (16 types of scene images). Therefore, the narrow rectangles aligned horizontally represent the errors of the same hyperspectral image rendered at different color temperatures. The vertically aligned narrow rectangles represent the list of errors of different hyperspectral images rendered with the same color temperature. For the statistics-based methods, as shown in Figs. 4(a)-(e), higher errors are aligned horizontally, i.e., that the error depends on the scene data, instead of on the illuminant color. In contrast, for deep learning-based method, as shown in Fig. 4(f), large errors are aligned vertically in the leftmost part, which indicates that the errors become significant with illuminations of low color temperatures. These results show that the accuracy of each method is good on average. However, for some images, the accuracy becomes very low, and the conditions for accuracy differ for the statistics- and deep learning-based methods. Therefore, if these methods compensate their individual shortcomings, we can improve the illumination estimation accuracy.
3. Combination of methods via the meta learner
In this section, we explain how the conventional methods are combined to construct an estimator with improved accuracy. We use terms base learner to refer to the conventional method that serves as the input to the meta learner and meta learner to refer to the learner that combines the predictions of base learners and produces the final prediction.
3.1 Training data for the meta learner
In this section, we explain the training data used in meta learning. For training, we use hyperspectral images rendered with various illuminant spectra. These images are the same as shown in Section 2.; however we carefully change the illuminant set as described below.
The color temperature should cover the range of daily illuminant colors to the maximum extent without any color bias. However, the data we mentioned in Section 2. do not have perceptually equal intervals, as shown in Fig. 2. As the color temperature becomes high, the spacing on the chromaticity diagram becomes narrower. The perceived color difference is closely related to the reciprocal of the color temperature (RCT) [16]. Therefore, we sample the training color temperature set using the RCT, as defined in Eq. (4).
where $T^{-1}$ is the RCT and $T$ is the color temperature. In the following, we discuss the range of light source colors. The color temperature of illumination commonly used daily is generally in the range of 3,000 K - 6,600 K, and that of both the sunrise and sunset is 1,900 K [17]. The colors for solid-state lighting, such as LEDs, in daily use are standardized as 2,200 K to 6,500 K [18]. Natural lights measured in Granada range from 3,804 K to $\infty$ K [19]. Therefore, in this study, we decide to sample the RCT from 0 (= $\infty$ K) to $550$ (= 1,818 K), in intervals of 10. The corresponding color temperatures are listed in Table 2. We use $INT\_MAX$ instead of $\infty$ because we cannot treat $\infty$ on a computer. These 56 temperatures are inputted in Planck’s equations and the illumination spectra are generated. The colors and the sampled spectra are shown in Fig. 5. Figures 5 (a)–(c) show the $xy$ values of illuminant color, spectral data, and pseudo colors, respectively. Note that some reddish illuminations are out of the sRGB scope, and the colors are inexact.Finally, the training dataset becomes 896 (= 16 types of hyperspectral image $\times$ 56 types of illumination ).
Fig. 5. Illumination for training data. Note that colors in (c) are inexact because some reddish illuminations are out of sRGB scope. Spectra shown in (b) are normalized at 550 nm.
Table 2. Color temperatures used in training
3.2 Meta learner and feature selection
The simplest approach to ensemble methods is the average or the weighted average of all outputs of conventional methods. For achieving better performance and avoiding overfitting, feature selection is performed, and the meta learner is examined.
First, we select the base learner. GW, SoG, and GE can be expressed by the same equation [6], and their properties are similar. Therefore, only GE, which presented the best accuracy, is adopted as the base learner among them. Thus, we choose GE, wGE, PCA, and FC4 as the base learners. Subsequently, we perform feature selection on the input features of the meta learner. Generally, a good feature subset should have a high correlation with the ground truth [20]. The features used in this study are the estimation results of the base learners. Figure 6 shows the correlation between the outputs of the base learners and ground truth. Based on Fig. 6, CIE-$x$ value of FC4 and the CIE-$y$ value of PCA are not used as the input because of their low correlation with GT. Therefore, the inputs to the meta learner are the CIE-$x$ value outputted from PCA, CIE-$xy$ values from GE and wGE, and CIE-$y$ value from FC4 yielding a total of six dimensions. If a statistical-based method fails to estimate the illuminant color, the method with the highest correlation is used to impute the missing value. PCA and wGE are imputed by GE, and GE is imputed by wGE.
For the meta learner, we refer to the sklearn cheat sheet [21] and investigate some representative methods. Finally, we adopt the linear SVR as the meta learner.
We show the final system in Fig. 7. With each base learner $i$, we estimate the $RGB_i$ value of an illuminant color and convert it to $x_i$ and $y_i$. These $x_i$ and $y_j$ values are used as the inputs to the meta learner. The meta learner subsequently estimates the final $x$ and $y$ of the illuminant color based on them.
4. Experiments
In this section, we compare the conventional methods with the proposed method described in Section 3.. Hereafter we denote the latter as SVR.
4.1 Experimental setup
We evaluate the performance of SVR with the following four experiments:
- 4. Experiment 4: laboratory image dataset with black body radiation (CAVE dataset $\times$ Fig. 9)
Fig. 11. Illumination for Experiment 3. Colors shown in (c) are inexact because many red and orange illuminations are out of sRGB scope. Note that normalization method of spectra shown in (b) is different from those of spectra in other figures because some illuminations have zero radiance at 550 nm.
Experiment 1 is similar to the experiment described in Section 2.. In this experiment, we investigate whether the accuracy of SVR depends on the color temperatures or the scene images. Experiments 2 and 3 are conducted using natural scene data rendered with real illuminant spectra. The experiments are conducted to verify the performance of the proposed method in such realistic scenes. Experiment 4 is an experiment conducted for reference using laboratory scene images. There are many images with biased object colors, such as an image with only two apples with black background. With these scenes, estimation of the illumination color is considered to be difficult. Although such scenes are rarely captured in daily life, we tested them to reference a bad case.
4.2 Test data
In Experiments 1, 2, and 3, we use hyperspectral images of natural scenes [22]. Each hyperspectral image consists of 33 grayscale images of size 1,344 $\times$ 1,024 pixels sampled at wavelengths from 400 nm to 720 nm. The total number of hyperspectral images is 30, but it contains very similar images that we used as the training data. Therefore, we exclude these 9 images, and the remaining 21 images are used as the test images. Examples of the test images are shown in Fig. 8 . For experiment 4, we use the CAVE multispectral image database [23]. This database contains 32 multispectral images in total that are taken in the laboratory. Each hyperspectral image consists of 31 grayscale images of size 512 $\times$ 512 pixels sampled at wavelengths from 400 nm to 700 nm. Examples of the images are shown in the website of CAVE project [24]. All images have black backgrounds except one image, and many of the images have only a few objects. All images contain a color checkerboard, and we mask them in advance.
In the following, we explain the illumination spectra. Black body radiations are used in Experiments 1 and 4. We sampled a total of 20 ranging from 2,250 K to 11,750 K at 500 K intervals. The chromaticities of the illuminant colors and the spectra are shown in Fig. 9. For Experiment 2, we use a natural outdoor sky spectral distribution [19], which is acquired outdoor in Granada, Spain. The dataset contains 2,600 spectral data. We randomly sample 210 data, which is the same number as the lamp data described subsequently. The chromaticities of the illuminant colors and the spectra are shown in Fig. 10. For Experiment 3, we use a lamp spectral dataset [25]. It consists of 210 lamp spectral distributions. We exclude 4 light therapy lamp data from them. Therefore, the number of used data is 206, and the dataset includes fluorescent lamps, halogen lamps, LEDs, HPS lamps, and other sources. As shown in Fig. 11, most of the data are reddish, and some are highly reddish or bluish, close to the spectral locus.
4.3 Results
In Table 3, we summarize the results of Experiment 1. We compute the mean, median, and tri-mean of the errors expressed in Eq. (1). The tri-mean is defined as Eq. (5).
where $Q1$, $Q2$, and $Q3$ are the first, second, and third quartiles, respectively. 25% best implies $Q_1$ and 25% worst implies $Q_3$.In Experiment 1, which was performed on natural images with black body radiations, SVR achieved the best performance. This result shows the possibility that images under natural illumination, such as sky or sunlight, can yield very high estimation accuracy. To investigate the error tendency of the proposed method, we performed the same error analysis as in Fig. 4. The results are shown in Fig. 12. In Figs. 12(a)-(f), we show the errors of the conventional methods for reference. In Fig. 12(g), we show the errors of SVR. It can be seen that most of the small squares are blue, indicating that the overall error is small. In particular, unlike FC4, the accuracy at low color temperature is high, indicating that the proposed method successfully compensates the shortcomings of FC4. In SVR, images ID 7 and 18, rendered with color temperature 11,750 K, have lower accuracy than other images. We show these two images in Fig. 13. In these two images, almost the entire scene is composed of yellowish walls or rocks, which may have caused an error in the estimation.
Table 4 summarizes the results of Experiment 2. Experiment 2 is conducted on natural scenes with sky spectra. SVR achieves a performance as good as in Experiment 1. This result ensures that SVR can produce high accuracy for natural images.
Table 3. Error metric values for Experiment 1 (values are scaled by $\times 10^3$)
Table 4. Error metric values for Experiment 2 (values are scaled by $\times 10^3$)
Table 5 summarizes the results of Experiment 3. Note that wGE fails to estimate the illuminant colors that are very much saturated such that the $xy$ value is near the spectral locus. Therefore, we could not calculate the mean of wGE, and denote it as "-". In Experiment 3, SVR presents a performance as good as in Experiments 1 and 2. Therefore, for natural scenes, our SVR method achieves the best performance.
Table 6 summarizes the results of Experiment 4. In Experiment 4, which was on the laboratory dataset rendered with black body radiations, SVR presents the best performance in terms of the mean and the second best in the 25% worst case. We plot the error map in Fig. 14. In this test dataset, the scenes are artificial laboratory settings, and the scenes do not satisfy the assumption of PCA. Therefore, the error in PCA becomes very large, as shown in Fig. 14. In GE and wGE, the errors of Image IDs 5-8 are very large. Compared with other methods, in SVR, the narrow boxes are rather bluish, which suggests that the overall error is low. We summarize the worst errors in Table 7. We can confirm that the maximum error is much smaller than those of other methods. Therefore, even though our median is not the best in the laboratory scene, our SVR method can perform better in the worst case than the other methods.
Table 5. Error metric values for Experiment 3 (values are scaled by $\times 10^3$)
Table 6. Error metric values for Experiment 4 (values are scaled by $\times 10^3$)
Table 7. Maximum error values for Experiment 4 (values are scaled by $\times 10^3$)
5. Conclusion
In this study, we compare conventional methods using hyperspectral images and proposed a new method to improve the estimation accuracy by combining some conventional methods with SVR. The results show that although the performance of machine learning is excellent, it is sometimes inferior to the statistics-based methods for light sources that are far from the training dataset. The results also show that it is possible to create a better estimator by integrating these methods. The proposed estimator yields better estimation results for outdoor scenes, better accuracy than FC4 for most metrics and datasets, and fewer images with more significant errors than the other methods. Overall, the present results indicate that machine learning is good; however, when a training dataset is biased, it may be able to provide better estimation and recognition by incorporating conventional knowledge. In the future, we plan to conduct experiments not only using hyperspectral images but also with more various types of images with various illuminants. We also would like to study a method to estimate the spectrum of the light source.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. M. Doi, T. Matsumoto, A. Kimachi, S. Nishi, and N. Ikoma, “Robust color object tracking method against illumination color Change,” in 2014 Joint 7th International Conference on Soft Computing and Intelligent Systems (SCIS) and 15th International Symposium on Advanced Intelligent Systems (ISIS), 718–722 (2014).
2. M. Afifi and M. S. Brown, “What else can fool deep learning? Addressing color constancy errors on deep neural network performance,” in Proc. IEEE International Conference on Computer Vision, 243–252 (2019).
3. G. Buchsbaum, “A Spatial Processor Model for Object Colour Perception,” J. Franklin Inst. 310(1), 1–26 (1980). [CrossRef]
4. J. Weijer, T. Gevers, and A. Gijsenij, “Edge-Based Color Constancy,” IEEE Trans. on Image Process. 16(9), 2207–2214 (2007). [CrossRef]
5. A. Gijsenij, T. Gevers, and J. Weijer, “Improving Color Constancy by Photometric Edge Weighting,” IEEE Trans. Pattern Anal. Mach. Intell. 34(5), 918–929 (2012). [CrossRef]
6. G. D. Finlayson and E. Trezzi, “Shades of Gray and Colour Constancy,” IS&T/SID Twelfth Color Imaging Conference, 37–41 (2004).
7. D. Cheng, D. K. Prasad, and M. S. Brown, “Illuminant Estimation for Color Constancy: Why spatial-domain methods work and the role of the color distribution,” J. Opt. Soc. Am. A 31(5), 1049–1058 (2014). [CrossRef]
8. B. Li, W. Xiong, W. Hu, and B. Funt, “Evaluating Combinational Illumination Estimation Methods on Real-World Images,” IEEE Trans. on Image Process. 23(3), 1194–1209 (2014). [CrossRef]
9. S. Bianco, F. Gasparini, and R. Schettini, “Consensus-based framework for illuminant chromaticity estimation,” J. Electron. Imaging 17(2), 023013 (2008). [CrossRef]
10. V. Cardei and B. Funt, “Committee-based color constancy,” in IS&T/SID Color Imaging Conference(CIC), 311–313 (1999).
11. B. Li, W. Xiong, and D. Xu, “A Supervised Combination Strategy for Illumination Chromaticity Estimation,” ACM Trans. Appl. Percept. 8(1), 1–17 (2010). [CrossRef]
12. Y. Hu, B. Wang, and S. Lin, “FC 4: Fully Convolutional Color Constancy with Confidence-weighted Pooling,” Proc. IEEE Conference on Computer Vision and Pattern Recognition, 4085–4094 (2017).
13. https://github.com/yuanming-hu/fc4 (Reference on July 7th, 2021).
14. S. M. C. Nascimento, F. P. Ferreira, and D. H. Foster, “Statistics of spatial cone-excitation ratios in natural scenes,” J. Opt. Soc. Am. A 19(8), 1484–1490 (2002). [CrossRef]
15. D. H. Foster, K. Amano, S. M. C. Nascimento, and M. J. Foster, “Frequency of metamerism in natural scenes,” J. Opt. Soc. Am. A 23(10), 2359–2372 (2006). [CrossRef]
16. A. V. Arecchi, T. Messadi, and R. J. Koshel, Field Guide to Illumination (SPIE, 2007).
17. J. Lin, X. Ding, C. Hong, Y. Pang, L. Chen, Q. Liu, X. Zhang, H. Xin, and X. Wang, “Several biological benefits of the low color temperature light-emitting diodes based normal indoor lighting source,” Sci. Rep. 9(1), 7560 (2019). [CrossRef]
18. “Electric Lamps - Specifications for the Chromaticity of Solid-state Lighting Products,” ANSI C78.377-2015.
19. J. Hernandez-Andres, J. Romero, J. L. Nieves, and R. L. Lee Jr., “Color and spectral analysis of daylight in southern Europe,” J. Opt. Soc. Am. A 18(6), 1325–1335 (2001). [CrossRef]
20. M. Hall, “Correlation-based Feature Selection for Machine Learning,” Ph.D. dissertation, University of Waikato (1999).
21. https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html (Rerefence on July 7, 2021).
22. S. M. Nascimento, K. Amano, and D. H. Foster, “Spatial distributions of local illumination color in natural scenes,” Vision Res. 120, 39–44 (2016). [CrossRef]
23. F. Yasuma, T. Mitsunaga, D. Iso, and S. K. Nayar, “Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum,” Tech. 1 Rep., Dept. of Comp. Sci., Columbia University CUCS-061-08 (2008).
24. CAVE dataset, https://www.cs.columbia.edu/CAVE/databases/multispectral/ (Reference on July 7, 2021).
25. LSPDD, http://galileo.graphycs.cegepsherbrooke.qc.ca/app/en/home (Reference on July 7, 2021).
