Open Access
Add to BibSonomy
Abstract
This study developed a method to simulate the sensor responses and verify the effectiveness on spectral reconstruction by a spectrum tunable LED system. Studies have shown that the spectral reconstruction accuracy could be improved by including multiple channels in a digital camera. However, the real sensors with designed spectral sensitivities were hard to manufacture and validate. Therefore, the presence of a quick and reliable validation mechanism was preferred when performing evaluation. In this study, two novel approaches, i.e., channel-first and illumination-first simulations, were proposed to replicate the designed sensors with the use of a monochrome camera and a spectrum-tunable LED illumination system. In the channel-first method, the spectral sensitivities of three extra sensor channels were optimized theoretically for an RGB camera and then simulated by matching the corresponding illuminants in the LED system. The illumination-first method optimized the spectral power distribution (SPD) of the lights using the LED system, and the extra channels could be determined accordingly. The results of practical experiments showed that the proposed methods were effective to simulate the responses of the extra sensor channels.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Spectral reflectance is the most fundamental characteristic of an object to specify its color. Only when a spectral match occurs between two objects can their colors match under arbitrary illuminant and observer conditions. The color of an object can be accurately reproduced using its spectral reflectance without considering the problem of metamerism [1].
A spectrophotometer can be used to measure the spectral reflectance of an object, but it can measure only one sample at a time. A convenient method is to reconstruct the spectral reflectance from the responses of a digital camera [2–5]. This approach is inexpensive and easy to employ, and is acknowledged as an efficient non-contact measurement technique. The responses of the camera can be extracted from the captured images and are used to reconstruct the spectral reflectance of objects. Numerous algorithms have been utilized in spectral reconstruction [6–9], including Pseudo-Inverse, Wiener Estimation, Smoothness Constraint, Principal Component Analysis (PCA), etc. However, the number of camera channels, often just three for commercial digital cameras, is a recurrent issue, limiting the spectrum reconstruction precision of these methods. It is an ill-posed problem to reconstruct the high-dimensional spectral data from the three-dimensional RGB responses of a camera [6]. Recently, approaches based on deep learning have been implemented in spectral reconstruction [10–14]. Nevertheless, it requires a huge amount of training datasets, and its accuracy cannot be ensured when tested on an unfamiliar scene.
The most dependable approach is based on physical level. The method of multiple exposures can be applied to expand the dimensions of the camera responses. One method is to use multiple cameras in a stereoscopic configuration [15], but this method requires cumbersome and costly devices. Another method is to use a single camera to capture images with multiple filters [16–20]. Various studies have focused on the optimization of filters in a multispectral imaging system. But its disadvantages are that it is time-consuming and of low efficiency due to the need for multiple exposures.
Therefore, it is preferable to obtain multiple responses from a single exposure, which can be accomplished by including multiple channels into a camera and determining their ideal spectral sensitivity combinations [21–23]. The optimization of multichannel sensor has been extensively studied [24–31]. Miguel A et al. [24–26] sought the optimal combination of three to five Gaussian sensors for the spectral recovery of skylight. S. Quan et al. [27] designed a six-channel camera aiming at both colorimetric and spectral performance in color reproduction using a parametric model of the manufacturing process. N. Hu and C. Wu [28] studied the optimal commercial six-sensor sets for linear model representation and daylight spectrum recovery. However, the majorities of the previous studies were based on theoretical calculations. In real practice, it’s time-consuming and complicated to create a real sensor and put it to use. The issue in question is how to evaluate the performance of the optimized multi-channel sensors before they are manufactured.
With the rapid development of LED technology, spectrum tunable LED illumination systems have been used in the multispectral imaging [32–35]. The LED-based multispectral imaging system contained multi-channel narrow bandwidth LEDs together with a monochrome camera or trichromatic camera. Meanwhile, previous researchers have focused on the optimal light sources for spectral reflectance reconstruction [11,36–38]. But few of them investigated the possibilities to simulate the camera responses using an LED system although this method is simple to implement and has a low cost.
In this study, it was proposed to replicate the sensor channels with the use of a monochrome camera and a spectrum-tunable LED illumination system. The sensor channels to be simulated were three extra channels designed for an RGB camera with the aim of improving the spectral reconstruction accuracy. Two approaches for sensor simulation, namely channel-first and illumination-first methods, were proposed. In the channel-first approach, the spectral sensitivities of the extra channels were optimized theoretically. They were then replicated by fine-tuning the LED illumination system to provide a specific light that could reproduce the same camera responses as the extra channels. In the illumination-first approach, the SPD that the LED illumination system could repeat was optimized. The additional channels were then determined accordingly. The proposed methods were verified in a physical experiment. It was shown that the extra sensor channels could be simulated by the LED system, and the accuracy of sensor simulation was evaluated.
2. Methods
With the assumption of the linear responses of a digital camera [39], the responses of each channel can be expressed as
After discrete sampling at different wavelengths, for example, from 400 nm to 700 nm, with an interval of 10 nm, Eq. (1) changes to the matrix multiplication, as given in Eq. (2).
R is a C by N matrix, representing the C-channel camera responses of N objects. S, L and r are the C by 31 spectral sensitivities of the camera, 1 by 31 SPD of the light, and the 31 by N spectral reflectance of the objects, respectively. Diag(L) is the diagonal matrix of L, with each element on the diagonal corresponding to the SPD of the light. E is the camera noise.
As has been described, the spectral reconstruction accuracy from camera responses was limited by the number of camera channels (usually only three). To improve the accuracy of spectral reconstruction, extra channels can be designed and added in addition to the existing three channels in a camera. However, it is time-consuming and complicated to build a genuine sensor, making it a challenge to verify the performance of the extra sensor channels.
In this section, two approaches (channel-first and illumination-first) were presented to simulate the responses of the extra sensor channels, with the use of a spectrum tunable LED illumination system and a monochrome camera. The channel-first approach aimed at simulating the extra channel with a specific spectral sensitivity. The extra channels were optimized theoretically and then simulated using an LED system. In this approach, sensor matching objective was proposed. In the illumination-first approach, in order to better simulate the SPD of the lights reproducible by the LED system, the extra sensor channels were optimized under the constraint of the LED system. It was equivalent to the direct optimization of the SPD of the lights in the LED system. It should be noted that, the goal of this study was sensor simulation instead of sensor optimization. The authors did not intend to enhance the sensor optimization algorithm. But the optimization procedure was necessary as it provided a set of reasonable sensor sensitivities for the subsequent simulation process. The detailed procedures were described as below.
2.1 Channel-first simulation
First of all, the spectral sensitivities of the extra sensor channels were theoretically optimized. A conventional Pseudo-Inverse method [41,42] was used for the spectral reflectance reconstruction. This method learned the relationship between the spectral reflectance and camera responses based on the training samples, and recovered the spectral reflectance of the testing samples from the camera responses. The spectral sensitivity of the extra sensor channel was expressed by a linear combination of a series of normalized Gaussian functions centered at different wavelengths, with Full Width at Half Maximum (FWHW) of 40 nm. The center wavelengths were ranged from 400 nm to 700 nm with an interval of 10 nm. Therefore, the spectral sensitivity of the extra channel could be expressed by the weighted sum of 31 Gaussian functions as given in Eq. (3), where B is the Gaussian basis functions, x is the coefficients to be optimized, and F is the spectral sensitivity of the extra channel. Using Gaussian basis functions could ensure the smoothness of the optimization results and better imitate the sensitivities of real sensors.
The optimization objective was to minimize the spectral reconstruction error between the recovered and the ground truth spectral reflectance of the testing samples. The minimized function was a colorimetric and spectral combined metric (CSCM) [26] as in Eq. (4). The colorimetric metric was the color difference CIEDE2000 (ΔE00) [43]. The chromaticity was calculated using CIE 1931 standard observer under CIE D65. The spectral metrics included RMSE (Root Mean Square Error) [37] and GFC (Goodness of Fit Coefficient) [44,45]. The CSCM function was motivated by [26] and was modified with RMSE included. The coefficients a, b, and c were the weights of different metrics. In this study, $a = b = c = 1$ was set.
Simulated annealing algorithm [18,25] was adopted to solve the optimization problem. This algorithm simulates the process of annealing (slow cooling after heating) of a thermodynamic system that is always in thermal equilibrium. It searches for the state with the minimum energy when the temperature decreases with time. In this study, the current state is the solution of the problem, i.e., the coefficients x. The corresponding energy is the value of the objective function CSCM. The optimization starts with a random initial solution, and then the existing state is randomly perturbed and its corresponding energy is calculated. If the energy is decreased, the new state will be accepted. Otherwise, it is accepted with a Boltzmann probability. The probability to accept a new state with higher energy will decrease as the temperature is decreased. This step decreases the probability of the algorithm to be trapped in a local minimum solution, but cannot guarantee to obtain the global minimum.
The setup of the parameters was described as follows. The initial and lower limit of temperature were 1000 K and 10−6 K, respectively. The decreasing rate of temperature was 0.9. The iteration times at each temperature was 100. The stop criterion was that the current temperature was below the lower limit. To achieve a physically realizable spectral sensitivity, nonnegative constraint was included.
Three extra camera channels were optimized in turn and followed a greedy strategy. The optimization process provides the desired extra sensor channels for the simulation section. It was again noted that the goal of this study was not on sensor optimization, but on the subsequent sensor simulation.
The channel-first approach intended to simulate an extra sensor channel with a given shape of spectral sensitivity. Following the above method, the extra sensor channels have been optimized and predetermined. A monochrome camera was selected to simulate the extra sensor channels due to its broadband sensitivity over the visible spectrum. One of the original three channels of the RGB camera could also be used for sensor simulation. However, for a specific channel, there were always some wavelength ranges with low sensitivity, making it more difficult to simulate an arbitrary shape of extra channel sensitivity. As a result, a monochrome camera with broadband sensitivity was preferred for sensor simulation in this study.
Suppose that the spectral sensitivity of the extra sensor channel to be simulated is ${F_C}(\lambda )$, and the sensitivity of the monochrome camera is ${F_M}(\lambda )$. The SPD of D65 illuminant is ${L_0}(\lambda )$. If an illuminant with SPD of $L(\lambda )$ as given in Eq. (5) can be matched,
the response of the monochrome camera to the light $L(\lambda )$ can then be calculated as,The SPD of the light produced in an LED system (denoted as $L^{\prime}(\lambda )$) can be expressed by a linear combination of the SPDs of all the LED channels, according to Eq. (7), where ${L_i}(\lambda )$ is the SPD of each LED channel, and ${k_i}$ is the coefficients to be determined under a non-negative constraint. Q is the quantity of the LED channels. The center wavelength shift of the LED channel with different luminance levels was ignored.
By combining Eq. (5) and Eq. (7), the spectral sensitivity of the real simulated extra channel ${F_C}^{\prime}(\lambda )$ can be inversely solved,
2.2 Illumination-first simulation
This section described the second approach for sensor simulation, namely illumination-first approach. Given that the light with SPD of $L(\lambda )$ in Eq. (5) might not be perfectly matched due to the limited number of available channels in an LED system, this approach optimized the SPD of light $L(\lambda )$ directly. Then the corresponding sensor channel could be determined according to Eq. (8). In other words, the extra sensor channels were optimized under the constraint of the LED system.
The coefficients of LED channels ${k_i}$ were optimized with a non-negative constraint, in order to minimize the CSCM function. This problem can be solved using a similar simulated annealing algorithm as described in Section 2.1. Once the coefficients were obtained, the SPD of light and the corresponding sensor channel could be simultaneously determined. This approach was considered as the illumination-first simulation of the sensor channel. The spectral sensitivities of the extra sensor channels optimized with this approach might be constrained by the SPDs of LED channels and the spectral sensitivities of the monochrome camera. But the corresponding lights could be exactly matched in theory, and thus the extra sensor channels were expected to be simulated and verified more effectively by the LED system.
3. Experiments and results
Following the above methods, both channel-first and illumination-first simulations were implemented. The flowchart of the sensor simulation experiments is illustrated in Fig. 1. In the channel-first simulation, the sensor sensitivities were first determined and then simulated by matching the illuminants in the multichannel LED system. In the illumination-first simulation, the SPDs of the lights were optimized, and then the related sensor channels were determined. The simulation accuracy was evaluated in terms of the simulated spectral sensitivities, spectral reconstruction accuracy, and the simulated camera responses.
In this study, the X-Rite Macbeth ColorChecker Chart (MCCC) was used as the training dataset. The X-Rite ColorChecker Digital SG (140 colors) and the DigiEye calibration chart DigiTizer (240 colors) were used as the testing datasets. The SG chart was used for the sensor optimization and the DigiTizer was used for further validation. The main camera of a Huawei P40 mobile phone was used as the test camera. All the image processing functions were switched off to get the raw images from the camera. The camera used to simulate the designed sensor was a high-speed monochrome industrial camera Hikrobot MV-CS060-10UM-PRO with 12-bit depth. Figure 2 shows the spectral sensitivities of the two cameras, measured using the Thouslite LEDSSF instrument [47]. This device contains an LED cube with multiple channels. The camera responses and the SPD of each LED channel were measured so that the spectral sensitivities of the camera could be determined. The test illuminant was a simulated D65 illuminant.
Fig. 2. The spectral sensitivities of the test cameras. (a) Huawei P40 main camera, (b) Hikrobot monochrome camera.
3.1 Channel-first simulation
3.1.1 Extra channels optimization
First of all, three extra sensor channels were theoretically optimized following the method in Section 2.1. The optimization was based on synthetic data. Random Gaussian noise with SNR of 45 dB was added [48]. Figure 3 shows the spectral sensitivities of the RGB camera together with the three optimized extra channels. It was found that all the three extra channels had different peak wavelength compared to the original three camera channels.
Fig. 3. The spectral sensitivities of the original RGB camera and the three optimized extra sensor channels.
Table 1 summarizes the prediction errors of spectral reflectance reconstruction with varying numbers of extra channels. Zero means no added channel and represents the initial results. The errors were calculated in terms of both colorimetric and spectral metrics. The colorimetric metric was the mean ΔE00. Note that the spectral reconstruction was carried out under the test illuminant D65. Then the ΔE00 error was calculated between the reconstructed and the ground truth reflectance under different illuminants: D65, A, and F11. So, the ΔE00 error could be considered as the Metamerism Index (MI) [49]. The spectral metrics included RMSE and GFC, indicating the similarity between the spectral shapes of reconstructed and ground-truth reflectance. A lower RMSE value means a closer match between the shapes of the two spectral reflectance, while a greater GFC value means a closer match of the relative shapes. The value of GFC ranges between 0 and 1. A GFC larger than 0.99 is considered acceptable quality [45].
Table 1. The errors of spectral reflectance reconstruction with different numbers of extra channels in terms of colorimetric metric (ΔE00) and spectral metrics (RMSE and GFC) based on synthetic data.
It was found that after adding an optimized channel to the existing camera, the spectral reconstruction accuracy of both testing samples could be significantly improved in terms of all the three metrics. The mean color difference of the three lights decreased by about 10%, the RMSE decreased by about a third, while the GFC value got bigger than 0.99. With more extra channels, the spectral reconstruction error could be reduced further. However, the decreasing trend slowed down as the number of extra channels increased, especially for the third extra channel. The ΔE00 under F11 even slightly increased and the GFC slightly decreased when introducing the third extra channel. This could be due to the accumulation of noise with the increasing numbers of sensor channels.
3.1.2 Extra channels simulation
The three extra sensor channels were then simulated in a physical experiment. The experiment was conducted in a Thouslite LEDView viewing cabinet equipped with an 18-channel spectrum tunable LED illumination system. Figure 4 shows the normalized SPDs of each LED channel. Totally, 14 of them were within the visible range of 400 nm to 700 nm. A D65 illuminant was matched in the LED system. The SPD was measured using a JETI-Specbos 1211 spectroradiometer on a white plate. This SPD could be considered as ${L_0}(\lambda )$. Figure 5 plots the measured SPD of the simulated D65 and the SPD of CIE D65.
The simulation with sensor matching objective was implemented. Figures 6(a) to (c) show the three optimized and matched illuminants. The SPDs of the matched illuminants were measured by the spectroradiometer. Figures 6(d) to (f) show the corresponding optimized and matched sensor sensitivities. Limited by the number of available LED channels, the shapes of the sensors could not be perfectly replicated, but the general shapes could be approximately simulated.
Fig. 6. The results of the channel-first sensor simulation. (a)∼(c) The SPDs of the three optimized and matched illuminants. (d)∼(f) The corresponding optimized and matched sensor sensitivities.
After matching the illuminants related to the desired extra channels, the training and testing charts were captured by the RGB camera under D65, and by the monochrome camera under the matched illuminants. Figure 7 shows the experimental setup. The color chart was placed on a holder with a fixed angle of 60°, in the center of the viewing cabinet. The mobile camera was mounted on a tripod and adjusted so that the major axis of the lens was perpendicular to the color chart, permitting distortion-free capture of the color chart. The illumination/viewing geometry was set at 60°/0°. It was discovered that the semi-glossy SG chart with a 45° geometry had a significant amount of specular reflection. As a result, the illumination geometry was set at 60° to reduce the specular reflection. During the experiment, all the image processing functions of the camera were switched off to get the RAW images of the sensor. The focus, ISO, and exposure time were adjusted to their optimal settings and then fixed, to ensure that no overexposure occurred for the white patches. Images of the three charts were taken under D65. Then the mobile camera was replaced by the monochrome camera, the images under three matched lights were taken. The original RGB responses of patches under D65 were extracted from the RAW RGB images, while the responses of the monochrome camera under the three matched lights were retrieved to simulate the responses of the extra channels under D65. Additionally, a completely dark image was captured to subtract the dark noise from the camera responses. A white board was captured under all the four lights in order to correct the illumination uniformity [50]. The linearity of the camera responses was verified by the six neutral patches on the MCCC under D65. It was found that for both cameras, the relationship between the camera responses and reflectance could be fitted by a linear proportional function with a correlation coefficient greater than 0.999. As a result, no further linearization was conducted. Finally, the 6-channel responses of the three charts under D65 were obtained. As previously described, MCCC was used as training samples, while the SG and DigiTizer charts were used as testing samples. The spectral reflectance of the testing charts was reconstructed using the Pseudo-Inverse method. The 3-channel (original RGB channels) and 4-6 channels (with 1-3 extra channels) camera responses were used for spectral reconstruction respectively. The ground-truth spectral reflectance of the charts was measured using a spectrophotometer Datacolor SF600. The measuring conditions were specular excluded, small aperture. It gave the results similar to those of 45°/0°.
Table 2 lists the spectral reconstruction error in the practical experiment. It can be seen that, for both testing samples, the RMSE error decreased and GFC increased as the number of extra channels increased. While the ΔE00 error decreased and then became nearly stable. The most noticeable decreasing tendency occurred when the first extra channel was added. With two more extra channels, the accuracy continued to increase but this tendency became less pronounced. This tendency was similar with that of the synthetic data, which could be due to the accumulation of noise. With three extra channels, the average ΔE00 values under D65 decreased by about 10% for both testing samples, and the RMSE errors decreased by about 20% and 34% for SG and DigiTizer, respectively.
Table 2. The measured error of spectral reflectance reconstruction in practical test with different numbers of extra channels.
In some cases, the color difference would slightly increase, such as the ΔE00 error of SG under F11. This could because, the F11 illuminant has peaky SPD and thus samples the spectrum at some specific wavelengths, which might not be fully considered in the optimization section. Moreover, it was observed that sometimes the colorimetric and spectral metrics have inconsistent variation trends. In this case, the optimization was the trade-off between the three metrics. The weights of different metrics in Eq. (4) could be adjusted to make a balance or place particular emphasis on one of the metrics.
It was found that the spectral reconstruction error based on measured data was larger than that of the synthetic data. The discrepancies between the measured camera responses and those inferred from the camera sensitivities led to a larger spectral reconstruction error in the practical experiment. This could be influenced by the accuracy of the camera spectral sensitivities, the nonuniformity of the illuminant, and the inaccurate noise model. Furthermore, the optimized illuminants were not perfectly matched in the LED system due to the limitation of LED channels. In other words, the spectral sensitivities of the simulated extra sensor channels were not perfectly identical to the optimized ones. As a result, the illumination-first optimization and simulation were implemented in the following section.
3.2 Illumination-first simulation
In this section, the SPDs of the three lights and the related three extra sensor channels were obtained by the illumination-first method as described in Section 2.2. Random Gaussian noise with SNR of 45 dB was added. Figure 8(a) shows the SPDs of the three optimized illuminants. Figure 8(b) plots the spectral sensitivities of the original RGB camera and the related three extra channels. Table 3 lists the spectral reconstruction error based on synthetic data.
Fig. 8. (a) The SPDs of the three optimized illuminants. (b) The spectral sensitivities of the original RGB camera and the three extra channels.
Table 3. The theoretical error of spectral reflectance reconstruction in the illumination-first optimization.
Then the three optimized lights were matched in the LED system. Figures 9(a)∼(c) plot the SPDs of the three optimized and matched illuminants, and (d)∼(f) plot the corresponding optimized and matched sensors. As expected, the optimized illuminants and the corresponding sensors were well matched by the LED system. Intuitively, the sensor sensitivity could be better matched than in the channel-first simulation.
Fig. 9. The illumination-first sensor simulation. (a)∼(c) The SPDs of the three optimized and matched illuminants. (d)∼(f) The corresponding optimized and matched sensors.
Table 4 lists the measured spectral reconstruction error in the practical experiment. It can be seen that the spectral reconstruction accuracy increased as the number of extra channels increased. With three extra channels, the average ΔE00 values fell by about 18% and 15% for the SG and DigiTizer charts, respectively. The RMSE errors decreased by about 32% and 37%, respectively. Besides, it can be observed that, in the practical experiments, the spectral reconstruction error in the illumination-first simulation was smaller than that of the channel-first simulation. This result was reasonable because in the channel-first simulation, the sensor channels were optimized and finally simulated by the LED system. It could not get better results than optimizing the lights directly as the illumination-first simulation. Otherwise, it means that we did not find the global optimized results in the lights optimization.
Table 4. The measured error of spectral reflectance reconstruction in the illumination-first simulation.
In a previous study [17], Nahavandi et al. proposed a manufacturable approach for the design of optical filters. They showed great benefits to apply the optimization of physical variables (the solvent dyes concentrations) in the filter manufacturing than the conventional approach of designing and then physically manufacturing the designed filters. The method ensured the feasible production of the designed filters. Meanwhile, this study proposed to simulate the designed sensors by a spectrum tunable LED system before they were practically manufactured. It should be noted that the above two studies had different research goals, but shared a similar notion of compositing the filters (or sensors) by a set of color primaries. The color primaries in the two studies were the concentrations of solvent dyes, and the intensities of multiple LED channels, respectively.
3.3 Simulation accuracy evaluation
In the preceding sections, the sensor simulation performance was examined by comparing the optimized and simulated spectral sensitivities visually, as well as the spectral reconstruction errors. In this section, it was further evaluated by the discrepancies between the theoretical and real-measured camera responses.
First, the accuracy of the original camera spectral sensitivities was calculated, including the test RGB camera and the monochrome camera used for simulation. Table 5 lists the mean and maximum percentage deviations between the measured and predicted camera responses. It was calculated by dividing the absolute difference by the measured camera response. The MCCC under D65 served as the test color. It indicated the accuracy of the measured camera spectral sensitivities using LEDSSF, and the effect of the potential noise.
Table 5. The accuracy of the spectral sensitivities of the RGB and monochrome cameras, calculated by the percentage errors between the measured camera responses and predicted responses by the spectral sensitivities, tested by the MCCC under D65.
Then we calculated the similar percentage deviations between the real-measured camera responses by sensor simulation and the theoretical camera responses calculated from the optimized sensor sensitivity. Table 6 lists the results, implying the simulation accuracy. For both simulation approaches, the simulated error was slightly larger than the error of the original camera spectral sensitivities. This increased error could be attributed to the inaccuracy of sensor simulation. In general, the simulated errors were in the acceptable range. It provided a convenient approach to evaluate the sensor simulation accuracy in terms of the simulated camera responses.
Table 6. The accuracy of channel-first simulation and illumination-first simulation, evaluated by the percentage errors between the measured camera responses in simulation and predicted responses by the spectral sensitivities of extra channels, tested by the MCCC under D65.
3.4 Discussion
This study presented two distinct approaches for sensor simulation. The first was the channel-first approach to simulate a new sensor channel with a specified shape of spectral sensitivity. The spectral sensitivity could be numerically optimized or designed with any reasonable shape. In this study, three extra sensor channels were optimized for a commercial RGB camera. The responses of the optimized sensor channels were simulated using a multi-channel LED system and a monochrome camera. This was accomplished by matching the illuminants corresponding to the extra channels in the LED system. A sensor matching objective was proposed. It was demonstrated that the LED system can basically simulate the camera responses of extra sensor channels, and evaluate the contributions on spectral reconstruction. However, due to the limited number of the LED channels, the shapes of the desired sensor channels cannot be perfectly replicated. This problem could be refined by using an LED system with more channels in future work.
In the illumination-first approach, the spectrum of the illuminant was optimized, and the related sensor channel could be accordingly determined. This approach was conducted for the sake of simulation simplicity, as it ensured that the LED system could, in theory, perfectly match the corresponding illuminants. So that the extra sensors could be more accurately simulated. However, the spectral sensitivities optimized with this approach were limited by the SPD of the LED channels and the existing monochrome camera that was used for simulation. So that this approach cannot simulate an arbitrary shape of desired spectral sensitivity.
In summary, the experimental results could demonstrate the effectiveness of our proposed methods. It was feasible to use spectrum tunable LED illuminants in conjunction with an existing monochrome camera to simulate a desired spectral sensitivity profile of sensor channel. In future work, it would be beneficial to simulate the sensors with different levels of noise by adjusting the luminance of the matched illuminants. To confirm the optimization and simulation results, more testing samples with distinct categories and wider colorimetric distributions were required. Future research is needed to model the sensor responses under different test illuminants, and to further improve the simulation accuracy, for example, to adopt an LED system with more channels for sensor simulation. If a real multi-channel sensor with intended spectral sensitivities can be manufactured, the simulation accuracy could be further tested by directly comparing the real and simulated sensors in terms of the camera responses and spectral reconstruction errors.
4. Conclusion
This study developed a method to simulate the responses of extra sensor channels and verify the effectiveness on spectral reflectance reconstruction. The method is based on a spectrum tunable LED system. Two different approaches were proposed for simulating the new sensor channels, namely channel-first and illumination-first approaches. The channel-first approach was to optimize the spectral sensitivities of the extra channels directly and then reproduce the responses in a spectrum tunable LED system. The illumination-first approach optimized the lights in the LED system and then determined the extra channels, in order to facilitate a more exact simulation. The practical experiments verified that the proposed methods were effective to simulate the responses of designed sensors, and to evaluate the contributions on spectral reconstruction.
Funding
National Natural Science Foundation of China (61775190).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. R. W. G. Hunt and M. R. Pointer, Measuring Colour (John Wiley & Sons, 2011).
2. J. Liang and X. Wan, “Optimized method for spectral reflectance reconstruction from camera responses,” Opt. Express 25(23), 28273 (2017). [CrossRef]
3. K. Xiao, Y. Zhu, C. Li, D. Connah, J. M. Yates, and S. Wuerger, “Improved method for skin reflectance reconstruction from camera images,” Opt. Express 24(13), 14934–14950 (2016). [CrossRef]
4. X. Zhang, Q. Wang, J. Li, X. Zhou, Y. Yang, and H. Xu, “Estimating spectral reflectance from camera responses based on CIEXYZtristimulus values under multi-illuminants,” Color Res. Appl. 42, 68–77 (2017). [CrossRef]
5. B. Cao, N. Liao, and H. Cheng, “Spectral reflectance reconstruction from RGB images based on weighting smaller color difference group,” Color Res. Appl. 42, 327–332 (2017). [CrossRef]
6. E. M. Valero, Y. Hu, J. Hernández-Andrés, T. Eckhard, J. L. Nieves, J. Romero, M. Schnitzlein, and D. Nowack, “Comparative performance analysis of spectral estimation algorithms and computational optimization of a multispectral imaging system for print inspection,” Color Res. Appl. 39(1), 16–27 (2014). [CrossRef]
7. H. L. Shen, P. Q. Cai, S. J. Shao, and J. H. Xin, “Reflectance reconstruction for multispectral imaging by adaptive Wiener estimation,” Opt. Express 15(23), 15545–15554 (2007). [CrossRef]
8. C. Li and M. R. Luo, “The Estimation of Spectral Reflectances Using the Smoothness Constraint Condition,” in IS&T/SID Ninth Color Imaging Conference (2001), pp. 62–67.
9. W. Yang, D. Zhao, Q. Huang, P. Ren, J. Feng, and X. Zhang, “Classification of emerald based on multispectral image and PCA,” in Proc. SPIE (2005), pp. 684–692.
10. Z. Shi, C. Chen, Z. Xiong, D. Liu, and F. Wu, “HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images,” in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (2018), 1052–1060.
11. Y. Fu, Y. Zou, Y. Zheng, and H. Huang, “Spectral reflectance recovery using optimal illuminations,” Opt. Express 27(21), 30502–30516 (2019). [CrossRef]
12. Y. Xiong, G. Wu, X. Li, S. Niu, X. Han, Y. Lu, Y. Gu, and S. Chen, “Spectral reflectance recovery using convolutional neural network,” in International Conference on Optoelectronic Materials and Devices (2021).
13. Y. T. Lin and G. D. Finlayson, “Physically Plausible Spectral Reconstruction,” Sensors 20(21), 6399 (2020). [CrossRef]
14. B. Arad, R. Timofte, O. Ben-Shahar, Y.-T. Lin, and G. Finlayson, “NTIRE 2020 challenge on spectral reconstruction from an RGB image,” in Conference on Computer Vision and Pattern Recognition Workshops (IEEE, 2020), pp. 1806–1822.
15. R. Shrestha, J. Y. Hardeberg, and A. Mansouri, “One-Shot Multispectral Color Imaging with a Stereo Camera,” Proc SPIE 7876, 1 (2011). [CrossRef]
16. R. Shrestha and J. Y. Hardeberg, “Spectrogenic imaging: A novel approach to multispectral imaging in an uncontrolled environment,” Opt. Express 22(8), 9123–9133 (2014). [CrossRef]
17. A. Mahmoudi Nahavandi and M. Amani Tehran, “A new manufacturable filter design approach for spectral reflectance estimation,” Color Res. Appl. 42, 316–326 (2017). [CrossRef]
18. H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39(35), 6621–6632 (2000). [CrossRef]
19. L. Pronti, P. Ferrara, F. Uccheddu, A. Pelagotti, and A. Piva, “Identification of pictorial materials by means of optimized multispectral reflectance image processing,” in International Workshop on Information Forensics and Security (2015).
20. A. F. Ibrahim, S. Tominaga, and T. Horiuchi, “Spectral imaging method for material classification and inspection of printed circuit boards,” Opt. Eng. 49(5), 057201 (2010). [CrossRef]
21. L. D. Miao, H. R. Qi, R. Ramanath, and W. E. Snyder, “Binary tree-based generic demosaicking algorithm for multispectral filter arrays,” IEEE Trans. on Image Process. 15(11), 3550–3558 (2006). [CrossRef]
22. R. Shrestha, J. Y. Hardeberg, and R. Khan, “Spatial Arrangement of Color Filter Array for Multispectral Image Acquisition,” Proc SPIE 7875, 1 (2011). [CrossRef]
23. R. Shrestha and J. Y. Hardeberg, “CFA Based Simultaneous Multispectral Imaging and Illuminant Estimation,” Lect Notes Comput Sc 7786, 158–170 (2013). [CrossRef]
24. J. Hernandez-Andres, J. L. Nieves, E. M. Valero, and J. Romero, “Spectral-daylight recovery by use of only a few sensors,” J. Opt. Soc. Am. A 21(1), 13–23 (2004). [CrossRef]
25. M. A. López-Álvarez, J. Hernández-Andrés, J. Romero, and R. L. Lee Jr, “Designing a practical system for spectral imaging of skylight,” Appl. Opt. 44(27), 5688–5695 (2005). [CrossRef]
26. M. A. López-Álvarez, J. Hernández-Andrés, E. M. Valero, and J. Romero, “Selecting algorithms, sensors, and linear bases for optimum spectral recovery of skylight,” J. Opt. Soc. Am. A 24(4), 942–956 (2007). [CrossRef]
27. S. Quan, N. Ohta, R. S. Berns, and N. Katoh, “Hierarchical approach to the optimal design of camera spectral sensitivities for colorimetric and spectral performance,” in Proc. SPIE (2003), pp. 159–170.
28. N.-C. Hu and C.-C. Wu, “Optimal selection of commercial sensors for linear model representation of daylight spectra,” Appl. Opt. 47(17), 3114–3123 (2008). [CrossRef]
29. P. L. Vora, “Inner products and orthogonality in color recording filter design,” IEEE Trans. on Image Process. 10(4), 632–642 (2001). [CrossRef]
30. P. L. Vora and H. J. Trussell, “Mathematical methods for the design of color scanning filters,” IEEE Trans. on Image Process. 6(2), 312–320 (1997). [CrossRef]
31. R. Piché, “Nonnegative color spectrum analysis filters from principal component analysis characteristic spectra,” J. Opt. Soc. Am. A 19(10), 1946–1950 (2002). [CrossRef]
32. M. A. T. Monsalve, G. Osorio, N. L. Montes, S. Lopez, S. Cubero, and J. Blasco, “Characterization of a Multispectral Imaging System Based on Narrow Bandwidth Power LEDs,” IEEE Trans. Instrum. Meas. 70, 1–11 (2021). [CrossRef]
33. M. Parmar, S. Lansel, and J. Farrell, “An LED-based lighting system for acquiring multispectral scenes,” Digital Photography Viii 8299, 1 (2012). [CrossRef]
34. R. Shrestha and J. Y. Hardeberg, “Multispectral imaging using LED illumination and an RGB camera,” in 21st Color and Imaging Conference (2013), pp. 8–13.
35. M. Safdar, M. R. Luo, Y. Wang, and X. Liu, “Multispectral Imaging System based on Tuneable LEDs,” in Multispectral Color Science Symposium (2015), pp. 130–134.
36. Z. Liu, K. Xiao, M. R. Pointer, Q. Liu, C. Li, R. He, and X. Xie, “Spectral Reconstruction Using an Iteratively Reweighted Regulated Model from Two Illumination Camera Responses,” Sensors 21(23), 7911 (2021). [CrossRef]
37. J. Liang, K. Xiao, and X. Hu, “Investigation of light source effects on digital camera-based spectral estimation,” Opt. Express 29(26), 43899–43916 (2021). [CrossRef]
38. L. Wang, A. Sole, J. Y. Hardeberg, and X. Wan, “Optimized light source spectral power distribution for RGB camera based spectral reflectance recovery,” Opt. Express 29(16), 24695–24713 (2021). [CrossRef]
39. R. Ramanath, W. E. Snyder, Y. Yoo, and M. S. Drew, “Color image processing pipeline,” IEEE Signal Process. Mag. 22(1), 34–43 (2005). [CrossRef]
40. M. Hironaga and N. Shimano, “Estimating the noise variance in an image acquisition system and its influence on the accuracy of recovered spectral reflectances,” Appl. Opt. 49(31), 6140–6148 (2010). [CrossRef]
41. E. Moore, “On the reciprocal of the general algebraic matrix,” Bull. Am. Math. Soc 26, 394–395 (1920).
42. R. Penrose, “A generalized inverse for matrices,” Math. Proc. Cambridge Philos. Soc. 51(3), 406–413 (1955). [CrossRef]
43. “Colorimetry Part 6: CIEDE2000 Color-Difference Formula,” ISO/CIE 11664–6:2014(E).
44. J. Hernandez-Andres, J. Romero, A. Garcia-Beltran, and J. L. Nieves, “Testing linear models on spectral daylight measurements,” Appl. Opt. 37(6), 971–977 (1998). [CrossRef]
45. J. Romero, A. GarciaBeltran, and J. HernandezAndres, “Linear bases for representation of natural and artificial illuminants,” J. Opt. Soc. Am. A 14(5), 1007–1014 (1997). [CrossRef]
46. MathWorks, “ https://ww2.mathworks.cn/help/optim/ug/fmincon.html”.
47. H. Fan, C. He, T. Huang, and M. R. Luo, “A new method for measuring the spectral sensitivities of a camera,” in Preceedings of AIC (2022).
48. N. Shimano, “Recovery of spectral reflectances of objects being imaged without prior knowledge,” IEEE Trans. on Image Process. 15(7), 1848–1856 (2006). [CrossRef]
49. C. S012/E:2004, “Standard Method of Assessing the Spectral Quality of Daylight Simulators for Visual Appraisal and Measurement of Color,” in Vienna, (CIE Central Bureau; Austria, 2005).
50. J. Liang, X. Wan, Q. Liu, C. Li, and J. Li, “Research on filter selection method for broadband spectral imaging system based on ancient murals,” Color Res. Appl. 41, 585–595 (2016). [CrossRef]
