1. Introduction

During the last decade, lighting has experienced its long overdue renaissance as an integral part of the architectural design process. Light in general has been shown to act as a mediator of various visual, biological, and behavioral responses in humans—both on a short- and long-term timescale—that not only determine how the lit environment is experienced by its occupants but also have a considerable effect on human health and well-being. Thus, the goal of any sustainable lighting design process must be to create adequate lighting conditions that, interacting with the architectural space, are tailored to match the visual, physiological, and emotional needs of the human users at any time during the day. This fundamental approach is also known as integrative or human-centered lighting (HCL) [1–3].

For the creation of lit environments that, in this context, not only satisfy minimal requirements for energy savings and visual task performance but also promote health, well-being, and a maximum of emotional comfort in the human user over time, it is therefore essential that modern lighting solutions are capable of monitoring the crucial lighting parameters in order to dynamically compensate for temporal changes in the environmental conditions that may cause severe deviations from their optimal values [4–7]. In recent years, multi-band color sensors have become more and more popular in both laboratory and field applications to be used as low-cost measurement devices for assessing the colorimetric, photometric, spectral, and even physiological properties of a light source or an illumination.

Based on such a multi-channel approach, Amirazar et al. [8] for example developed a wearable low-cost spectrometer intended to monitor a person’s individual lighting exposure in real time. A multi-layer perceptron (MLP) architecture was used to set up the artificial neural network (ANN) for reconstructing a light source’s relative spectral power distribution (SPD) from corresponding spectral-sensor readouts. However, despite their efforts in maximizing the overall estimation accuracy by testing several different activation functions and learning algorithms to derive the final ANN, a rather poor to moderate performance was observed on real-world data: Normalized root-mean-squared error (NRMSE) rates ranged from 11 to 40 % [8], with the largest deviations between measured and reconstructed SPDs being observed for daylight conditions.

Instead of attempting to directly reconstruct the complete SPD of a light source from color or spectral sensor readouts, Botero et al. [9] confined themselves to the estimation of certain color rendition features, such as TM 30-18 $R_{\mathrm {f}}$ and $R_{\mathrm {g}}$ values or the CIE $R_{\mathrm {a}}$ color rendering index. They tested both MLP and extreme learning machine (ELM) architectures for setting up the respective ANNs. The absolute metric prediction error reported for the best performing ANN was less than 1 % with an overall correlation between the actual and estimated values close to 99 % [9]. However, it must be emphasized that these excellent results were obtained for simulated RGB and multi-band sensor responses only and not for real sensor data. As the latter usually offer less controllable preconditions, considerably worse performance can be expected in this case.

The same group of Botero el al. [10] also proposed a $k$-nearest neighbor approach to classify artificial light sources into one of the incandescent, fluorescent, or LED-type categories using trichromatic responses from low-cost RGB color sensors. A high classification accuracy of more than 96 % [10] was reported for a sample of 54 different light sources commonly found in residential and commercial environments. In addition, cluster-specific regression models were applied successfully to estimate correlated color temperatures (CCTs) and color rendition measures. In an earlier work, the same authors [11] used linear regression to first transform the RGB sensor outputs to CIE $XYZ$ tristimulus values before applying McCamy’s approximation method [12] to calculate corresponding CCT values from $XYZ$. CCT estimation errors of less than 6 % [11] were reported for a selection of typical indoor light sources.

Another application scenario of color sensing was considered by Agudo et al. [13]. Based on the Arduino architecture, they developed a portable low-cost color-picking device for the extraction and management of color information from any non-self-luminous object by using an RGB color sensor with an integrated high-power white LED. Color calibration of the system was performed by finding an optimized $3\times 3$ transformation matrix to convert from RGB sensor data to corresponding $XYZ$ tristimulus values. Even though the residual chromaticity errors clearly exceeded those observed when using proper scientific equipment, Agudo et al. emphasized that the reported performance accuracy might still be sufficient for many non-scientific use cases.

Trinh et al. [14] recently introduced a new methodology for estimating the circadian effectiveness of both daylight and artificial light sources in terms of the physiologically relevant circadian stimulus (CS) metric by using simple and low-cost RGB color sensors. For daylight conditions, CS prediction errors of less than 0.0025 [14] were observed when tested on real data. For artificial light sources, on the other hand, the maximal CS prediction error increased to 0.028, which, although being considerably larger compared to daylight, was concluded by the authors to be still in an acceptable range for typical indoor lighting applications.

Other fields of application where low-cost RGB color sensors have been applied successfully for monitoring and control purposes are biomass cultivation and precision agriculture. Benavides et al. [15] for example developed a color-sensor-based setup for the online measurement of microalgae concentration within a photo-bioreactor. Compared to commercially available, application-ready spectrophotometric solutions, a similar performance and accuracy in predicting the biomass concentration of microalgae cultures could be achieved with their proposed system at a much lower price. However, a large-scale validation of their approach is still pending. In a different yet related setting, Brambilla et al. [16] used a low-cost RGB color sensor to monitor the nutritional status of basil plants that were administered different doses of nitrogen fertilizer by considering their leaf color as an index for nutrient diagnosis. Compared to commercial solutions, such as optical leaf meters, the proposed system still showed some weaknesses as it performed worse in assessing the nutritional status for increasing levels of supplied nitrogen than the reference device.

Apart from the application scenarios discussed so far, where color sensors were used as low-cost measurement devices to approximate certain aspects of spectroradiometric functionality, optical sensors with three or more distinct spectral channels have also been applied successfully for the development of closed-loop lighting control schemes in multi-primary settings [17]. Breniuc et al. [18] for example used an empirically derived formalism to directly calculate illuminance and CCT values from the raw sensor readouts of an RGBC sensor that was optically coupled to a tunable-white LED-based lighting device. The ’C’ stands for a clear channel, which approximately shows the spectral sensitivity of the underlying silicon photodiode and, in combination with the remaining color channels, allows for an estimation of the lighting’s infra-red component, for details see Breniuc et al. [18]. After proper system calibration, maximal deviations from target CCT values were found to be less than $\pm{25}$ K across various illuminance levels [18]. It should be noted that these results were obtained under strictly controlled experimental conditions excluding any ambient or external light sources, which, however, are usually present in real-world applications and interfere with the sensor’s predictions. Thus, Breniuc et al.’s work only allows for concluding the feasibility of color-sensor-based lighting control on a pilot level.

More complex lighting control scenarios, on the other hand, have been addressed by Chew et al. [19] and Maiti and Roy [20]. Both research groups used data from optical color sensors to model human trichromatic responses directly in the feedback loop for the implementation of a more vision-related online optimization. This procedure basically resulted in high performance accuracy, where maximum deviations in chromaticity from highly-dynamic daylight patterns and other time-varying target spectra were found to fall well within a five-step $\Delta u'v'$ unit circle [19]. In addition, corresponding deviations in CCT [19,20] and illuminance [20] were reported to be smaller than 5 % of their target values for all test settings.

Finally, Ashibe et al. [21] proposed a spatially distributed lighting control scheme based on color sensor feedback that allows for the realization of different illuminance levels and target chromaticities at different room locations to match the lighting preferences of the individual user without the need of introducing additional, workplace-related lighting fixtures, such as floor or desk lamps, to supplement the ceiling lights. Their test setting comprised a total of fifteen tunable RGB fluorescent tube luminaires mounted to the ceiling of a 9 m $\times$ 9.6 m experimental room and two spatially separated RGB color sensors used for position-specific lighting control. Despite the system’s limited spectral capabilities, Ashibe et al. were able to show convergence for a certain variety of distinct illuminance and chromaticity combinations given as target conditions for the two different sensor locations, which provided evidence for the general feasibility of their proposal.

As can been seen from this brief summary on the use of multi-band color sensors in the broader lighting context, most of the previous work found in the literature considered RGB-like sensors for modelling and predicting trichromatic responses and/or integral measures of lighting quality only. Unfortunately, for the development of proper human-oriented lighting solutions, this appears to be insufficient. Instead, the key requirement for any intelligent system design that addresses all aspects of HCL is to know the complete spectral characteristics of the composed lighting conditions at any point in time. For example, the CIE system for metrology of optical radiation for ipRGC-influenced (intrinsically-photosensitive retinal ganglion cells) responses to light (CIE S 026/E:2018) [22] bases its informative value for the description of non-image-forming (NIF) effects through photoreceptor stimulation on computations from the SPD of the lighting condition under consideration. Spectroradiometric approaches, however, despite offering high accuracy, usually require expansive and bulky measurement equipment, which makes them unsuitable for use in the general lighting context. Latest advances in color sensing technology fortunately have led to the development of a new generation of low-cost, multi-band (i.e., with considerably more than three spectral channels) color sensors whose electro-optical response behavior to arbitrary light stimuli can, in theory, be used for sufficiently accurate SPD estimations in the field [8,23–25].

The present work thus aims at bridging the gap between theory and application: It introduces a new methodology for estimating the spectral irradiances of real illumination scenarios from spectral sensor responses by using—in this combination—a novel convolutional neural network (CNN) approach. CNNs stem from the field of object detection and recognition in computer vision and excel in locating known (from training) patterns in images. For those tasks CNNs usually outperform simpler network architectures like MLPs or ELMs. Their main advantages are the automatic extraction of meaningful features from the data as well as their computational efficiency due to equivalent representation, sparse interactions, and parameter sharing [26,27]. We therefore employ the convolutional blocks of typical CNNs in our reconstruction network under the hypothesis, that spectral reconstruction from normed sensor responses can be interpreted as the task to recognize patterns from lower dimensional data and project those features to an output with higher spectral resolution.

Based on this approach, the present work demonstrates for the first time the true potential of applying multi-channel optical sensors for the proper (i.e., on an absolute level) spectral reconstruction of real-world lighting data. Realistic illumination scenarios comprising combinations of LED, fluorescent, tungsten, and daylight lighting will be considered to allow for a comprehensive performance analysis.

2. Materials and methods (theory and calculation)

Starting from the basic model formalism of a multi-channel optical sensor, this section introduces the proposed CNN framework for estimating a light source’s SPD from corresponding sensor responses. In addition, a brief summary of the MLP should be given, as this represents another type of neural network architecture that has recently been applied successfully for similar spectral reconstruction tasks. A performance comparison between both architectures will eventually be part of the results section of this work. All of the subsequent chromatic calculations were performed using LuxPy. For network model definition and training Keras was used in conjunction with the Tensorflow backend. Efficient and functional network deployment for online (edge) computations is a topic of active research and development. In theory, networks similar to the ones trained in this work can be deployed to embedded systems like microcontrollers directly by converting them to a TensorFlow Lite model, whose byte array representation can be loaded from device memory and used for inference with the TensorFlow Lite for Microcontrollers C++ library. However, dealing with the details and obstacles of such an embedded system integration goes beyond the scope of the present work, which, at this point, focuses on providing a solution to the problem itself, namely the proper reconstruction of spectral information from multi-channel spectral sensor responses.

2.1 Multi-channel spectral sensor model

The electro-optical response behavior of a multi-channel spectral sensor can be described by Eq. (1) [28]. As can be seen, the output value $c_k$ of the $k$th sensor channel is modelled by a non-linear function $F$ depending on the integration time $e$, the sensor’s gain factor $\kappa$, and the response signal $S$. Here, the latter is given by the integral over the spectral irradiance $\phi (\lambda )$ weighted by the channel’s spectral sensitivity function $s_k(\lambda )$. The term $n_k$ in $S$ represents additive noise that is imposed on the response signal.

In this work, the AS7341 optical sensor from ams-OSRAM AG (Premstaetten, Austria) with 8 spectral channels in the visible range, which are supplemented by an additional clear and near-infrared (NIR) channel, serves as the input device for performing the spectral reconstruction task. With regard to a later application in the general lighting context, the bare sensor was equipped with a diffusion foil and an IR cutoff filter, both mounted to a small optical tube attached to the sensor’s circuit board and aligned with the sensor’s optical axis. The tube aperture thus defines the sensor’s field of view (FOV). The corresponding relative spectral sensitivities of the optical sensor system are shown in Fig. 1. These sensitivities were measured using an MSH 300 monochromator setup (Quantum Design GmbH, Darmstadt, Germany) with a 300 W xenon arc lamp [14]. For proper sensor characterization, the light emission was varied from 380 nm to 780 nm in steps of $\Delta \lambda = 1$ nm, where each of the resulting probe stimuli showed a full-width at half-maximum (FWHM) of approximately 3 nm. The exit slit of the monochromator was connected to the input port of a six-inch integrating sphere (Labsphere Inc., North Sutton, NH, USA) comprising a highly reflective white PTFE coating. One of the sphere’s exit ports held the sensor, while the other was attached to a Spectro 320D R5 spectroradiometer (Instrument Systems GmbH, Munich, Germany) to allow for the simultaneous measurement of the relative spectral irradiance of the probe stimulus and the resulting sensor counts, from which the spectral sensitivities of each sensor channel can eventually be calculated, see e.g. Myland et al. [7] for further details.

 

Fig. 1. Relative spectral sensitivities of the used sensor with mounted optical assembly consisting of a tube, diffuser foil and IR blocking filter, which are discussed in further detail in subsection 2.4. Shown are the eight narrow band channels of the visible wavelength regime, the clear channel, and the residual sensitivity of the NIR channel for wavelengths smaller than the IR blocking filter cut-off at approximately 750 nm. This cut-off is indicated by the diminishing clear channel sensitivity for greater wavelengths.

Download Full Size | PDF

Here, it should be noted that, due to the supervised nature of the applied learning approach, knowing the sensor’s actual spectral sensitivities does not pose a necessary precondition for the training of the neural network and its subsequent application for spectral reconstruction in case one is only interested in the determination of relative SPDs. However, as it will been seen later in Section 2.6, knowledge of the sensor’s spectral response behavior is required when trying to convert from these relative SPDs to measures of absolute spectral irradiance.

2.2 Spectral reconstruction with multi layer perceptron network architecture

The MLP is a feed-forward network with universal approximation capabilities [29], where data flows from the input trough the hidden layers to the output. Each layer node receives the outputs of the previous layer, combines them in a weighted sum, adds a bias vector, and applies a mathematical function to the result. Key parameters for the MLP are the number of (hidden) layers between input and output, the number of nodes (neurons) in each layer, and the applied (activation) function. Training the network for a supervised regression task means to find the weights and bias vectors for all the nodes in the network, so that the network as a whole predicts the output for a given input with minimal error compared to the true, expected output of a set of training samples. The most common learning algorithm for MLPs is back propagation [30].

MLPs using a sigmoidal (nonlinear) activation function can approximate any continuous multivariate function to an arbitrarily exact degree [31–33]. With the rising availability and quality of multi-channel spectral sensors first steps towards the development of low-cost radiometric measurement devices were taken by pairing these sensors with the approximation capabilities of MLP networks.

Here, Botero et al. [23] were among the first to show the general feasibility of this approach by looking at simulated sensor responses from a total of 10 distinct narrow-band sensor channels, whose spectral sensitivities were distributed over the complete visible regime. They employed a 4-layered MLP network with 25 neurons per layer and a sigmoidal tangent activation function to calculate a $1\times 81$-dimensional output vector, representing the light source’s relative SPD, from the $1\times 10$-dimensional sensor input (cf. Fig. 2). For the training data, which consisted of approximately 42 000 light source spectra generated from pairwise mixing of a representative selection of 84 real-measured light source base spectra, a typical NRMSE between original and reconstructed SPD of less than 2 % could be obtained. Unfortunately, no further testing of the proposed MLP architecture was performed regarding unknown light sources that were not part of the training data. However, as for example reported by Amirazar et al. [8], who adopted a similar MLP-based spectral reconstruction approach for assessing individual light exposure (see also section 1.), considerably larger errors must be expected when being applied to such unknown lighting conditions.

 

Fig. 2. Network architecture of the spectral reconstruction MLP as used in [23], with 10 sensor channel inputs, 4 hidden layers with 25 fully connected neurons each, sigmoid activation functions and 81 output nodes for wavelengths between 380 and 780 nm.

Download Full Size | PDF

2.3 Spectral reconstruction with convolutional neural networks

Convolutional neural networks (CNNs) have achieved a fundamental place in machine learning by excelling in a number of image processing benchmarks [34].

In CNNs the convolution operation is used with a number of filter kernels on the input data to compute so-called feature maps as an abstraction step in the computation task [35]. As with the MLP neuron, the standard configuration of most convolutional layers is to add a bias vector first and eventually apply a mathematical function to the result. This abstraction is then reduced in dimension by a pooling operation that combines nearby values to achieve invariance to small local shifts and merge semantically similar features. Convolutional layers and pooling layers can be repeated alternately to increase the depth and abstraction ability of the network [29,34].

The output layers usually are fully connected. Key parameters for the CNN are the number of convolution and pooling blocks, the number and size of the convolution kernels, the size of the pooling windows, the stride of the convolution and the pooling operators, the activation functions, as well as the number and width of the fully connected output layers. Training the network for a supervised regression task means to find the weights in the filter kernels of the convolutional layers, the bias vectors of the convolutional layers, and the weights and bias vectors for all the fully connected nodes in the output block of the network, so that the network as a whole predicts the output for a given input with minimal error compared to the true, expected output of a set of training samples.

Although CNNs became popular with image inputs and 2D convolution, the methodology can also be applied on 1D data (with 1D convolution). In this work, we demonstrate for the first time the applicability of such a 1D-CNN model (cf. Fig. 3) for the reconstruction of an arbitrary illumination’s SPD from multi-channel spectral sensor data.

 

Fig. 3. Network architecture with 10 sensor channel inputs, employing a single convolutional layer with 150 filters of kernel size 9, a pooling layer, a fully connected hidden layer with 96 neurons (relu activation) and 201 output nodes for wavelengths between 380 and 780 nm.

Download Full Size | PDF

As with the previous work from other authors, the network itself operates on max normed sensor data to estimate the relative spectral irradiance for wavelengths between 380 and 780 nm: For a successful training of any neural network it is often favorable to normalize the input and output data [36,37]. In our application scenario, the benefit of normalization can be explained by the inherently relative nature of human color perception. In colorimetry, the relative distribution of energy over the wavelengths of the input spectra determines the perceptual correlates and colorimetric measures like the color locus or the illumination’s CCT. When aiming at reconstructing spectral information with minimal colorimetric errors, it is therefore convenient to train the network focused on those relative input and output relations. Furthermore, the operation on normalized data drastically reduces the amount of required training samples for setting up the network (otherwise, sensor samples and light spectra for a huge variety of spectral irradiance levels would be required to train the absolute reconstruction network with any hope of generalization). Nonetheless, it should be noted, that this initial reduction in complexity does not rule out a later estimation of the illumination’s absolute SPD from the same sensor responses. In subsection 2.6, a dedicated procedure to convert from relative SPDs to absolute measures of spectral irradiance will be introduced accordingly.

2.4 Real world data generation

The data for network training were collected on different days between August 6th and August 24th, 2021. Validation data, on the other hand, were acquired between August 16th and August 25th, 2021. In both cases, the AS7341 spectral sensor in conjunction with a small spectroradiometer (CSS45, Gigahertz Optik GmbH, 82299 Tuerkenfeld, Germany) were used for data collection. As can be seen from Fig. 4(a), both devices were mounted to an office floor lamp facing perpendicularly down towards the user’s lit working area. A floor lamp was chosen as test platform because of its portability (data capturing in multiple locations) and widespread use in office lighting applications, especially for lighting concepts, where local/individual light is provided at the workplace supplementing the general room lighting conditions.

 

Fig. 4. Spectrometer and sensor setup: (a) Mounted to the floor lamp facing downwards, (b) close-up of spectrometer and sensor geometry and (c) separate components of the sensor assembly with tube, filter, diffuser, tube holder and sensor.

Download Full Size | PDF

For data acquisition, the floor lamp was placed in three different locations (cf. Fig. 5): A conference room, a working place close to a window in a lab and a working place on the windowless side of the same laboratory room.

 

Fig. 5. Image representation of the three measurement locations of the floor lamp setup including the attached spectrometer and spectral sensor: (a) Laboratory workplace facing a north-west window, (b) laboratory workplace facing away from any windows, and (c) meeting room table next to a south-east window. From (b), the fluorescent lamps of the ceiling can also be seen.

Download Full Size | PDF

The floor lamp is equipped with three individually addressable LED channels (warm white, cool white and cyan) and comprises separate light-emitting surfaces for both down and up light functionality, which can also be controlled independently. Sweeps of the three individual down channels as well as random mixtures of all six channels (up- and downlights) were recorded in darkness (blinds closed) and under different ambient lighting conditions as provided by different mixtures of i) daylight entry through the window, ii) light coming from the fluorescent ceiling lamps, and iii) light emitted by a tungsten table lamp. Corresponding data samples, in each case, eventually consist of a set of sensor channel responses, the measured spectral irradiance (from 380 nm to 780 nm in 1 nm steps), and a coded semantic notation of the individual mixture components.

The sensor was used with a gain of 64x and an integration time of 728.75 ms, while, as recommended by the manufacturer and in view of its potential later use in general lighting applications, a diffuser sheet and an IR blocking filter were held in place in front of the sensor by a solid angle limiting tube (cf. Fig. 4(c)). The use of an IR filter (FESH750, Thorlabs, Inc., Newton, New Jersey, United States) with a cut-off wavelength of 750 nm was necessary because of the non-negligible residual response behavior to infrared light observed for the bare sensor channels, which strongly impaired the spectral reconstruction in the visible region (i.e., between 380 nm and 780 nm) for light sources containing infrared components. Another possible approach to deal with this problem is to acquire a light source’s spectral irradiance for a larger wavelength range that complies with the complete sensitivity range of the underlying silicon diodes. However, no portable spectroradiometer with such an ample measurement range was available by the time of this study.

The tube, which was put in place in front of the sensor, limits the angle under which incoming light enters the IR cutoff filter and, therefore, ensures spectral filter transmission as shown in Fig. 6. It exhibits a height of 15 mm and an inner diameter of 17.2 mm resulting in a solid viewing angle of 59.65° as seen from the bottom center of the tube. In order to have both sensor and spectroradiometer collect and register light from approximately the same solid angle, the latter was also equipped with an accordingly dimensioned tube (cf. Fig. 4(b)).

 

Fig. 6. Transmission of diffuser and filter measured against a FEL lamp. The filter edges at 385 nm and 750 nm are clearly visible. The diffuser foil has only a small spectral impact in the pass band of the filter.

Download Full Size | PDF

Collected data pairs with an illuminance of less than 25 lx, which corresponds to approximately 2.5 lx on the sensor’s surface (i.e., after passing the diffuser and filter planes), were excluded because of an insufficient signal-to-noise ratio for meaningful reconstruction results. This yields a total number of 418 omitted data points. Here, it should be noted that such low-light conditions might have been captured by further increasing sensor gain or exposure time. However, for this work, both of these sensor parameters were kept fixed to avoid possible problems such as gain non-linearities, varying noise levels, and timing mismatches between spectroradiometer and sensor measurements. Ultimately, 876 light spectra/sensor response pairs have been selected for training and another 1338 for validation purposes.

With regard to the training data, the following data-balancing procedure is applied to avoid sample bias and ensure proper spectral reconstruction for all—and not only for the most common—mixture combinations in the original data set: For each coded semantic notation of the mixture components, 80 samples are randomly selected. If not enough of such unique entities are available for a certain mixture combination, the missing data pairs are randomly selected with repetition from the available sample subset until sufficient data pairs (i.e., 80 samples in total) have been added to the training set for this specific mixing category.

The 1338 validation spectra, on the other hand, are randomly divided into two subsets of equal size. The first one is used for evaluating the performance of the network parameter determination (cf. subsection 2.5), whereas the second one is used for network comparison and final performance testing (cf. section 3.). Note again, that these validation data were captured on different days than the data used to train the network and that the workplaces serving as measurement locations were occupied frequently for other purposes between the different days of data acquisition so that the exact positions of the floor lamp and measurement devices were subject to variations similar to those expected for real-world applications in the office lighting context.

2.5 Network parameter determination

The straightforward way of finding optimal hyperparameters is to look through a multidimensional grid of all hyperparameter combinations within given parameter boundaries. This approach is called grid search [38]. The grid and, thus, the required number of network training runs to find an optimal combination of parameters grows exponentially with the number of parameters to be considered. This specific approach therefore appears to be suboptimal for network parameter determination with many parameters, as for example required for the CNN architectures considered in this work. Naturally, a lot of computation time will be spent on grid locations of low importance, while possibly undersampling areas with potent hyperparameter combinations because of the chosen grid step size [39].

For large search spaces, like the ones considered here, evolutionary algorithms may represent an excellent alternative [38]. Inspired by genetics, an evolutionary search can be performed by sequentially selecting, combining, and randomly varying parameters in a very efficient way that resembles biological evolution.

In this work, an even more efficient approach is used for parameter determination called hyperband tuning [40], which is implemented in Keras Tuner [41]. This bandit based approach trains a large number of models for a few epochs and keeps only the best performing 50% of the models for the next round. With each round, the models are given more training epochs which results in quick convergence to a high performing model.

Table 1 eventually shows the optimal parameter combinations for the task of spectral reconstruction with the MLP and CNN architectures introduced in the preceding sections. For the corresponding hyperband parameter determination runs, a maximum of 1000 epochs per model were allocated for training with a reduction factor of 3 and 10 hyperband iterations, respectively. In each case, the MSE validation loss between reconstruction and sub-sampled ground truth spectra was evaluated as optimization criterion.

Table 1. Tuned parameters for the CNN models with 1 or 2 convolution blocks and the MLP model. The last column describes the search space for the individual parameters with start, stop, and increment.

View Table | View all tables in this article

2.6 Estimating absolute spectral power distributions

The procedure to extend the relative spectral reconstruction results from the various network approaches to absolute SPDs can be split into two consecutive steps: First, the absolute spectral sensitivities of the sensor (including all optical elements) have to be determined. One option would be to compute the absolute channel sensitivities directly from the monochromator measurements (see subsection 2.1) by using a calibrated photodiode placed at the exact sensor position in front of the integrating sphere to determine the irradiance of the monochromatic probe stimuli for every wavelength step. In this work, however, we employ a more practical approach by using the data set of training spectra $\phi _{i}(\lambda )$ to compute the channel scaling factors $\delta _{k}$ that minimize the error between real $c_{k,i}$ and computed sensor responses $\hat {c}_{k,i}$ as obtained from scaled relative spectral sensitivities $s_k(\lambda )$:

The second step for estimating absolute SPDs then aims at determining suitable re-scaling factors $\eta _{i}$ that can be applied to the relative network output $\hat {\phi }_{i}(\lambda )$ to yield accordingly rectified spectral estimates $\phi _{i,abs}$. With the previously determined absolute spectral sensitivities $s_{k,abs}$, virtual absolute sensor responses $\hat {c}_{k,i,abs}$ can be computed for each $\hat {\phi }_{i}(\lambda )$. Dividing the real sensor responses $c_{k,i}$ by this virtual sensor responses and averaging over the total number of sensor channels eventually gives a single re-scaling factor $\eta _{i}$. Finally, this factor is applied to the reconstructed relative light spectrum $\hat {\phi }_{i}(\lambda )$ to obtain an estimate for $\phi _{i,abs}$ from the relative network output. The corresponding equations read:

The overall performance of this estimation is evaluated in section 3.

3. Results

This section summarizes the results of the proposed network-based spectral reconstruction approach applied to real-world lighting conditions as discussed in subsection 2.4. When it comes to the evaluation of colorimetric errors, care must be taken in choosing a proper metric formalism. Albeit being popular, CCT and Duv (distance of the light source chromaticity to the Plankian locus) are known to be imprecise in describing human color difference perception of spectrally "structured" sources [42] (e.g. LED mixtures). Furthermore, the CCT is a nonlinear metric, the difference between lower CCT values (e.g. 3000 to 3200K) is perceived much more prominently than between higher CCTs (e.g. 6500 to 6700K). Following the current CIE recommendation regarding chromaticity difference specification for light sources (CIE TN 001:2014) [43], the CIE 1976 UCS color space with its coordinates $u'v'$ is the most appropriate model to describe the color of light and color differences between light sources. Fig. 7 shows for each mixing combination the test spectrum exhibiting the largest $\Delta u'v'$ error. Note that the results shown here are limited to those cases where the average $\Delta u'v'$ error for the respective mixing group is greater than 0.0015. The illustrated light spectra thus represent worst case scenarios in terms of color deviations from their corresponding ground truth that must be expected to occur when applying the proposed reconstruction approach to real-world data. The semantic annotations for the various mixing types given in the titles of the subplots encode the individual mixing components (up to three, comma-separated) of the respective lighting conditions. For decoding, please refer to Table 2.

 

Fig. 7. Worst case spectral reconstruction results of the test data for the various ANN approaches (dotted lines) in terms of color deviations (largest $\Delta u'v'>0.0015$ per mixture type) from their ground truth spectra (green solid lines). For orientation, the $\Delta u'v'$ and nRMSE error measures corresponding to the 1 block CNN configuration are given in the legends. The subplot titles encode the mixture components according to Table 2.

Download Full Size | PDF

Table 2. Illuminant abbreviations used to shorten the description of mixture components.

View Table | View all tables in this article

In each case, results are shown for the three different network configurations MLP, 2 block CNN, and 1 block CNN introduced in subsection 2.5. Additionally indicated for the 1-CNN configuration are corresponding nRMSEs and $\Delta u'v'$ scores.

The term nRMSE is used here to denote the RMSE calculated between two spectra normalized to their respective maximum. Note that this has to be distinguished from the standard NRMSE definition as for example used by Amirazar et al. [8] and Botero et al. [23]. Since normalization to the maximum is most common practice in lighting research and application for providing spectral and/or colorimetric information of a light source or lighting condition, the proposed nRMSE definition appears to offer the greater value for assessing spectral reconstruction quality.

For an overview, Fig. 8 illustrates the resulting errors of the spectral reconstruction obtained for the test data set by using representative boxplots. As can be seen, the various lighting conditions are divided by their respective mixture types and categorized into five different illuminance classes covering the range from 0 lx to 458 lx with an equal illuminance spread, i.e., illuminance class 1 entails lighting conditions with $0\;\textrm{lx} \leq E_{\mathrm {v}} < {114.5}\;\textrm{lx}$, illuminance class 2 with ${114.5}\;\textrm{lx} \leq E_{\mathrm {v}} < {229}\;\textrm{lx}$, and so on. It should be noted that the annotation template discussed above has been extended accordingly by a preceding number indicating the illuminance class of the lighting mixture.

 

Fig. 8. Performance boxplots by illuminance and mixture groups (encoded according to Table 2) for the 1 block CNN configuration. Indicated illuminance classes cover the range from 0 lx to 458 lx with an equal illuminance spread, i.e., illuminance class 1 entails lighting conditions with ${0}\;\textrm{lx} \leq E_{\mathrm {v}} < {114.5}\;\textrm{lx}$, illuminance class 2 with ${114.5}\;\textrm{lx} \leq E_{\mathrm {v}} < {229}\;\textrm{lx}$, and so on. Table 3 gives the numerical overview of the data plotted here.

Download Full Size | PDF

In this context, Fig. 9 depicts the worst case scenarios of maximal illuminance error observed from reconstructing absolute light source spectra by applying the proposed methodology. Note that only mixture types with an illuminance error of 3.5 lx or greater are shown here.

 

Fig. 9. Worst case reconstruction results of the test data for the 1 block CNN configuration (dotted blue lines) in terms of irradiance deviations from their ground truth spectra (green solid lines). Additionally shown are the corresponding $\Delta u'v'$ and $\Delta E_{v}$ error measures. The subplot titles encode the mixture components according to Table 2.

Download Full Size | PDF

Table 3 finally summarizes the mean and maximum chromaticity ($\Delta u'v'$), illuminance ($\Delta E_{\mathrm {v}}$) and spectral (RMSE calculated on normalized spectra) errors as quantifiable outcome measures for the different mixture types encoded in the table’s first column. Here, columns denoted with an overline represent the mean results of the corresponding evaluation measures, whereas the hatted columns contain the respective metric’s maximum values.

Table 3. Mean and maximum $\Delta u'v'$, nRMSE, absolute and relative illuminance error tabulated by spectral type (mixture components). Columns with an overline represent the mean results of the corresponding metric, whereas the hatted columns contain the maximum values.

View Table | View all tables in this article

4. Discussion

4.1 General aspects

As can be seen from Fig. 7, all three different network configurations are able to deliver plausible reconstruction results, even for the worst case scenarios considered here. However, a slightly worse overall performance should be noted for the MLP when compared to both CNN approaches. With the 1 block CNN model giving very similar results as its counterpart with 2 convolution blocks but at a considerably reduced number of required network parameters comparable to the MLP approach (see again Table 1), the subsequent performance analysis focuses on the 1 block configuration only.

Except for the two lighting conditions "pcLED_WW, IndrDlght,-" and "pcLED_CW,IndrDlght, - ", which exhibit considerably larger nRMSE errors for all network configurations (see also subsection 4.2 for further discussions), most of the shown worst-case reconstruction results still agree reasonably well with their measured ground truth spectra in terms of both spectral nRMSEs and chromatic $\Delta u'v'$ scores. As can be seen, only the "pcLED_CW,IndrDlght,-", "LEDmix,IndrDlght,-", and "LEDmix,Flr,-" mixtures exceed a chromaticity error of $\Delta u'v'=0.003$, which is commonly considered as the perceptional threshold in general lighting applications [44–46]. Interesting to note is that, despite showing a notable deviation between reconstruction and direct measurement reflected by a relatively large nRMSE, the "pcLED_WW,IndrDlght,-" mixture is reconstructed with a $\Delta u'v'$ below this chromaticity threshold. This finding can be explained by the relative nature of color perception: As long as the spectral balances, i.e., the ratios between light components of different wavelengths, agree reasonably well within the range of the CIE color matching functions modelling human color perception, the perceived chromatic content of the illumination stays approximately the same leading to negligibly small chromaticity errors.

In summary, it can be stated that the methodology for reconstructing spectral information of real-world lighting conditions from real sensor data proposed in this work shows a positive overall performance, where even the worst case scenarios yield error scores that are still of the same order of magnitude as reported by previous studies focusing on theoretical considerations and synthetic data only (see e.g. [8,23]).

The positive model performance concluded from exploring the worst case scenarios of Fig. 7 can also be seen in further detail from Fig. 8: Only three of the various mixture types considered in this work yield $\Delta u'v'$ reconstruction errors greater than 0.003; and this only for a very limited number of cases, so that, in general, a sufficiently accurate reconstruction performance can be expected when applying the proposed methodology. In addition, variations in the illuminance level and, thus, in the sensor’s signal level, do not seem to affect the colorimetric reconstruction accuracy for the examined set of real-world lighting data.

Regarding the error in illuminance determination (see subsection 2.6 for reference), there appears to be a slight tendency to overestimate its true value by up to 9 %, which is primarily observed for tungsten mixtures. As can be seen from Fig. 9, mixtures comprising a tungsten component show considerably larger errors when estimating their absolute SPDs, leading to the observed illuminance deviations from ground truth. Moreover, tungsten mixtures of higher total illuminance exhibit significantly larger illuminance error variances than just the tungsten source without any further lighting components. From these findings, it can be concluded that, despite using an additional 750 nm cut-off filter, the residual infrared and off-band sensitivties of the sensor channels still have a non-negligible, perturbing impact on the corresponding response signals used as input for the spectral reconstruction. In case of dealing with lighting conditions that comprise a considerable amount of infrared components, the sensor’s channel responses are no longer a direct measure for the energy deposited in the narrow wavelength bands determined by the channels’ respective sensitivity maxima but are superimposed by an offset of broader wavelength origin. This offset affects to some extend all sensor channels at the same time leading to the slight overestimation of absolute SPDs and illuminances observed from Fig. 9. This basically implies that the spectral sensitivity curves of Fig. 1, although measured with great care and the most sophisticated equipment available for this purpose, are not capable of accurately describing the sensor’s response behavior to heavy infrared radiation, whereas other "difficult" spectral mixtures, like e.g. those containing peaked fluorescent components, can be reconstructed with very small and, for most practical applications, highly acceptable errors.

For the RMSE boxplots computed on the normalized spectra shown in the bottom part of Fig. 8, again the mixtures of indoor daylight with any (or all) of the office floor lamps individual LED channels exhibit noticeably larger errors. A possible explanation for these observations can be found in the generally high nRMSE values obtained for the reconstruction of bare indoor daylight spectra without any further mixing of other lighting components. Thus, it can be concluded that the increased nRMSE observed for the various daylight mixtures is a direct consequence of the spectral errors obtained for the bare daylight conditions only. The slightly better results of the former compared to the latter in terms of corresponding nRMSE distributions can be explained by the fact that the non-daylight components within the mixture are still reconstructed by the network approach fairly well reducing the overall spectral errors. Again, no evidence for an impact of different illuminance levels on the reported nRMSE distributions can be observed. Nonetheless, it can be seen that large illuminance errors are a consequence of overestimating the required re-scaling factor to convert from relative to absolute SPDs (see subsection 2.6), which basically leads to an overscaling of the good to excellent relative spectral reconstruction results indicated by small nRMSE scores. These findings again comply with the error mechanisms discussed for lighting mixtures of high levels of infrared radiation.

Finally, comparing both relative and absolute illuminance error types shown in Table 3, it can be noticed that both error types strongly correlate with each other in such a way that the relative illuminance errors are largest whenever the absolute illuminance errors also exhibit large maximal values. This supports the conclusion that the illuminance level itself has no strong impact on the spectral reconstruction errors presented here. For most mixture types, except for those containing tungsten components, the relative illuminance error is below 4 %, which represents a sufficiently accurate performance threshold in line with handheld light meters. For tungsten mixtures, on the other hand, the relative error scores exhibit considerably larger values of 10 to 20 %. Possible approaches to get hold of this problem, apart from applying a more sophisticated infrared blocking, will be presented in the outlook section of this paper.

4.2 Sensor metamers

As can be seen from Fig. 7, the lowest accuracy in reconstructing spectral information is obtained for mixtures of (indoor) daylight with phosphor-converted white LEDs. It appears that these mixtures are reconstructed with an offset between 400 nm and 700 nm but match the ground truth for wavelengths above 700 nm. Despite the observed offset, it must be stated that the general (relative) trend of the measured ground truth spectra is reconstructed reasonably well over all wavelengths. Compared to bare daylight conditions, the observed differences in the relative course of the light spectra are mainly subject to the the balance between shorter and longer wavelength radiation. This gives rise to the question, whether the applied sensor in its current configuration including IR blocking filter is actually capable of distinguishing between these different visible to infrared ratios, or if the sensor responses for these spectra are nearly identical and, therefore, any type of spectral reconstruction based on this input is prone to observer (sensor) metamerism.

Figure 10 gives two fundamental answers: Firstly, by comparing in the figure’s upper row the ten closest sensor signal matches in terms of their euclidean distance to the sensor signals of an arbitrarily chosen measured spectrum of interest which is potentially subject to sensor metamerism (here: a mixture of indoor daylight and cool white LED), it can be seen that, despite the large variance in the spectra depicted in the upper right, there is no clear distinction between the eleven sets of corresponding sensor channel responses shown in the upper left. Most importantly, the huge spectral variability observed in the spectra for wavelengths larger than 700 nm is not reflected by the sensor responses, even though the IR blocking filter has its steep edge only at 750 nm. Secondly, when looking at the max normalized data depicted in the bottom row of Fig. 10, it becomes clear that the applied normalization used for feeding the network further aggravates the problem of sensor metamerism: While there is hardly any difference between the normalized sensor signals across all channels shown in the lower left, the corresponding normalized spectra illustrated in the lower right diverge considerably. It is thus easy to see that whenever the normalized sensor signals used for network training do not differ significantly, the network will be trained to output a reconstruction which minimizes the loss for all those indistinguishable or very similar inputs.

 

Fig. 10. Visualization of the sensor metamers for a mixture of daylight and a cool white LED. The original SPD and its corresponding sensor responses are shown in black. On the top left subplot, the 10 closest sensor response matches from the test database in terms of euclidean distance are shown in comparison. In the top right subplot, correspondingly colored lines represent the light spectra associated with these sensor responses. Additionally plotted are the measured (black solid line) and reconstructed (red solid line) spectrum of interest. While there is similarity between all shown spectra up to 700 nm, there also exists a large variance in the near infrared region, which is not adequately captured by the sensor in its configuration with the IR-cutoff filter at 750 nm. Making a similar comparison for normalized sensor responses and corresponding relative spectral data (bottom row of subplots), it can be stated that normalization of the data partly shifts the problem of ambiguity between sensor responses and spectral outcome to the visible region, which negatively affects spectral reconstruction performance.

Download Full Size | PDF

4.3 Lessons learned

Despite the overall progress this work provides towards the real-world application of spectral sensors for general and human-centered lighting purposes, there are still multiple starting points left for system improvement and further technological studies. Ideally, spectral sensors, similar to professional photometers, should have a cosine-corrected directional sensitivity. In this work, a piece of a simple diffusing foil was therefore used to suppress diverging sensor channel responses as a result of the finite spatial arrangement of the sensor diode array (cf. Fig. 4). As can be seen from the corresponding spectral transmission curve shown in Fig. 6, the diffusion foil absorbs more than 80 % of the light attempting to reach the sensor and, thus, drastically deteriorates the sensor’s combined sensitivity and resulting signal-to-noise ratios. Future work should therefore focus on finding the best trade-off solution between the fulfillment of spatial diffusion requirements, a proper cosine weighting of incident light, prevention of a too strong reduction of sensor signals, and costs.

Previous work on the use of optical sensors for the spectral reconstruction of lighting information further revealed that the impact of IR radiation on the sensor responses can severely diminish reconstruction performance. In the present work, an additional IR blocking filter mounted to the sensor’s housing was used to mitigate this problem. However, the filter used here is by no means a practical solution, neither in price nor regarding its dimensions. Current developments on the sensor market, however, lead to the conclusion that the upcoming generation of spectral sensors will possess improved IR blocking capabilities with corresponding filters being directly attached to the sensor chip avoiding the need for additional actions to be taken by the user.

Nevertheless, when working with silicon-based optical sensors, it may always be expedient to assess, or at least check, the sensor channels’ off-band sensitivities either by performing spectral characterization on a monochromator that offers a tunable range up to 1100 nm or by conducting qualitative measurements on the individual sensor channels using a dedicated IR source (e.g., a tungsten lamp with a high-pass IR transmission filter). With both kinds of measurements, it is essential to choose integration times and gains that do not "hide" small residual sensitivities to IR radiation within the sensor’s quantization level. Due to the large amount of energy usually carried in the NIR/IR wavelength regime by IR sources, such as for example tungsten or daylight emitters, even the smallest residual off-band sensitivities to IR radiation can severely affect the registered sensor responses. In case that full spectral characterization is possible, the measured sensitivities can be corrected accordingly, e.g. by subtracting ratios of the other sensor channels to correct each channels response from off-band signals. Note that the channels in the visible regime may not only suffer from residual off-band sensitivities in the IR range, but can also show a non-negligible response behavior to visible radiation of significantly shorter or longer wavelength than their respective band-pass filter maxima (cf. Fig. 1). More sophisticated correction strategies to deal with these systematic errors need to be developed.

In the present work, using an abbreviated notation of the mixing components together with the actual sensor and spectrometer data proved to be a valuable tool for network training, data balancing, and performance analysis. However, some improvements can still be identified for the future: Adding further information on the mixing ratios of the individual lighting components may help to clearly distinguish between truly "mixed" spectra and those spectra that are made of a dominant component, e.g., tungsten or daylight, and some negligible (i.e., spectrally irrelevant) other parts. This would greatly improve the informative value of quantitative considerations (cf. Fig. 8) in such a way that lighting conditions with only negligible secondary mixture components can be identified easily and excluded from the mixture group for separate analysis.

Finally, the present notation scheme is limited to lighting mixtures comprising three components only. As we focused on the application of "white" light in the general lighting context, this provides an adequate and sufficient dimension of complexity. However, as soon as we want to deal with more sophisticated lighting scenarios, in particular with mixtures of multiple colored LEDs, a finer but at the same time also more performant notation scheme will be needed.

5. Conclusion

This work demonstrated for the first time ever the in-field applicability of multi-band optical sensors for the spectral reconstruction of real-world lighting scenarios. Compared to the MLP approach known from previous studies simulating the sensor behavior, the newly proposed convolutional network architecture proved to offer a slightly better overall reconstruction performance when applied to real sensor data. It is likely that this advantage further increases with the number of sensor channels used for reconstruction, as this also increases the usefulness of the convolution operation for feature extraction and abstraction. In addition to performing the spectral reconstruction on normalized data leading to relative SPDs, a new two-step process was presented and tested to properly convert from relative to absolute spectral information based on the sensor’s response behavior.

It has been shown that when restricted to white light mixtures of different spectral components, proper spectral reconstruction is feasible using response signals of a 10-channel spectral sensor as input. However, analyzing real world sensor responses, it becomes evident that sensor metamerism is a limiting factor for the achievable spectral reconstruction accuracy. This especially concerns mixtures with spectrally broad power distributions, like tungsten or daylight spectra. The problem of similar sensor responses for spectrally different lighting scenarios is not only caused by the low number or width of the sensor channels, but also by the residual offband sensitivity of the channels (c.f. Fig. 1), increasing channel response correlation and decreasing ability to differentiate spectral nuances (e.g. other mixture components) for broad spectra. Despite these observed limitations, on average, spectral deviations of less than 1.6 % in terms of nRMSEs could be achieved. From a colorimetric point of view, the average error was smaller than 0.001 $\Delta u'v'$, while the illuminance could be estimated from the absolute reconstruction results with an average accuracy of 2.7 %.

Besides this good to excellent system performance, there are several aspects that could further be considered and potentially enhanced in the future. Obviously, the reconstruction quality strongly depends on the number and distribution of the different sensor channels. With the number of viable spectral sensors on the market increasing, a simple way of improving the spectral resolution would be to combine multiple different sensors to form a conjunct system unit. For such an approach, it would be essential that the different sensors’ channel sensitivities, especially their maxima, do not overlap extensively. At the same time, each sensor’s channel sensitivity should be truly limited to a small band of wavelengths, so that the channel response is maximally correlated with the incoming irradiance in the passband.

The use of several of such spectral sensors attached to luminaires or distributed separately in a room may also allow for drawing conclusions about the ratio of daylight (e.g., sensor at the window) and artificial light (e.g., sensor at the workplace) an individual user is exposed to. By creating a corresponding network of sensors, detailed monitoring of the lighting conditions in a room or building becomes possible. Crucial questions in this context, which have not yet been addressed in the literature, are how to determine the optimal number of sensor units and their positions in a room, how much information about a lighting situation at a specific remote location (e.g., on the user’s workplace surface) can actually be retrieved from the sensor network, and how to deal with environmental changes in the FOV of the individual sensor (e.g., when a person working in the sensor’s FOV deteriorates its signal quality). Further extensive studies will be necessary to provide answers to all these questions.