Open Access
Add to BibSonomy
Abstract
A near-infrared (NIR) spectral sensor consisting of a 25-channel dielectric multi-patterned filter array (MFA) and CCD is proposed and fabricated. The MFA consists of a wavy dielectric multilayer with a gradient layer profile on a silica substrate with surface grating. The incoming NIR spectrum is predicted by Wiener estimation utilizing the MFA’s spectral responses and a set of training spectral data. Estimation performance is evaluated under various optical shot-noise conditions.
© 2019 Optical Society of America
1. INTRODUCTION
In the fields of agriculture, pharmaceutical and food industries, and remote sensing, visible (VIS) to near-infrared (NIR) spectroscopy has been widely used for the measurement of target qualities from a distance without touching or destroying it [1]. In this method, one needs to measure the incoming light spectra with high accuracy at a specified wavelength range and resolution specific to each target. The major interest of development is now shifting to personal-use or compact-type spectroscopic devices that individual operators can equip [2,3].
For on-site measurement of agricultural products, it is favorable for each vegetable or fruit product to be checked in a short time from the viewpoint of effort reduction in labor. In this application, we note that the incoming spectra have some common shapes because the target always belongs to a specific plant group [4,5]. Thus, we do not need the spectroscopic device to have the ability to predict totally unknown spectra, as lab-use general-purpose spectrometers.
To satisfy these requirements, a snapshot-type device configuration is most promising [6]. The configuration consists of an area image sensor and a multi-patterned filter array (MFA). The most straightforward building block of MFA is the band-pass filters (BPFs). Use of BPFs enables prediction of spectral intensity at the peak (center) wavelength of each filter element. Now the BPF-based imaging spectrometer is commercially available [7].
On the other hand, recently, another class of MFA has emerged, which predicts total spectral shape utilizing a set of non-BPFs. This class includes colloidal quantum dots [8] and holey metallic films supporting the surface plasmon polarition effect [9]. Because each filter element does not necessarily correspond to a single wavelength, a spectral restoration algorithm must be applied to captured intensity data [10].
Two of the criteria for the MFA design of this class is randomness and independence. In Ref. [11], the authors proposed MFAs consisting of pseudo-random and rapidly changing filter elements covering whole VIS wavelengths. They successfully demonstrated the recovery of the sharp radiation spectrum of a mercury lamp through numerical simulation. As a next step, a realistic manufacturing method of such a complicated MFA is awaited.
In this paper, as a solution to the above problem, we propose an MFA consisting of gradient-layer-profile-type photonic crystals. We fabricated and experimentally evaluated its spectral prediction performance. The filter elements consisted of dielectric thin-film multilayers deposited on a surface grating [12–14]. By adding a gradation to the thickness profile, we tried to weaken the Bragg reflection effect and then disperse the passband to wide wavelength range, which was originally separated by the photonic bandgap. In our type of MFA, the initial grating pattern on the substrate can be almost arbitrarily designed by the CAD layout for electron beam lithography (EBL), which in turn enables pixel-based MFA for CCD/CMOS sensors. We have so far experimentally demonstrated polarization and spectroscopic imagers [15]. In addition to fabrication of the filter sample, we evaluated the effect of detection noise (optical shot noise) on the spectral restoration performance.
This paper consists of six sections. In the following section, we show the detail of the structure and the spectral response of the photonic-crystal MFA. In the next couple of sections we describe the method of preparing test spectra, and the spectral recovery algorithm. In Sections 5 and 6 we show the results of experimental and simulation results followed by a discussion about the effective working channels. Finally, we summarize this work.
2. PHOTONIC-CRYSTAL-TYPE MULTIPATTERN FILTER ARRAY
A. Structure and Principle of Operation
Figure 1 illustrates a schematic view of the photonic-crystal MFA [13]. Each filter element consists of a multilayer stacked on a fused silica substrate having a surface grating with the pitch of around 400 nm. This functions as an MFA for a snapshot spectrometer by tiling the equi-pitch region with the integer multiple of the pixel pitch of the CCD/CMOS image sensor. Detail of the fabrication procedure is described in Refs. [13,16].
The film profile is shown in Fig. 2(a). To suppress the effect of Bragg reflection, we intentionally changed the film thickness from the bottom to the top. In the current experiment, we employed the most simple, linear gradient, i.e., the thickness was increased from 220 nm (bottom layer) to 260 nm (top layer) by even 5 nm increments. Finally, to further reduce the multiple reflection, we made the outermost thickness half. Layout of the grating pitch is shown in Fig. 2(b). We prepared elementary filter regions with the pitch linearly increased from 440 nm to 640 nm. The dimension of each region is , which covers pixels of the CCD used (pixel ). Hereafter, we will call these elementary filter regions “channels.” A set of 25 channels will be called a “block.” Also, we will call this type of filter “PhC-DPF” (photonic crystal-type distributed passband filter).
In the fabrication experiment we used EBL equipment (Elionix ELS-125, beam , ) at the Nanotechnology Fabrication Center of Tohoku University, Sendai, Japan [17]. Dry etching and multilayer deposition by auto-cloning method [18] was carried out by Photonic Lattice, Co. Ltd [19].
B. Image Sensor Specification and Procedure for MFA Integration
The PhC-DPF was attached on a CCD image sensor (SONY ICX-205AL, pixel , pixels, bit ) with a UV-curable adhesive (Norland NOA-61). During the assembling process, we monitored the captured image of the CCD in real time and adjusted each pixel using a manipulator. A picture of the final sensor is shown in Fig. 3.
C. Measurement of the Spectral Response
We mounted the above CCD on a camera (Prosilica GC-1350) and measured the filter’s spectral response. Broadband light of a halogen lamp (Edmund Optics, Co. Ltd., MI-150) was guided to a spectrometer (Shimadzu Co. Ltd., SPG-120IR, input and output slit width: 200 μm) through a long-wave pass filter (cutoff wavelength: 610 nm), which removes higher-order light from the spectrometer. The bandwidth of the so-created monochromatic light was about 3.3–3.7 nm. The CCD image was captured while scanning the center wavelength. Example images at 700 nm and 800 nm are shown in Fig. 4. Depending on the spectral response of each MFA channel, the mosaic pattern changed gradually. By normalizing the channel intensities by the spectral sensitivity of a bare CCD sensor, we obtained the PhC-DPF’s spectral response (transmission spectrum). The result is shown in Fig. 5. As expected, a complex-shaped passband distributed over a wide spectral range.
Fig. 5. Measured spectral response of the 25-channel PhC-DPF. Normalized by the spectral sensitivity of a bare CCD.
Note that the random-like transmission curve is not due to the imperfection of the fabrication process. If we made a perfect photonic crystal (no gradient thickness) by the same process, a wide stopband whose transmittance was close to zero and a flat and wide passband were observed [14,16].
3. PREPARATION OF THE TEST SPECTRA FOR PERFORMANCE EVALUATION
We used peaches, as their reflection spectra and correlation to sweetness (Brix) or firmness has been well investigated using VIS or NIR spectroscopy [20–22]. Twenty samples with different maturities, sizes, and appearances were provided from a local peach farm (Hirai fruits farm in Fukushima prefecture). Reflection spectra were measured at four points on the equator by the interactance setup [20,23] using a ring lightguide (Edmund Optics #54-174). Then principal component analysis (PCA) was applied to them, and 600 virtual test spectra in total were synthesized from their loading vectors and statistical information of score distribution. Several examples of so-obtained test spectra are plotted in Fig. 6.
4. METHOD OF SPECTRUM ESTIMATION
Each target spectrum is to be sampled at points on the wavelength axis and represented by a row vector . Accordingly, the filter responses are also expressed as an by matrix . Here, denotes the total number of channels. Then the sensor output through each channel, , becomes an -dimensional row vector represented as follows:
where is another -dimensional row vector expressing a noise overlaid on each sensor channel. Our goal is to predict as correct as possible by using , which is obtained at every shot, and the pre-measured . To do this, in this study, we adopted Wiener estimation [24,25]. This method is a class of linear estimation, which minimizes the expectation for the L2 norm of the difference between a true and predicted spectrum as This is under a constraint that the predicted spectrum is expressed as a linear function of , i.e., where denotes the predicted spectrum. The operator means the expectation over a number of samples. By arranging the above formula, we finally obtain a prediction matrix as follows: where and denote the expectation for the auto-correlation matrices of and , respectively. The superscript denotes transposing.To calculate , we need to know the statistical nature of . Instead, in this study, we calculated for a part of the synthesized spectra (100 out of 600) and took the average of them.
On the other hand, is an square matrix representing the noise statistics. Provided that the optical shot noise is dominant for the presented device configuration, and that there is no correlation between the noise intensities on different channels, can be approximated by a diagonal matrix. Here the shot noise intensity of the ’th channel can be expressed as with the number of electrons generated at the ’th detector, the number of binning or frame accumulation, and the circuit-dependent constant. Thus, the ’th diagonal element of becomes . In the following simulations, we set for simplicity. This assumption corresponds to the ideal situation where the shot-noise variance is equal to its intensity.
5. PERFORMANCE EVALUATION
A. Procedure of Evaluation
In Section 3.A, we produced 600 sample spectra. One hundred of them were used to create the auto-correlation matrix as mentioned above, and the remaining were used for performance evaluation by repeating the following steps.
- 1. Calculate the ground truth sensor output by .
- 2. We assume that the maximum electron capacity of the detector is . A scalar number is multiplied by so that the maximum element of reaches . This operation corresponds to the exposure time extension and binning (or frames accumulation) in the practical experiment. The ’th channel intensity thus becomes . Optical shot-noise level on each channel is estimated as , where represents the Gaussian distribution of , standard .
- 3. Now that the noise matrix is determined, the Wiener estimator for this test spectrum can be calculated.
- 5. Calculate the predicted spectrum using .
- 6. The above (1) to (5) are repeated over 500 validation spectra with 200 random trials for (4) to (5). Finally, total root mean square error (RMSE) is evaluated using the following expression: where and are the numbers of sample spectra and random trials. is the ’th true spectrum. is the estimated spectrum of ’th random trial.
The above explanation is for the estimation of a spectrum incident on a particular MFA block ( channels area). By repeatedly applying the same matrix operation procedure to the remaining parts, we are able to synthesize a two-dimensional distribution of the spectrum. Like all other MFA-type spectral imagers, in our device, the spectral intensity at each pixel position is calculated as a function of neighboring filter pixels (not a single pixel). Filter channels consisting of MFAs work together to predict a spectrum at the center of them. Therefore, after the rendering process, the mosaic-like feature on the original image is almost suppressed. The detail of this de-mosaicing procedure was demonstrated in our previous paper [15] for polarization images. If one needs to suppress residual mosaic-like noise, conventional spatial filtering methods such as filtering in Fourier space is useful. Again, this type of spectral sensor, “snap-shot type,” is especially suitable for imaging applications where motion or deformation of the target object is unavoidable (real-time imaging ability). Such a requirement frequently appears in on-site quality measurement of agricultural products and inline quality monitoring of medicine, for example.
B. Result of Spectrum Estimation
Results for three typical different test spectra are shown in Figs. 7–9. Left, center, and right columns correspond to noise-free condition (a), noisy condition without binning (b), and with 36 pixels or frames accumulation (c). The upper plot denotes raw spectra, whereas the lower ones express the differential intensities between true and predicted spectra. For the noise-free case, the deviation is less than 0.1%, and therefore two plots almost overlap in the upper figure. Despite the noisy condition, the global feature of the target spectra is fairly reproduced as shown in the plots of (b). The deviation from true spectra is on the order of (hatched region) for raw pixel data (b). This deviation is reduced by about 1/2 by 36 pixels or frames averaging (c), as indicated by the hatch. Roughly speaking, the prediction performance is relatively better for longer wavelength and worse for shorter wavelength edge. This is because the divergence of calibration and validation spectra is relatively large in the short wavelength side. A summary of the RMSE for all validation spectra is shown in Table 1.
Fig. 7. Result of estimation for an example spectrum. Upper and lower plots correspond to spectral intensity and differential spectra between target and estimated ones. (a) Noise-free case and (b) with optical shot noise, without binning. Hatched area denotes . (c) With optical shot noise, with 36 data binning. Hatched area is .
Fig. 8. Result of estimation for another example spectrum. (a) Noise-free case; (b) with shot noise, without binning; and (c) with shot noise, with 36 data binning. Hatched areas in (b) and (c) denote and , respectively.
Fig. 9. Result of estimation for another example spectrum. (a) Noise-free case; (b) with shot noise, without binning; and (c) with shot noise, with 36 data binning. Hatched areas in (b) and (c) denote and , respectively.
Table 1. Summary of RMSE (: Binning Number)
It is rather difficult to find directly comparable research works about random MFA-type spectrometers in the NIR region. However, in the VIS wavelength region, BPF-type spectrometers have been reported. For example, Kurokawa’s group [26] anb Shen’s group [27] both utilize BPF-type MFAs and deal with broadband spectra as in our work. For the former, aggregated relative error was , which may correspond to RMSE . In the latter, the typical performance (“spectral rms error”) for 16-channel, noise was 0.0103. Though the total wavelength coverage and MFA construction are both completely different, it is worth pointing out that the RMSE levels of their and our devices are on the same order as .
6. DISCUSSION
A. Number of Working Channels
The device reconstructs target spectra using a diverged set of MFA output intensities. We investigated how many channels out of a total of 25 represent large variation. Figure 10 shows the MFA output () for 100 test spectra under the noise-free condition. The point at which ch. 10 stayed constant irrespective of spectral feature shows that its output was always the largest among all (note that in our simulation, we virtually extended the exposure time to the period where one of the channels reaches the full well capacity of the sensor). In Fig. 10, the outputs of ch. 9 and ch. 11 are also almost invariant, showing that these channels were also used only for normalizing other channels.
Fig. 10. Calculated MFA output for various spectral datasets. Upper and lower plots correspond to the filter output intensity (left scale) and variation factor (, right scale).
To see the function of the remaining channels, we plotted the variation factor (ratio of the standard deviation to the average) of as the lower plot in Fig. 10. The relative variation was found to be large around ch. 5 and after ch. 18. According to Fig. 5 (MFA transmission spectra), we can see a jump of primary transmission peak between ch. 17 and ch. 18. This may be the main reason for such discontinuity.
The variation factor differed by about five to 10 times from channel to channel. The factor was on the order of at ch. 5, ch. 16, and ch. 17. This number is several times larger than the expected intensity fluctuation by shot noise, which enables spectral estimation as shown in Figs. 7(b), 8(b), and 9(b). However, there is still room for design optimization, so that variation becomes much larger, especially in ch. 1 to ch. 15, in order to do precise estimation without the aid of pixel binning or frame averaging as in Figs. 7(c), 8(c), and 9(c).
B. MFA Sensitivity for Principal Components
The target spectrum is composed of a finite number (in this simulation, 10) of principal components (PCs). By denoting the n’th PC and its score as and , respectively, the incoming spectra can be expressed as
By noting the output intensity of ’th PC through ’th MFA channel as , total output intensity of each channel can be rewritten as The ratio of such PC-specific MFA output to the output for average spectrum with is plotted in Fig. 11. The vertical axis is with the channel number. As we explain in Fig. 10, in most cases ch. 10 took the maximum intensity. Therefore in this figure, we also normalize the intensity by that of ch. 10. From this result, we see the following insights. First, as represents the rough shape of the spectrum, the plot of the figure should have been almost flat (similar MFA output as average spectrum incidence). However, we see significant divergence after ch. 19. This indicates that what represents the global feature of the spectrum is ch. 1 to ch. 18. This deviation implies that in the present spectral dataset used, residual PC components other than were substantially included.Fig. 11. Calculation for MFA output for first three principal components (PCs). Intensities are normalized to that of channel 10.
The higher PC components play an important role in non-destructive interior measurement by NIR spectroscopy. To see the responsivity of and in Fig. 11, it is clear that they behave differently as around ch. 5 and ch. 15. For , output near ch. 14 is smaller than , while showing a similar tendency before ch. 9. On the other hand, exhibits much smaller response between ch. 3 to ch. 7, and also fairly small output in ch. 14 to ch. 18. These results lead to the degree of output suppression before and after ch. 10 depending on the magnitudes of and contained.
The above discussion depends on the class of target spectrum. There is no guarantee that similar insight is applicable to spectral datasets other than what we used for the current study. At least, we need to optimize the MFA response functions so that output divergence becomes as large as possible for each PC loading. Detailed optimization will remain as our next task.
C. Resolution
If this sensor is to be applied to an imaging spectrometer, final spatial resolution will be on the image plane, corresponding to pixels/channel. On the other hand, resolution on the object space depends on the magnification of the objective lens.
For spectral domain, one of the measures is the narrowest possible monochromatic spectral peak to be resolved. We evaluated the resolution in this context by first constructing a self-correlation matrix using a number of Gaussian peaks with a finite bandwidth. Then a monochromatic spectrum was estimated using the algorithm in Section 5.A. An example result is shown in Fig. 12, where 20 trials with random shot noises are drawn by black lines together with the original one (red line). Estimated full width at half maximum (FWHM) was about 12 nm, while the ground truth was 10 nm. This bandwidth was found to be never reduced less than 8 nm, even when we applied the Wiener estimation on narrower peaks. Thus, the spectral resolution of the proposed MFA-CCD spectrometer is about 8–10 nm. This fundamental limitation should be attributed to the spectral steepness of our photonic crystal MFA. Further improvement will be possible by, for example, increasing the number of multilayers.
Fig. 12. Estimation result for a monochromatic peak centered at 850 nm, . Red and black curves correspond to original and estimated spectra, respectively. The latter consists of twenty trials with shot noises. Binning of 36 pixels as in Figs. 7–9 are assumed.
7. CONCLUSION
An integrated NIR spectral sensor consisting of multilayer photonic-crystal MFA and a CCD image sensor was demonstrated. The MFA exhibited distributed passband spectral characteristics, which enabled spectral prediction over a wide wavelength range. The Wiener estimation algorithm was utilized for spectrum reconstruction. The RMSE level on the order of was achieved. By considering the statistical property of the optical shot noise, the criteria for pixel binning or frames accumulation were shown. Application to multispectral image construction, by binding all pixel information together with MFA response optimization, will be conducted in our future study.
Funding
Tateishi Science and Technology Foundation and SME Support, Japan
Acknowledgment
We are grateful for Hirai Farm in Fukushima for providing us sample fruits. The authors declare that there are no conflicts of interest related to this article.
REFERENCES
1. B. B. Osborne, T. Fearn, and P. H. Hindle, Practical NIR Spectroscopy (Pearson Education, 1993).
2. F. Sigernes, M. Syrjasuo, R. Storvold, J. Fortuna, M. E. Grotte, and T. Johansen, “Do it yourself hyperspectral imager for handheld to airborne operations,” Opt. Express 26, 6021–6035 (2018). [CrossRef]
3. C. Zhang, G. Cheng, P. Edwards, M.-D. Zhou, S. Zheng, and Z. Liu, “G-Fresnel smartphone spectrometer,” Lab Chip 16, 246–250(2016). [CrossRef]
4. S. Gunasekaran, M. R. Paulsen, and G. C. Shove, “Optical methods for nondestructive quality evaluation of agricultural and biological materials,” J. Agric. Eng. Res. 32, 209–241 (1985). [CrossRef]
5. B. M. Nicolai, K. Beullens, E. Bobelyn, A. Peirs, W. Saeys, K. I. Theron, and J. Lammertyn, “Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: a review,” Postharvest Biol. Technol. 46, 99–118 (2007). [CrossRef]
6. M. Najiminaini, F. Vasefi, B. Kaminska, and J. J. L. Carson, “Nanohole-array-based device for 2D snapshot multispectral imaging,” Sci. Rep. 3, 2589 (2013). [CrossRef]
7. A. Lambrechts, P. Gonzalez, B. Geelen, P. Soussan, K. Tack, and M. Jayapala, “A CMOS-compatible, integrated approach to hyper- and multispectral imaging,” in IEEE International Electron Devices Meeting (2014), pp. 10.5.1.
8. J. Bao and M. G. Bawendi, “A colloidal quantum dot spectrometer,” Nature 523, 67–70 (2015). [CrossRef]
9. D. Inoue, A. Miura, T. Nomura, H. Fujikawa, K. Sato, N. Ikeda, D. Tsuya, Y. Sugimoto, and Y. Koide, “Polarization independent visible color filter comprising an aluminum film with surface-plasmon enhanced transmission through a subwavelength array of holes,” Appl. Phys. Lett. 98, 093113 (2011). [CrossRef]
10. B. Redding, S. F. Liew, R. Sarma, and H. Cao, “Compact spectrometer based on a disordered photonic chip,” Nat. Photonics 7, 746–751 (2013). [CrossRef]
11. J. Oliver, W.-B. Lee, and H.-N. Lee, “Filters with random transmittance for improving resolution in filter-array-based spectrometers,” Opt. Express 21, 3969–3989 (2013). [CrossRef]
12. S. Kawakami, “Fabrication of submicrometre 3D periodic structures composed of Si/SiO2,” Electron. Lett. 33, 1260–1261 (1997). [CrossRef]
13. Y. Ohtera, T. Onuki, Y. Inoue, and S. Kawakami, “Multichannel photonic crystal wavelength filter array for near-infrared wavelengths,” J. Lightwave Technol. 25, 499–503 (2007). [CrossRef]
14. Y. Ohtera and H. Yamada, “Multichannel bandpass filters utilizing multilayer photonic crystal,” Opt. Lett. 38, 1235–1237 (2013). [CrossRef]
15. K. Shinoda, Y. Ohtera, and M. Hasegawa, “Snapshot multispectral polarization imaging using a photonic crystal filter array,” Opt. Express 26, 15948–15961 (2018). [CrossRef]
16. Y. Ohtera, D. Kurniatan, and H. Yamada, “Design and fabrication of multichannel Si/SiO2 autocloned photonic crystal edge filters,” Appl. Opt. 50, C50–C54 (2011). [CrossRef]
17. Tohoku University Micro System Integration Center, http://www.mu-sic.tohoku.ac.jp/index_e.html.
18. T. Kawashima, Y. Sasaki, K. Miura, N. Hashimoto, A. Baba, H. Ohkubo, Y. Ohtera, T. Sato, W. Ishikawa, T. Aoyama, and S. Kawakami, “Development of autocloned photonic crystal devices,” IEICE Trans. Electron. E87-C, 283–290 (2004).
19. Photonic Lattice, Inc., https://www.photonic-lattice.com/en/.
20. S. Kawano, H. Watanabe, and M. Iwamoto, “Determination of sugar content in intact peaches by near infrared spectroscopy with fiber optics in interactance mode,” J. Japanese Soc. Hortic. Sci. 61, 445–451 (1992). [CrossRef]
21. R. Lu and Y. Peng, “Hyperspectral scattering for assessing peach fruit firmness,” Biosystems Eng. 93, 161–171 (2006). [CrossRef]
22. V. Ziosi, M. Noferini, G. Fiori, A. Tadiello, L. Trainotti, G. Casadoro, and G. Costa, “A new index based on vis spectroscopy to characterize the progression of ripening in peach fruit,” Postharvest Biol. Technol. 49, 319–329 (2008). [CrossRef]
23. G. S. Birth, G. G. Dull, J. B. Magee, H. T. Chan, and C. G. Cavaletto, “An optical method for estimating papaya maturity,” J. Am. Soc. Hortic. Sci. 109, 62–66 (1984).
24. W. K. Pratt and C. E. Mancill, “Spectral estimation techniques for the spectral calibration of a color image scanner,” Appl. Opt. 15, 73–75 (1976). [CrossRef]
25. H. Haneishi, T. Hasegawa, A. Hosoi, Y. Yokoyama, N. Tsumura, and Y. Miyake, “System design for accurately estimating the spectral reflectance of art paintings,” Appl. Opt. 39, 6621–6632 (2000). [CrossRef]
26. U. Kurokawa, B. I. Choi, and C.-C. Chang, “Filter-based miniature spectrometers: spectrum reconstruction using adaptive regularization,” IEEE Sens. J. 11, 1556–1563 (2011). [CrossRef]
27. H.-L. Shen, P.-Q. Cai, S.-J. Shao, and J. H. Xin, “Reflectance reconstruction for multispectral imaging by adaptive Wiener estimation,” Opt. Express 15, 15545–15554 (2007). [CrossRef]
