Three-dimensional integral imaging in photon-starved environments with high-sensitivity image sensors

Adam Markman; Timothy O’Connor; Hisaya Hotaka; Shinji Ohsuka; Bahram Javidi

doi:10.1364/OE.27.026355

1. Introduction

Three-dimensional (3D) visualization techniques have been studied for many decades [1,2]. Integral imaging (InIm) [3] is an imaging technique that captures multiple perspectives of a scene using a lenslet array or an array of cameras. As multiple two-dimensional perspectives of a scene are captured, known as elemental images, a 3D reconstruction can be performed optically or computationally. Moreover, computerized modern realization of integral imaging can perform extraction of profilometric information of objects and removal of occlusion [4]. The fields of application of computational 3D integral imaging include 3D imaging in turbid water [5], 3D tracking of occluded objects [6], human gesture recognition [7], and 3D visualization in low-light-level conditions [8].

Since integral imaging integrates multiple elemental images, it is expected to perform well for low-light-level 3D imaging in the presence of statistical fluctuation in each of the elemental images including additive noise generated from imaging devices as well as fluctuations in the detected number of photons. In [9], simulated photon-limited elemental images were computationally generated from images acquired by a conventional digital camera and used for 3D integral imaging reconstruction, which showed improved image performance compared to a single 2D elemental image. Real integral imaging experiments were carried out in low-light-levels using a 16-bit cooled CCD camera [8]. The 3D reconstructed images demonstrated improved signal-to-noise ratio (SNR). Experiments using a low-cost compact CMOS camera were also carried out in the visible spectrum [10]. Readout noise limited elemental images were acquired and 3D integral imaging reconstruction was computationally performed. It was shown that the 3D reconstructed image is optimal in the maximum-likelihood sense for readout noise limited images as the noise is additive Gaussian noise. After 3D integral imaging, face detection was achievable as SNR improved in addition to occlusion removal whereas face detection fails for a single 2D elemental image due to the occlusion and low SNR. Although these previous studies showed the possibility of integral imaging for 3D visualization under poor illumination, experimental demonstration has not been presented so far in low-light-level conditions where the fluctuation of the number of incident photons is significant.

Capturing information from a scene where only a limited number of photons are available is of particular interest in applications spanning from astronomy [11] to marine biology [12]. Using conventional imaging cameras such as CCD or CMOS cameras in low-light-level conditions, the images become readout noise dominant wherein readout noise is added to the images at the amplification stage prior to analog-to-digital conversion. To overcome these limitations, advanced sensors have been introduced that minimize the readout noise enabling single- to several-photon sensitivity. Two sensors in particular are the scientific complementary metal- oxide semiconductor (sCMOS) camera [13] and the electron multiplying charge-coupled device (EM-CCD) camera [14]. Although the underlying operation of the cameras differ, these cameras minimize the readout noise allowing for imaging in extremely low-light-level environments. Examples of the fields where EM-CCD cameras have been used are biomedical imaging [15] and quantum imaging [16]. While the EM-CCD camera performs well in photon-number-limited environments, the camera suffers from electron multiplication noise, requires heavy cooling below −50°C, and has a relatively low frame rate. An alternative is the sCMOS camera as it offers a higher frame rate and does not need to be heavily cooled at the expense of not performing as well in extreme low-light-level conditions. The sCMOS camera has found success in applications including localization microscopy [17,18].

In this paper, we present a theoretical and experimental investigation of 3D integral imaging under low-light and extreme low-light-level conditions. In the experiments, the number of photons detected per pixel on average ranges from only a few photons, and up to several tens of photons. The experiments are carried out using both the sCMOS camera and the EM-CCD camera to capture a mannequin behind occlusion under these low illumination conditions and 3D reconstructions of the scene are computed. A theoretical model of the probability distribution of the pixel values is derived for the EM-CCD camera, then fitted with the experimental data to determine its camera parameters. Similarly, pixelwise calibration is performed on the sCMOS camera to determine its camera parameters. The camera parameters are required to determine the number of photons detected per pixel. Furthermore, the expected SNR is provided for each image sensor and corroborated by the experimental findings. The SNR and the mean number of detected photons are calculated for the object of interest and a background region to compare quantitatively the SNR of the integral imaging output with different image sensors in low-light-level conditions. Improvement of image quality metrics including the SNR, Contrast-to-Noise ratio (CNR), and the Perception based Image Quality Estimator (PIQE) are computed and compared to the metrics computed for a single 2D image. Improvement of image quality metrics in the 3D reconstructed images is successfully confirmed compared with those of the 2D elemental images.

This paper is organized as follows: in Section 2, the experimental setup for integral imaging in low light levels is presented. We describe the state-of-art high-sensitivity cameras used in the experiments, along with the camera calibration results which are needed to determine the camera parameters for evaluation of low light imaging experimental results. Also, in this section we shall present some metrics for image evaluation in low light levels. In section 3, the SNRs of elemental images acquired by cameras are compared, and 3D reconstructed images are presented. Also, improvements of the reconstructed image quality by 3D integral imaging in low light levels are discussed based on the metrics for image evaluation. Finally, conclusions are given in Section 4.

2. Experimental method

In this section, the integral imaging experimental setup used in low light levels experiments is presented. The integral imaging system uses two high-sensitivity cameras for low light experiments, that is an EM-CCD camera and an sCMOS camera which we shall describe in section 2.1. We describe the theoretical and experimental approach for camera calibration to determine the camera parameters needed to compute the number of photons detected per pixel, and the SNR of the elemental images and 3D reconstructed images in the low light imaging experiments. To analyze the improvements obtained by our proposed approach, we shall present a number of metrics for image evaluation in low light levels in section 2.2.

2.1 High-sensitivity cameras and calibration methods

In the following, high-sensitivity EM-CCD and sCMOS cameras used in our low light experiments, along with their calibration methods are described.

2.1.1 EM-CCD camera and calibration results

In conventional CCDs, photons arriving at the image area are stochastically converted to electrons and sent to the store area, which is followed by the readout register. The electrons transferred through the readout register are converted to a voltage by the on-chip output amplifier, where the readout noise is added, then read out by the analog-to-digital converter. In low-light-level imaging, the signal becomes smaller than the readout noise resulting in little information visibly perceived. EM-CCDs have an additional multiplication register after the readout register and prior to the output amplifier, which provides the extra gain of over 1000 known as the electron multiplication (EM) gain through the series of impact-ionization processes [14,19]. Although the electron multiplication process introduces the so-called excess noise due to its stochastic nature [20], the large EM gain can effectively reduce the readout noise to sub-electron levels. The signal-to-noise ratio measured by the number of detected photons with an EM-CCD camera is given by the following equation [21]:

(1)$$SN{R_{EM - CCD}} = \frac{{S + D}}{{\sqrt {{F^2}(S + D) + {\sigma ^2}/{\mu ^2}} }},$$

where S represents the mean number of electrons per pixel caused by photoelectric conversion, D is the number of spurious electrons per pixel generated without photon incidence, F is the excess noise factor of the electron multiplication process, σ is the standard deviation of the readout noise in analog-to-digital unit (ADU), and μ is the EM gain [ADU/electron]. In the case where D ≪ S because of the short exposure time, the device temperature is low enough, and when the EM gain is high enough so that σ²/μ² becomes negligible and F² approaches 2, SNR_EM-CCD can be approximated simply as

(2)$$SN{R_{EM - CCD}} = \sqrt {S/2} .$$

For the purpose of quantitative evaluation of image qualities, calibration of characteristic parameters such as the EM gain and the offset of the pixel value is necessary. The probability distribution of the pixel values (v) corresponding to the output from the EM register for an input of a single primary electron is given approximately by [22]

(3)$$p(v) = \frac{1}{\mu }\exp \left( { - \frac{v}{\mu }} \right).$$

When the probability q for a single electron generation per pixel is much smaller than 1, the distribution of the pixel values is expressed as [23]

(4)$$p(v) = ({1 - q} )\delta (v )+ \frac{q}{\mu }\exp \left( { - \frac{v}{\mu }} \right).$$

The first term of the right-hand side, where δ (·) is Dirac’s delta-function, denotes the pixels with no primary electron. Since the Gaussian noise is attached to the pixel value and the offset value is also added in the readout process, Eq. (4) should be convolved with the readout noise distribution and the distribution of real pixel values is given by following equation:

(5)$$p(v) = \frac{{1 - q}}{{\sigma \sqrt {2\pi } }}\exp \left\{ { - \frac{{{{(v - b)}^2}}}{{2{\sigma^2}}}} \right\} + \frac{q}{{2\mu }}\exp \left( {\frac{{{\sigma^2}}}{{2{\mu^2}}} - \frac{{v - b}}{\mu }} \right)\textrm{erfc}\left( {\frac{\sigma }{{\sqrt 2 \mu }} - \frac{{v - b}}{{\sqrt 2 \sigma }}} \right),$$

where b is the offset value, and erfc (·) is the complimentary error function. Figure 1(a) shows a partial image of 128 × 128 pixels extracted from the dark image frame acquired with the EM-CCD camera used in this study (Hamamatsu ImagEM 1K C9100-14) with the lens cap on and Fig. 1(b) shows a distribution plot of the pixel values for the whole image area (1024 × 1024 pixels). Spurious electrons are detected, and their pixel values form an exponential distribution. Since the EM-CCD camera operates below −50 °C and the exposure time of 103 ms is short enough to neglect dark current, the dark current can be ignored as a source of spurious electrons. Another source of the spurious events is the clock induced charge (CIC). Among the CIC events, those generated in the parallel register (pCIC events) are indistinguishable from photon detection events because they travel the full length of the EM register. On the other hand, since CIC events generated in the EM register (sCIC events) pass through shorter length of the EM register and therefore have lower pixel values, sCIC events are distinguishable from photon detection events based on their pixel values [24]. Thus, by fitting the model given by Eq. (5) to the distribution of the pixel values in the adequate range as shown in the graph [see Fig. 1(b)], the EM gain for the photon detection events can be estimated. The solid line in Fig. 1(b) shows the fitted model distribution. The estimated characteristic parameters are as follows: b = 3208.67 ± 0.09 [ADU], μ = 133.1 ± 4.3 [ADU/electron], σ = 15.16 ± 0.06 [ADU], and q = 0.0609 ± 0.0027.

Fig. 1. (a) Partial image of 128 × 128 pixels extracted from the dark image frame acquired with the lens cap on. Color bar depicts grayscale values in ADU. (b) Distribution plot of pixel values of the whole dark image frame. Solid line is a model distribution given by Eq. (5). Circles correspond to experimental values.

Download Full Size | PDF

2.1.2. sCMOS camera and calibration results

The sCMOS camera operates on the principle of a CMOS sensor in that amplification occurs on each pixel [13]. Thus, the readout noise and the coefficient of conversion from the number of electrons to the pixel value varies pixel to pixel. The SNR in terms of number of detected photons in a pixel of an sCMOS camera is given by the following equation when the dark current electrons are negligible:

(6)$$SN{R_{sCMOS}} = \frac{{\kappa S}}{{\sqrt {{\kappa ^2}S + {\sigma ^2}} }} = \frac{S}{{\sqrt {S + {\sigma ^2}/{\kappa ^2}} }},$$

where κ is the conversion coefficient [ADU/electron]. Pixel-wise calibration of the characteristic parameters of the sCMOS such as κ can be performed according to the procedure described in [25]. Figure 2 shows calibration results of the sCMOS camera used in this study (Hamamatsu ORCA Flash4.0 LT C11440-42U). Histograms of the characteristic parameters for the whole 2048 × 2048-pixel image are presented, including histograms of the offset value b as shown in Fig. 2(a), the conversion coefficient κ as shown in Fig. 2(b), and the variance of the readout noise σ² as shown in Fig. 2(c). From the histograms, mean values of the parameters are calculated as follows: b_avg = 99.7 [ADU], κ_avg = 2.15 [ADU/electron], and σ²_avg= 6.76 [ADU²]. The subscript “avg” denotes averaging over all pixels. Since the standard deviation of the histogram of b and κ are small, for example 0.54 [ADU] for b, b_avg and κ_avg are used in the following analysis to clarify the calculation. The pixel size of the imaging devices affects imaging performances such as sensitivity and resolution. The sCMOS camera (C11440-42U) has a pixel size of 6.5 µm × 6.5 µm and the EM-CCD camera (C9100-14) has 13 µm × 13 µm pixels. For quantitative comparison between the two cameras, 2 × 2-pixel binning was applied to the images acquired with the sCMOS camera hereafter. The SNR of the sCMOS camera after binning is given by

(7)$$SN{R_{sCMOS}} = \frac{{{S_{binned}}}}{{\sqrt {{S_{binned}} + 4{r^2}} }},$$

where S_binned is, in this case, the number of electrons per binned pixel and r² = σ²_avg/κ_avg².

Fig. 2. Histograms of (a) the offset value, (b) the conversion coefficient, and (c) the variance of the readout noise of whole 2048 × 2048 pixels of the sCMOS camera.

Download Full Size | PDF

2.2 Image evaluation metrics

In the following section, metrics listed below are used to evaluate the quality of elemental images and 3D reconstructed images quantitatively. The metrics chosen were motivated by the work done in [26] which discussed the effect of imaging in low-light conditions on both signal-to-noise ratio and contrast. Some of the metrics are calculated based on the statistics of the pixel value inside regions of interest (ROIs) which are carefully selected on the images. We choose a 10 × 10-pixel region such that local uniformity of the intensity of incident photons inside the ROI is assumed. To simplify the calculation, the offset value, b for EM-CCD images and 4b_avg for binned sCMOS images, is subtracted from the pixel values beforehand.

The mean number of photons detected: This metric n_ph corresponds to S in Eq. (2) and S_binned in Eq. (7) and is given by

(8)$${n_{ph}} = \left\{ \begin{array}{l} \langle v \rangle /\mu \textrm{, for EM - CCD}\\ \langle v \rangle /\kappa \textrm{, for sCMOS} \end{array} \right.$$

where 〈v〉 is the mean pixel value inside the ROI.

SNR inside ROI: The SNR of the pixel value inside the ROI is given as a ratio of the mean to the standard deviation by the following equation:

(9)$$SN{R_{ROI}} = \frac{{\langle v \rangle }}{{\sqrt {\langle{\Delta {v^2}} \rangle } }},$$

where 〈Δv²〉 denotes the variance of the pixel values inside the ROI.

Contrast-to-noise ratio (CNR): When an object is detected from the background, the signal contrast is given by the difference of the mean pixel values between the object and background region and the noise is the square root of the sum of the variances of both regions. Thus, the CNR for the object-to-background contrast is given as follows:

(10)$$CNR = \frac{{\langle{{v_o}} \rangle - \langle{{v_b}} \rangle }}{{\sqrt {\langle{\Delta {v_o}^2} \rangle + \langle{\Delta {v_b}^2} \rangle } }},$$

where 〈v_o〉, 〈v_b〉 are the mean pixel values inside the object and background ROI respectively, and 〈Δv_o²〉 and 〈Δv_b²〉 are the variance of the pixel values inside the object and background ROI, respectively.

Perception based image quality evaluator (PIQE): [27] This metric, which assigns a score from 0 to 100 with 100 being the worst score, provides a no-reference image quality score of the images. The PIQE algorithm works by estimating the distortion within blocks of an image or image region then pooling the estimated block level distortions for an overall evaluation of the image quality. The blocks are constructed as non-overlapping segments of the image, wherein each block is classified as either uniform or spatially active based on the variance of its Mean Subtracted Contrast Normalized (MSCN) coefficients [28]. Within each spatially active block, the level of distortion is evaluated using the MCSN coefficients, then a threshold is applied to the evaluated distortion levels to classify a block as distorted or undistorted. Finally, the PIQE score is taken as the mean of the scores from the distorted blocks. This method considers the human visual system’s tendency to focus on spatial active regions and that overall quality can be assessed by pooling the quality of local block regions. Moreover, this metric assumes an input image suffers from additive Gaussian noise or blocking artifacts.

2.3. Three-dimensional integral imaging

3D integral imaging is an imaging technique that captures multiple perspectives of a scene, such as through the use of a camera array. Figure 3(a) depicts the 3D integral imaging pick up process while Fig. 3(b) is a depiction of optical reconstruction of a reference scene imaged in regular illumination conditions. From Fig. 3(c) depicting the corresponding images of the reconstruction stage for our reference scene in regular illumination, we can observe the occlusion is located at a distance of 3 m whereas the object is located at 4.1 m.

Fig. 3. Integral imaging system for the (a) pick-up stage and (b) 3D reconstruction stage. (c) Corresponding images of the reconstruction stage for the reference scene imaged with a conventional camera (Allied Vision Mako G-192 GigE) in regular illumination conditions. Insets of (c) depict an example elemental image, reconstructed image at a depth of z = 3 m showing clear reconstruction of the occlusion, and reconstructed image at z = 4.1 m showing clear reconstruction of the object. p_x and p_y are the pitch between image sensors in the x and y directions, respectively.

Download Full Size | PDF

After 2D elemental image acquisition, computational 3D integral imaging is performed as follows [10]:

(11)$$I({x,y;z} )= \,\,\,\frac{1}{{O({x,y} )}}\sum\limits_{a = 0}^{A - 1} {\sum\limits_{b = 0}^{B - 1} {{E_{a,b}}\left( {x - a\frac{{{L_x} \times {p_x}}}{{{c_x} \times M}},y - b\frac{{{L_y} \times {p_y}}}{{{c_y} \times M}}} \right),} } \,$$

where (x, y) is the pixel index, z is the reconstruction distance, O(x, y) is the overlapping number on (x, y), A and B are the total number of elemental images obtained in each column and row, respectively; E_a,b is the elemental image in the a-th column and b-th row, L_x and L_y are the total number of pixels in each column and row, respectively, for each E_a,b, M is the magnification factor and equals to z/f, f is the focal length, p_x and p_y is the pitch between image sensors, c_x and c_y are the size of the image sensor.

Two experiments were performed to image a mannequin behind occlusion using the EM-CCD camera and the sCMOS camera with an exposure time of 103 ms in different illumination conditions (Scene 1 and Scene 2). The experiments were also conducted with the occlusion removed resulting in a total of 4 low illumination scenes for each camera. Furthermore, an additional experiment was conducted using the sCMOS camera without occlusion in high illumination conditions (Reference Scene) and a shortened exposure time to serve as a reference. For all 3D integral imaging experiments conducted, the imaging setup consisted of translating the EM-CCD camera or sCMOS camera on a translation stage in a grid-like fashion to mimic a 3 × 24 array with a pitch of 7.5 mm (H) × 60 mm (V). After each translation step, an image was captured resulting in a total of 72 elemental images.

A reference scene in high illumination conditions was first captured using the sCMOS camera. In this experiment, all camera parameters remained the same, except for the exposure time, which was reduced to 3.2 ms to prevent saturation. Moreover, 2 × 2 binning was used during image acquisition. Under these conditions, the median detected number of photons (n_ph) on the mannequin’s face for 72 elemental images taken in regular illumination conditions was calculated as 10,745 photons detected per pixel. The Lux of the well illuminated scene was measured using a light meter as ranging from 173.6 Lux to 241.0 Lux. Results of this experiment are depicted in Fig. 4, which shows an elemental image of the scene (Fig. 4(a)), a reconstructed image at 4.1 meters (Fig. 4(b)), and a reference image taken by iPhone 8 (Fig. 4(c)) for further comparison.

Fig. 4. (a) Elemental image of the reference scene taken in regular illumination without occlusion. (b) Reconstructed Image of the reference scene at Z = 4.1 m. (c) Image of the reference scene taken by iPhone 8. Green box indicates the ROI on the mannequin’s face used to calculate the detected number of photons (n_ph).

Download Full Size | PDF

The first experimental scene was under a low illumination condition where a computer monitor facing away from the scene was the main source of illumination; the light from the monitor was reflected off of a wall. The second experimental scene was under lower illumination conditions where the computer monitor was covered to remove its effect as a light source. The occlusion used in the experiments was an artificial plant located approximately 3 m from the pickup stage while the mannequin was located approximately 4.1 m from the pickup plane. Figure 3(b) depicts the reconstruction of the reference scene used in the experiment. Figure 3(c) depicts the 2D elemental image consisting of a mannequin partially occluded by a plant in a regular illumination condition with a conventional camera and corresponding reconstructions at two depths. The reconstructed image at z = 3 m shows clear reconstruction of the occluding plant and the reconstructed image at z = 4.1 m shows the in-focus mannequin where the occlusion has successfully been removed.

3. Results and discussion

For the first scene, Scene 1, where the only source of illumination was from the computer monitor, Fig. 5(a) shows an example of a 2D elemental image acquired by a conventional camera (Allied Vision Mako G-192 GigE) where the readout noise is dominant and no information is obtained due to the limited number of available photons. As mentioned in the previous section, the statistics of the pixel values are calculated inside two ROIs of 10 × 10 pixels set in each elemental image, the object ROI and the background ROI which are depicted by green and red rectangles, respectively in Fig. 5(b).

Fig. 5. Scene 1 using EM-CCD camera and sCMOS camera. (a) Example of 2D elemental image using a conventional CMOS (Allied Vision Mako G-192 GigE) camera under the low illumination conditions used in the experiment. (b) Part of an elemental image without occlusion acquired by the EM-CCD camera with the object ROI (green rectangle) and the background ROI (red rectangle) used for the metric calculations. (c) Part of the 3D reconstructed image for the EM-CCD camera. (d) 2D elemental image acquired by the EM-CCD camera in which the object ROI is occluded by the artificial plant and (e) not occluded. (f) 3D reconstructed image at z = 4.1 m for the EM-CCD camera. CNR for (d), (e) and (f) are 2.68, 3.58 and 22.4, respectively. (g) 2D elemental image acquired by the sCMOS camera in which the object ROI is occluded by the artificial plant and (h) not occluded. (i) 3D reconstructed image at z = 4.1 m for the sCMOS camera. CNR for (g), (h) and (i) are 2.88, 4.76 and 19.0, respectively. Color bars depict the grayscale values in ADU.

Download Full Size | PDF

For the experiments without occlusion, Fig. 6 shows a plot of the experimentally calculated SNR_ROI versus n_ph for the object and the background ROIs of the elemental images along with the theoretical prediction of the SNR from Eqs. (2) and (7). Theoretical prediction for EM-CCD and the sCMOS cameras are presented as a solid line for the EM-CCD camera and a dashed line for the sCMOS camera. Both cameras show good agreement between theoretical and experimental calculations. From this result the sCMOS camera with 2 × 2 pixels binning is preferably applicable to integral imaging in conditions where more than several photons are detected per binned pixel and the EM-CCD camera is expected to be a suitable imaging device for integral imaging in the photon counting light-level conditions where less than single photon is detected per pixel on average.

Fig. 6. Plot of SNR_ROI versus mean number of detected photons inside the object and the background ROIs of the elemental images acquired in Scene 1 and Scene 2 without occlusion by the EM-CCD camera (red circle) and the sCMOS camera (blue circle). The solid line and the dashed line are theoretical predictions based on Eq. (2) and Eq. (7) with r² = 1.46, respectively. Circles illustrate experimental results.

Download Full Size | PDF

For Scene 1 without occlusion the median values of CNR given by Eq. (10) and calculated from the 72 elemental images for the EM-CCD camera and the sCMOS camera are 3.60 and 3.34, respectively and the medians of the number of detected photons inside the object ROI are 60.9 and 27.7 photons detected/pixel, respectively. Recall that images from the sCMOS camera are binned by 2 × 2 pixels. After 3D integral imaging reconstruction, the CNRs of object-to-background contrast are improved as follows: CNR = 30.0 for the EM-CCD camera and CNR = 26.6 for the sCMOS camera, corresponding to 8.31 times and 7.96 times improvement, are achieved. Figure 5(c) shows a partial image of the reconstructed 3D image for the EM-CCD camera. Improvement of the SNR is evident compared with Fig. 5(b). For the experiments in the presence of occlusion, two of the 2D elemental images acquired by the EM-CCD camera are shown in Figs. 5(d) and 5(e) with the object ROI occluded and not occluded by the artificial plant, respectively. CNR of 5(d) and 5(e) and the median value of 72 elemental images are 2.68, 3.58, and 2.81, respectively. Likewise, for the sCMOS camera two of 2D elemental images are shown in Figs. 5(g) and 5(h) with the occluded object ROI and the visible object ROI, respectively. CNR of Figs. 5(g) and 5(h) and the median value of 72 elemental images are 2.88, 4.76, and 3.78, respectively. The median values of n_ph inside the object ROI are 46.7 and 36.1 photons detected/pixel for the EM-CCD camera and sCMOS camera respectively. The 3D reconstructed images at z = 4.1 m for the EM-CCD camera and the sCMOS camera are shown in Figs. 5(f) and 5(i), respectively. The occlusion is clearly removed in both images and CNR increases to 22.4 for the EM-CCD camera and to 19.0 for the sCMOS camera, which equals 7.95 times and 5.03 times improvement, respectively, when compared with the median values of CNR for the elemental images despite the presence of occlusion. Note this would not be possible through the summation of multiple 2D images taken from a single perspective. 3D imaging reconstruction allows one to separate out objects of interest in different planes especially in the presence of occlusion. This is in contrast to 2D imaging where the occlusion is superimposed on the objects of interest.

For the second scene, Scene 2, there was lower amount of illumination compared with the previous scene as the computer monitor was covered. Without occlusion, the median values of CNR of the elemental images are 2.59 and 0.374 for the EM-CCD camera and for the sCMOS camera, respectively, with corresponding median values of n_ph inside the object ROI of 25.8 and 1.50 photons detected/pixel. After 3D reconstruction CNR is improved to 22.3 for the EM-CCD camera and to 2.94 for the sCMOS camera. Figure 7(a) and 7(b) depict two of the elemental images for the EM-CCD camera captured in Scene 2 with occlusion present, in which the object ROI is occluded and visible, respectively. CNR of Figs. 7(a) and 7(b) and the median value for 72 elemental images are 0.48, 2.16, and 1.70, respectively with the median n_ph of 15.3 photons detected/pixel. For the sCMOS camera corresponding elemental images are shown in Figs. 7(d) and 7(e) with CNR of 0.62 and 0.79, respectively and the median is 0.562 with the median n_ph of 2.40 photons detected/pixel. Figure 7(c) is the 3D reconstructed image at z = 4.1 m for the EM-CCD camera in which CNR is improved by 7.38 times to 12.5 and Fig. 7(f) shows the corresponding reconstructed image for the sCMOS camera with the 7.98 times improvement of CNR to 4.49.

Fig. 7. Scene 2 using EM-CCD camera and sCMOS camera. (a) 2D elemental image acquired by the EM-CCD camera in which the object ROI is occluded by the artificial plant and (b) not occluded. (c) 3D reconstructed image at z = 4.1 m for the EM-CCD camera. CNR for (a), (b) and (c) are 0.48, 2.16 and 12.5, respectively. (d) 2D elemental image acquired by the sCMOS camera in which the object ROI is occluded by the artificial plant and (e) not occluded. (f) 3D reconstructed image at z = 4.1 m for the sCMOS camera. CNR for (d), (e) and (f) are 0.62, 0.79 and 4.49, respectively.

Download Full Size | PDF

A summary of the results is shown in Table 1. For the experiments without occlusion the range of fractional improvement of CNR is from 7.84 to 8.62. In the case where the distributions of the pixel values inside the ROI of each elemental image can be considered independent and identical, CNR is expected naively to improve by the factor of the square root of the number of elemental images, which is $\sqrt {72} = 8.49$ times.

Table 1. Summary of experimental results

View Table | View all tables in this article

To further assess the performance improvement with integral imaging, we use the Perception based Image Quality Evaluator (PIQE) to provide a no-reference image quality score of images [27]. This metric assumes an image suffers from additive Gaussian noise or blocking artifacts. For each scene, the PIQE score was calculated by considering the same pixel region about the mannequin’s head. Before calculation, normalization of the pixel values to [0, 1] without offset subtraction is applied to elemental images and 3D reconstructed images. From the PIQE scores shown in Table 2, it is evident the integral imaging reconstruction improves the image quality even in the presence of occlusion. For Scene 2 with the sCMOS camera, fractional improvement of the PIQE score of the 3D reconstructed image is smaller than those for other experiments. This is because in the low-light condition of a few photons detected per pixel, the sCMOS camera suffers from the low SNR of elemental images below 1.0 as shown in Fig. 6, which highlights the potential advantage of the EM-CCD camera for extreme low-light imaging as predicted in Fig. 6.

Table 2. Summary of PIQE scores

View Table | View all tables in this article

4. Conclusion

In summary, we have presented a theoretical and experimental investigation of 3D integral imaging in low-light-level conditions where no more than several tens of photons, and as little as a few photons were detected on average per pixel. The experiments were performed with two state-of-art high-sensitivity cameras, that is an EM-CCD camera and an sCMOS camera. Experimental data from the EM-CCD was fitted to the theoretically derived probability distribution of the pixel values, and pixelwise calibration of the sCMOS was performed to determine the camera parameters for further analysis. Moreover, theoretical derivation of the expected SNR was provided for each image sensor and confirmed by experiments. In the experiments, a partially occluded object was imaged in a low-light condition, then the scene illumination was further decreased. The object was again imaged using the 3D integral imaging system in the extreme low-light scene and as little as a few photons per pixel on average were detected. For both cameras, we have demonstrated that the image quality metrics such as the SNR and the Perception based Image Quality Evaluator of the 3D reconstructed images improved compared with those of the 2D imaging. Our experiments illustrated that using 3D integral imaging with dedicated image sensors and algorithms may be successful in recovering poorly illuminated scenes by substantially improving image quality metrics while also providing depth information of the object and partial occlusion removal which was not possible by 2D imaging even when state of the art image sensors were employed. Future work includes object recognition, 3D profilometry, a more in depth analysis of 3D imaging compared with 2D imaging, and integral imaging in the photon counting light-level condition which will be published in the near future. In addition, alternative three-dimensional imaging techniques will be investigated such as axially distributed sensing [29] which can be implemented using a single camera. Various metrics can be considered in future work [30].

Funding

Hamamatsu Photonics K.K.; U.S. Department of Education; Air Force Office of Scientific Research (FA9550-18-1-0338); Office of Naval Research (N000141712405, N00014-17-1-2561).

Acknowledgments

Timothy O’Connor wishes to acknowledge support from Education Department under Graduate Assistance in Areas of National Needs (GAANN). B. Javidi wishes to acknowledge support from Hamamatsu Photonics K. K., Air Force Office of Scientific Research (AFOSR) (FA9550-18-1-0338); Office of Naval Research (ONR) (N000141712405); and (ONR) (N00014-17-1-2561).

References

1. G. Lippmann, “Épreuves réversibles donnant la sensation du relief,” J. Phys. Theor. Appl. 7(1), 821–825 (1908). [CrossRef]

2. F. Okano, J. Arai, H. Hoshino, and I. Yuyama, “Three-dimensional video system based on integral photography,” Opt. Eng. 38(6), 1072–1077 (1999). [CrossRef]

3. A. Stern and B. Javidi, “Three-dimensional image sensing, visualization, and processing using integral imaging,” Proc. IEEE 94(3), 591–607 (2006). [CrossRef]

4. X. Xiao, B. Javidi, M. Martinez-Corral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications [invited],” Appl. Opt. 52(4), 546–560 (2013). [CrossRef]

5. M. Cho and B. Javidi, “Three-dimensional visualization of objects in turbid water using integral imaging,” J. Disp. Technol. 6(10), 544–547 (2010). [CrossRef]

6. Y. Zhao, X. Xiao, M. Cho, and B. Javidi, “Tracking of multiple objects in unknown background using Bayesian estimation in 3D space,” J. Opt. Soc. Am. A 28(9), 1935–1940 (2011). [CrossRef]

7. V. J. Traver, P. Latorre-Carmona, E. Salvador-Balaguer, F. Pla, and B. Javidi, “Human gesture recognition using three-dimensional integral imaging,” J. Opt. Soc. Am. A 31(10), 2312–2320 (2014). [CrossRef]

8. A. Stern, D. Aloni, and B. Javidi, “Experiments with three-dimensional integral imaging under low light levels,” IEEE Photonics J. 4(4), 1188–1195 (2012). [CrossRef]

9. B. Tavakoli, B. Javidi, and E. Watson, “Three dimensional visualization by photon counting computational integral imaging,” Opt. Express 16(7), 4426–4436 (2008). [CrossRef]

10. A. Markman, X. Shen, and B. Javidi, “Three-dimensional object visualization and detection in low light illumination using integral imaging,” Opt. Lett. 42(16), 3068–3071 (2017). [CrossRef]

11. J. A. Tyson, “Progress in low-light-level charge-coupled device imaging in astronomy,” J. Opt. Soc. Am. A 7(7), 1231–1236 (1990). [CrossRef]

12. B. T. Phillips, D. F. Gruber, G. Vasan, C. N. Roman, V. A. Pieribone, and J. S. Sparks, “Observations of in situ deep- sea marine bioluminescence with a high-speed, high-resolution sCMOS camera,” Deep Sea Res., Part I 111, 102–109 (2016). [CrossRef]

13. B. Fowler, C. Liu, S. Mims, J. Balicki, W. Li, H. Do, J. Appelbaum, and P. Vu, “A 5.5Mpixel 100 frames/sec wide dynamic range low noise CMOS image sensor for scientific applications,” Proc. SPIE 7536, 753607 (2010). [CrossRef]

14. P. Jerram, P. J. Pool, R. Bell, D. J. Burt, S. Bowring, S. Spencer, M. Hazelwood, I. Moody, N. Catlett, and P. S. Heyes, “The LLCCD: low-light imaging without the need for an intensifier,” Proc. SPIE 4306, 178–186 (2001). [CrossRef]

15. E. Ciarrocchi and N. Belcari, “Cerenkov luminescence imaging: physics principles and potential applications in biomedical sciences,” EJNMMI Phys. 4(1), 14 (2017). [CrossRef]

16. M. P. Edgar, D. S. Tasca, F. Izdebski, R. E. Warburton, J. Leach, M. Agnew, G. S. Buller, R. W. Boyd, and M. J. Padgett, “Imaging high-dimensional spatial entanglement with a camera,” Nat. Commun. 3, 984 (2012). [CrossRef]

17. S. Saurabh, S. Maji, and M. P. Bruchez, “Evaluation of sCMOS cameras for detection and localization of single Cy5 molecules,” Opt. Express 20(7), 7338–7349 (2012). [CrossRef]

18. F. Huang, T. M. P. Hartwich, F. E. Rivera-Molina, Y. Lin, W. C. Duim, J. J. Long, P. D. Uchil, J. R. Myers, M. A. Baird, W. Mothes, M. W. Davidson, D. Toomre, and J. Bewersdorf, “Video-rate nanoscopy using sCMOS camera-specific single-molecule localization algorithms,” Nat. Methods 10(7), 653–658 (2013). [CrossRef]

19. C. D. Mackay, R. N. Tubbs, R. Bell, D. J. Burt, P. Jerram, and I. Moody, “Subelectron read noise at MHz pixel rates,” Proc. SPIE 4306, 289–298 (2001). [CrossRef]

20. J. Hynecek and T. Nishiwaki, “Excess noise and other important characteristics of low light level imaging using charge multiplying CCDs,” IEEE Trans. Electron Devices 50(1), 239–245 (2003). [CrossRef]

21. M. S. Robbins and B. J. Hadwen, “The noise performance of electron multiplying charge-coupled devices,” IEEE Trans. Electron Devices 50(5), 1227–1232 (2003). [CrossRef]

22. A. G. Basden, C. A. Haniff, and C. D. Mackay, “Photon counting strategies with low-light-level CCDs,” Mon. Not. R. Astron. Soc. 345(3), 985–991 (2003). [CrossRef]

23. T. Plakhotnik, A. Chennu, and A. V. Zvyagin, “Statistics of single-electron signals in electron-multiplying charge- coupled devices,” IEEE Trans. Electron Devices 53(4), 618–622 (2006). [CrossRef]

24. K. B. W. Harpsøe, M. I. Andersen, and P. Kjægaard, “Bayesian photon counting with electron-multiplying charge coupled devices (EMCCDs),” Astron. Astrophys. 537, A50 (2012). [CrossRef]

25. S. Watanabe, T. Takahashi, and K. Bennett, “Quantitative evaluation of the accuracy and variance of individual pixels in a scientific CMOS (sCMOS) camera for computational imaging,” Proc. SPIE 10071, 100710Z (2017). [CrossRef]

26. F. A. Rosell, “Limiting resolution of low-light-level imaging sensors,” J. Opt. Soc. Am. 59(5), 539–547 (1969). [CrossRef]

27. N. Venkatanath, D. Praneeth, M. C. Bh, S. S. Channappayya, and S. S. Medasani, “Blind image quality evaluation using perception based features,” in Proceedings of IEEE Conference on Communications (IEEE, 2015), pp. 1–6.

28. A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Trans. on Image Process. 21(12), 4695–4708 (2012). [CrossRef]

29. R. Schulein, M. DaneshPanah, and B. Javidi, “3D imaging with axially distributed sensing,” Opt. Lett. 34(13), 2012–2014 (2009). [CrossRef]

30. X. Li, M. Zhao, Y. Xing, L. Li, S.-T. Kim, X. Zhou, and Q.-H. Wang, “Optical encryption via monospectral integral imaging,” Opt. Express 25(25), 31516–31527 (2017). [CrossRef]

Scene	Camera	Without occlusion			With occlusion
Scene	Camera	EI^a	3D^b	Fractional Improvement^c	EI^a	3D^b	Fractional Improvement^c
Scene 1	EM-CCD	57.31	18.88	3.03	64.94	10.93	5.94
Scene 1	sCMOS	62.82	16.57	3.79	60.35	14.22	4.24
Scene 2	EM-CCD	71.12	9.33	7.62	62.65	7.79	8.04
Scene 2	sCMOS	62.39	31.94	1.95	61.84	41.25	1.50

Scene	Camera	Without occlusion			With occlusion
Scene	Camera	EI^a	3D^b	Fractional Improvement^c	EI^a	3D^b	Fractional Improvement^c
Scene 1	EM-CCD	57.31	18.88	3.03	64.94	10.93	5.94
Scene 1	sCMOS	62.82	16.57	3.79	60.35	14.22	4.24
Scene 2	EM-CCD	71.12	9.33	7.62	62.65	7.79	8.04
Scene 2	sCMOS	62.39	31.94	1.95	61.84	41.25	1.50

Three-dimensional integral imaging in photon-starved environments with high-sensitivity image sensors

Abstract

1. Introduction

2. Experimental method

2.1 High-sensitivity cameras and calibration methods

2.1.1 EM-CCD camera and calibration results

2.1.2. sCMOS camera and calibration results

2.2 Image evaluation metrics

2.3. Three-dimensional integral imaging

3. Results and discussion

4. Conclusion

Funding

Acknowledgments

References

Cited By

Figures (7)

Tables (2)

Equations (11)

Optics Express

Experiment		Camera	n_ph^a	CNR-object
				EI^b	3D^c	Fractional Improvement^d
Without Occlusion	Scene 1	EM-CCD	60.9	3.60	30.0	8.31
	Scene 1	sCMOS	27.7	3.34	26.6	7.96
	Scene 2	EM-CCD	25.8	2.59	22.3	8.62
	Scene 2	sCMOS	1.50	0.374	2.94	7.84
With Occlusion	Scene 1	EM-CCD	46.7	2.81	22.4	7.95
	Scene 1	sCMOS	36.1	3.78	19.0	5.03
	Scene 2	EM-CCD	15.3	1.70	12.5	7.38
	Scene 2	sCMOS	2.40	0.562	4.49	7.98