Three-dimensional polarimetric integral imaging in photon-starved conditions: performance comparison between visible and long wave infrared imaging

Kashif Usmani; Timothy O’Connor; Xin Shen; Pete Marasco; Artur Carnicer; Dipak Dey; Bahram Javidi

doi:10.1364/OE.395301

1. Introduction

Polarimetric imaging has been used in many applications such as object recognition, materials classification, and segmentation [1–3]. Most of the light reflected from objects is predominantly polarized and may contain information about the surface of the materials, therefore, the polarization properties of light can be used in object classification [1,2]. The polarization state of reflected beam can be described in terms of Stokes parameters and degree of polarization (DoP) [4]. Four quantities of Stokes parameters (S₀, S₁, S₂ and S₃) are required to measure the DoP. These quantities can be recorded by placing polarizing elements in front of an image sensor, and recording images with different orientations of the polarizing elements. In low light conditions, long wave infrared (LWIR) imaging sensors are commonly employed to obtain the polarimetric information of objects [5–7]. Employment of infrared (IR) imaging in low light is suitable for DoP calculation; however, the IR imaging systems can be bulky and expensive in comparison with visible range imaging systems. The measurement of DoP using visible cameras could be an effective technique in photon-starved conditions but the dominance of camera noise in photon-starved conditions can degrade the accuracy of DoP calculations. Passive three-dimensional (3D) polarimetric integral imaging (InIm) [8–11] is one of the prominent techniques to overcome the aforementioned issues with visible cameras operating in photon-starved conditions. 3D InIm [12–22] is performed by recording two-dimensional (2D) images from different perspectives, then reconstructing the three-dimensional (3D) scene through either optical or computational reconstruction methods. Different perspective images known as elemental images can be captured by a single image sensor with a lenslet or camera array, or by a single moving image sensor [12–22]. The computational reconstruction process is performed by the back projection of the optical rays through a virtual pinhole to the desired reconstruction distance [23].

In this paper, we compare a visible range polarimetric 3D integral imaging system with an LWIR polarimetric 3D integral imaging system in low light conditions in terms of signal-to-noise ratio (SNR) and polarimetric object detection. Passive 3D polarimetric InIm [8–11] is used to extract the polarimetric information of visible and LWIR polarimetric objects in low light conditions. Since the captured visible range images are significantly degraded by read noise in the photon-starved conditions, it becomes challenging to calculate the Stokes parameters and the corresponding DoP. In order to extract the polarimetric information for a visible range camera in these conditions, a mathematical model for signal restoration is applied to each elemental image [24–26]. Moreover, the camera read noise is reduced during 3D InIm reconstruction of polarimetric images, which is optimal in the maximum likelihood sense [27]. Furthermore, the total variation (TV) denoising [28] algorithm is used for noise reduction of DoP images for the visible range InIm systems. The mathematical model for signal restoration and TV denoising are not applied in case of the LWIR imaging system because enough thermal photons are present in the scene.

Our experiments for polarimetric 3D integral imaging under low light conditions indicate that the visible range imaging system produces an SNR that may be higher than the LWIR imaging system. In our experiments, we also find that a small polarimetric object may be detectable using the visible range 3D integral imaging system but is barely detectable by the LWIR 3D integral imaging system. Furthermore, we derive the theoretical probability density functions (PDF) for the 2D and 3D DoP images and show strong similarity between the theoretically derived distributions and the probability distribution functions found through experimental measurements.

2. Theoretical background of degree of polarization (DoP)

The Stokes parameters provide quantitative measurements of polarimetric imaging [4]. Let us define the expression for quasi-monochromatic light propagating along the z direction as

(1)$$\overrightarrow {\mathbf E} (z,t) = {E_{ox}}\cos [{({kz - \omega t} )+ {\phi_x}} ]\widehat {\mathbf x} + {E_{oy}}\cos [{({kz - \omega t} )+ {\phi_y}} ]\widehat {\mathbf y},$$

where $\widehat {\mathbf x}$ and $\widehat {\mathbf y}$ are orthogonal unit vectors, E_0x and E_0y are the amplitude of electromagnetic components in the $\widehat {\mathbf x}$ and $\widehat {\mathbf y}$ directions, respectively, and ϕ_x and ϕ_y are the corresponding phase terms. The Stokes parameters for quasi-monochromatic light are defined as:

(2)$$\begin{array}{l} {S_0} = \left\langle {E_{0x}^2} \right\rangle + \left\langle {E_{0y}^2} \right\rangle = {I^{0^\circ }} + {I^{90^\circ }}\\ {S_1} = \left\langle {E_{0x}^2} \right\rangle - \left\langle {E_{0y}^2} \right\rangle = {I^{0^\circ }} - {I^{90^\circ }}\\ {S_2} = \left\langle {2{E_{0x}}{E_{0y}}\cos \phi } \right\rangle = {I^{45^\circ }} - {I^{135^\circ }}\textrm{ }\\ {S_3} = \left\langle {2{E_{0x}}{E_{0y}}\sin \phi } \right\rangle = {I^{45^\circ ,{\raise0.7ex\hbox{$\pi $} \!\mathord{\left/ {\vphantom {\pi 2}} \right.}\!\lower0.7ex\hbox{$2$}}}} - {I^{135^\circ ,{\raise0.7ex\hbox{$\pi $} \!\mathord{\left/ {\vphantom {\pi 2}} \right.}\!\lower0.7ex\hbox{$2$}}}}, \end{array}$$

where $\left\langle \textrm{.} \right\rangle $ represents time averaging and ϕ = ϕ_y - ϕ_x is the phase difference between the two orthogonal components of the electric field. S_i (i=0,1,2,3) denotes the Stokes parameters. I^θ is the intensity of polarized light recorded when the linear polarizer in front of the imaging sensor is placed at an angle of θ with respect to the x-axis, and I^{θ, π/2} is the intensity recorded after inserting the quarter wave plate (QWP) in addition to the linear polarizer. The degree of polarization (DoP) and degree of linear polarization (DoLP) in terms of Stokes parameters are defined as:

(3)$$\textrm{DoP} = \frac{{\sqrt {S_1^2 + S_2^2 + S_3^2} }}{{{S_0}}},$$

(4)$$\textrm{DoLP} = \frac{{\sqrt {S_1^2 + S_2^2} }}{{{S_0}}}.$$

The value of DoP ranges from 0 to 1, with 1 representing completely polarized light and 0 representing completely unpolarized light. Generally, in passive polarimetric imaging applications, circularly polarized light S₃ is very small and rarely measurable. As such, in these experiments, the value of S₃ is taken to be zero. When the value of S₃ is assumed as zero, the equations for DoLP and DoP become interchangeable. Therefore, in the following sections, all analysis is carried out for the DoP values with S₃ set to zero, and only four polarimetric images [I^0°, I^45°, I^90°, I^135°] are required for each perspective.

3. Experimental methods

The 3D scene consists of a linear polarizer film, a mannequin, and test tubes filled with hot water to provide an IR source, as shown by a reference image taken in high illumination conditions using the low noise visible range camera (Hamamatsu C11440-42U) by Fig. 1(a). Figure 1(b) shows the same 3D scene imaged using the visible range camera in low light illumination conditions. The estimated number of photons per pixel under low illumination conditions was calculated as 1.23 photons/pixel [10,29]. For the LWIR imaging, the same 3D scene is used, however the linear polarizer is replaced by an IR wire grid polarizer in the wavelength range of 7-15µm. The 3D scene is imaged using the LWIR camera (Tamarisk 320 LWIR camera 60 Hz) as shown in Fig. 1(c). Note, the ambient illumination in Fig. 1(c) is the same as that of Fig. 1(b) but the LWIR camera is not affected by the scene illumination. In this paper, the polarimetric visible and LWIR images were captured using synthetic aperture integral imaging (SAII) [23] with the visible range camera and the LWIR range camera in photon starved conditions. SAII consists of a camera on a moving platform [9–10] as shown in Fig. 2(a). Also shown in Fig. 2(a), a polarizer is placed in front of the imaging sensor for polarimetric imaging. The SAII pickup process of a 3D scene is shown in Fig. 2(b) and reconstruction of the 3D scene using the SAII algorithm is shown in Fig. 2(c). To provide a fair comparison of DoP between the visible and LWIR cameras, the pixel size of the visible camera is binned with 3 by 3 binning to 19.5${\times} $19.5 µm in order to be comparable with the LWIR camera which has a pixel size 17${\times} $17 µm. Both cameras use nearly the same f-number to capture the polarimetric object.

Fig. 1. (a) Reference scene in full illumination, (b) the scene in Fig. 1(a) under low illumination using the visible range camera, and (c) the scene in (a) using the LWIR camera.

Download Full Size | PDF

Fig. 2. Experimental setup of polarimetric 3D imaging system: (a) polarimetric integral imaging system, (b) pick-up process and (c) reconstruction process.

Download Full Size | PDF

In 3D polarimetric imaging reconstruction process, the 3D image I_z^θ(x, y) at the reconstruction plane z from the camera system is expressed as [23]:

(5)$$I_z^\theta (x,y) = \frac{1}{{O(x,y)}}\sum\limits_{m = 0}^{M - 1} {\sum\limits_{n = 0}^{N - 1} {\left[ {I_{m,n}^\theta \left( {x - \frac{{m \times {L_x} \times {p_x}}}{{{c_x} \times z/f}},y - \frac{{n \times {L_y} \times {p_y}}}{{{c_y} \times z/f}}} \right) + \varepsilon } \right]} } .$$

In this equation, M and N are the number of polarimetric elemental images in the x- and y- directions and O(x, y) is the overlapping pixel number at (x, y). p_x and p_y are the camera pitch size in horizontal (H) and vertical (V) directions respectively. L_x and L_y are the total number of pixels in each column and row on images, c_x × c_y is the sensor size of camera. The focal length of camera lens is f, and ɛ is the additive camera noise. I^θ_m,n is the ideal 2D polarimetric elemental image, θ represents the different polarization states of the image while the subscripts m, n represent the location of elemental image. For 3D integral imaging in these experiments, a total of 25 perspective images [5(H) × 5(V)] with a camera pitch of 30 mm in both directions are recorded. Table 1 summarizes the values of different integral imaging and camera parameters used in the experiments.

Table 1. Integral imaging and camera variables

View Table | View all tables in this article

When using a visible range sensor in low light illumination conditions, very few photons are reflected from the scene and may be attenuated by the atmospheric particles in the environment, resulting in captured images that are dominated by camera read noise. Therefore, the captured images using a visible camera in photon-starved conditions become degraded and sparse. In photon-starved conditions, the electron counts are very small due to the low number of photons reflected from the object scene. To alleviate the problem of camera noise, we apply a mathematical signal restoration model. Before applying the signal restoration model, first we must subtract the camera offset from the captured elemental images. The camera offset is required to prevent the clipping of very small signals during image capture that would otherwise occur because some signals, due to noise, may be less than zero when digitized [30]. This offset can be measured by recording of large number of single camera bias reference frames, which are obtained by setting the exposure time of camera to be a minimum (3 ms) and closing the aperture to avoid the atmospheric photons [10,30]. After subtraction of the camera offset image, a mathematical model for signal restoration in low light conditions is applied to restore the visibility of the 2D elemental images in photon-starved conditions.

Dark channel prior-based mathematical models for haze removal are widely used for image restoration and enhancement [24]. Additionally, it has been reported that the dark channel prior method is very useful for low light image enhancement [25,26]. The dark channel prior method was derived from the statistics of outdoor images. From the set of outdoor images, it was shown that in most of sky-free regions of the images, some pixels (defined as dark pixels) have very low intensity in at least one of the color (RGB) channels. The process of selecting the minimum intensity pixels from an image serves as a prior, and the method of image enhancement is referred to as the dark channel prior method [24]. The prior information of the dark pixels can provide an accurate estimation of transmission function caused by the haze. This dehazing model has been used to enhance the image in low light illumination conditions since inversion of the low light image results in an image that appears very similar to a hazy image [25,26]. Therefore, in this work, we apply the dark channel prior method to the inverted low light images to enhance the image in photon starved conditions.

Since we are using the Hamamatsu C11440-42U sensor, which is a monochromatic image sensor, the monochromatic version of dark channel prior method is used to recover the image. Instead of performing the optimization over all the channels, in the monochromatic version of dark channel prior method, the minimization of global cost function is performed over the single channel. For the mathematical model of signal restoration, let us consider I(ξ) as the image to be recorded in photon starved condition where ξ is the vector representation of the pixel location (x, y). To facilitate the signal recovery using the monochromatic version of dark channel prior mathematical restoration model, we invert the observed image intensity I(ξ) using R(ξ) = 255-I(ξ). The inverted image R(ξ) is similar to the hazy image and fed into the widely used signal restoration model [24–26]:

(6)$${\bf R}({\boldsymbol{\mathrm{\xi}}})\textrm{ = }{\bf J}({\boldsymbol {\mathrm{\xi}}})t({\boldsymbol {\mathrm{\xi}}})\textrm{ + }{\bf A}(\textrm{1 - }t({\boldsymbol {\mathrm{\xi}}}))\textrm{,}$$

where R(ξ) is the recorded intensity of the inverted input image, J(ξ) is the ideal image to be recovered, A is the atmospheric light, and t(ξ) is portion of light that is not scattered and reaches the camera. Using the monochromatic version of dark channel prior, A and t(ξ) can be estimated. The transmission t(ξ) value ranges from 0 to 1, for complete dark regions, t(ξ) becomes 0 and for brightest regions, t(ξ) becomes 1. We have chosen the top 0.1% of brightest pixels in R^dark. The highest pixel value among the brightest pixels is selected as estimated atmospheric light A′. The final image can be recovered with improved quality in comparison to the original inverted image R(ξ) by estimating J(ξ) using the above method. Recovered image J(ξ) is then inverted back to the final enhanced image S(ξ) by S(ξ) = 255-J(ξ).

The noise in the processed images are further reduced using TV denoising algorithm [28]. The TV denoising algorithm is applied to the four polarimetric 2D images [I^0°, I^45°, I^90°, I^135°] in the 2D case, and after 3D reconstruction of polarimetric images [I⁰_z^°, I⁴⁵_z^°, I⁹⁰_z^°, I¹³⁵_z^°], before calculating the 3D DoP image, in the 3D case.

4. Results and discussion

4.1 DoP calculation in low light using visible and LWIR range cameras

For the experiments conducted using the visible range sensor, an average of 1.23 photons per pixel were estimated prior to pixel binning [10,29]. The captured visible 2D elemental images are used to calculate the 2D Stokes parameters and 2D DoP using Eq. (2) and Eq. (4). When the number of photons per pixel is low, the noise present in the image is enhanced during the DoP calculation due to the nonlinear combination of the Stokes parameters. In this case, the polarimetric information in 2D image is very noisy and produces erroneous results [see Fig. 3(a)]. The reconstructed 3D integral image is obtained from the 2D elemental images using Eq. (5) and the 3D Stokes parameters and the 3D DoP image are calculated using Eq. (2) and Eq. (4), respectively. The amount of noise in the 3D DoP image [see Fig. 3(b)] is reduced in comparison to the 2D DoP image; but it is still noisy. Further reduction of noise in the DoP images can be achieved using dark channel modelling along with total variation (TV) denoising algorithm. To further demonstrate the proposed method of dark channel modeling along with TV denoising in comparison to prior arts, we compare the 2D DoP and 3D DoP images with no processing, using only dark channel modeling, using only TV denoising and using the combination of dark channel modeling and TV denoising (i.e. the proposed method). Figure 3 shows the 2D DoP and 3D DoP images, respectively, for results with no processing [Figs. 3(a), 3(b)], using only dark channel modeling processing [Figs. 3(c), 3(d)], using only total variation denoising processing [Figs. 3(e), 3(f)] and using the proposed method of both dark channel modeling and total variation denoising processing [Figs. 3(g), 3(h)].

Fig. 3. Visible polarimetric imaging comparison in low light illumination conditions for the scene in Fig. 1(a). (a) 2D DoP image and (b) 3D DoP image without any processing. (c) 2D DoP image and (d) 3D DoP image using dark channel modeling. (e) 2D DoP image and (f) 3D DoP image with only TV denoising. (g) 2D DoP image and (h) 3D DoP image using both dark channel modeling and TV denoising. The estimated photons/pixel is 1.23 prior to binning.

Download Full Size | PDF

For quantitative comparison of each part of the image processing pipeline, the signal-to-noise ratio (SNR) of polarimetric (signal) and non-polarimetric (background) areas is measured. The SNR is defined as $\textrm{SNR} = ({\mu _s} - {\mu _b})/\sqrt {\sigma _s^2 + \sigma _b^2}$, where the mean of the polarimetric signal is µ_s, and the mean of the non-polarimetric area (background) is µ_b. σ_s² and σ_b² are the variances of the polarimetric and non-polarimetric areas, respectively [10,29]. For fair comparison of the visible range DoP images using different image processing pipelines, the same image regions were considered within the red (signal) and yellow (background) windows selected from DoP images as shown in Fig. 3(b). The dark channel modeling alone provides a small improvement in the SNR of the visible DoP image. The SNR of the visible DoP images is significantly improved when using the TV denoising algorithm. Further improvement of noise reduction can be achieved by using dark channel modeling along with TV denoising. Therefore, all the experimental results of visible DoP images in photon-starved condition are processed using both dark channel modeling and TV denoising. The results of the SNR comparison for different processing pipelines are shown in Table 2.

Table 2. SNR comparison of polarimetric imaging in visible range^a

View Table | View all tables in this article

The process of polarimetric imaging with the LWIR range camera is the same as that of the visible range camera except that the mathematical restoration model and TV denoising are not needed for the LWIR sensor because there are sufficient thermal photons in the scene. Visible polarimetric images are corrupted by camera read noise which has Gaussian distribution in low light conditions. However, the polarimetric images in case of LWIR range were recorded in sufficient thermal photon conditions. Therefore, TV denoising in addition of dark channel were applied only on the visible data to remove the Gaussian noise in photon-starved conditions. Moreover, after applying the denoising algorithms in the case of the LWIR images, there may be a reduction in the SNR of the DoP images. The 2D elemental images recorded with an LWIR camera are used to reconstruct 3D Integral Images using Eq. (5). The Stokes parameters and DoP of LWIR range camera are calculated using Eq. (2) and Eq. (4), respectively. For the scene in Fig. 1(a), the 2D and 3D DoP images in the LWIR range are illustrated in Fig. 4 for results with no processing [Figs. 4(a), 4(b)], using only dark channel prior [Figs. 4(c), 4(d)], using only total variation denoising [Figs. 4(e), 4(f)], and using the proposed method of both dark channel prior and total variation denoising [Figs. 4(g), 4(h)]. For quantitative comparison of each part of the image-processing pipeline using the LWIR sensor, the SNR between the polarimetric (signal) and non-polarimetric (background) areas is measured. From the results shown in Fig. 4 and Table 3, it is evident that there is no benefit of applying the dark channel prior method and TV denoising to the LWIR data. The highest SNR is achieved for the 3D DOP image with no additional processing.

Fig. 4. Polarimetric imaging of LWIR range for the scene Fig. 1(a). (a) 2D DoP image and (b) 3D DoP image without any processing. (c) 2D DoP image and (d) 3D DoP image using dark channel modeling. (e) 2D DoP image and (f) 3D DoP image with only TV denoising. (g) 2D DoP image and (h) 3D DoP image using both dark channel modeling and TV denoising. The red (signal) and yellow (background) windows are selected from DoP images for the calculation of SNR.

Download Full Size | PDF

Table 3. SNR comparison of polarimetric imaging in LWIR range

View Table | View all tables in this article

4.2 Summary of comparison between 3D visible and 3D LWIR polarimetric imaging systems

For an imaging system with a fixed f-number and a fixed object distance, the resolution of imaging systems is determined by the pixel size and wavelength of light source [31]. Since the pixel size and f-number of our imaging systems were nearly the same, the resolutions of our systems are expected to differ only based on the wavelength of light source. The LWIR range camera shows a low polarimetric signature of the object as shown by the 3D DoP image [see Fig. 4(b)]. However, the visible range camera displays a comparatively high polarimetric signature of polarimetric object as shown by the 3D DoP image [see Fig. 3(h)] using the proposed method of both dark channel modeling and total variation denoising. Moreover, the polarimetric information in the visible range 3D DoP image is comparatively enhanced, and the detailed information is recovered.

We quantitatively compared the DoP images from the LWIR and visible range cameras by measuring the SNR of 3D DoP images. The maximum SNR achieved in the visible range is 23.25 of 3D DoP image [see Fig. 3(h)] using the dark channel modeling along with TV denoising. However, the maximum SNR achieved in the LWIR range is 8.24 of 3D DoP image [see Fig. 4(b)]. Therefore, in our experiments, an improvement of 182.16% was achieved in SNR using the 3D DoP image in the visible range in comparison to the 3D DoP image in the LWIR range.

In order to provide further comparison of polarization capabilities for both the visible and LWIR range cameras, a one-dimensional line plot of the DoP intensity is examined. An aperture with 3 mm diameter was placed in front of the polarimetric object as shown by a reference image taken in high illumination conditions using low noise visible range camera by Fig. 5(a). The experiments were repeated to record the polarimetric information in the visible range and the LWIR range as shown in Figs. 5(b) and 5(d), respectively. The visible range camera provides the ability to identify the polarimetric object of size 3 mm and has a higher DoP value as shown by the 1D intensity plot in Fig. 5(c). However, using the LWIR range camera, the polarimetric object of size 3 mm is barely detectable, and 1D intensity plot of the DoP value [Fig. 5(e)] is nearly constant around zero.

Fig. 5. (a) Reference scene of a polarizer with an aperture of diameter of 3 mm in full illumination. (b) Visible 3D DoP image under low illumination condition, (c) the corresponding 1D intensity line plot, (d) LWIR 3D DoP image, (e) the corresponding 1D intensity line plot. The estimated photons/pixel is 1.23 prior to binning.

Download Full Size | PDF

4.3 Statistical analysis of 2D and 3D DoP images

Lastly, we propose a statistical approach to derive the statistical distribution of the 2D and 3D DoP images in low light conditions. Under low light conditions, the captured visible 2D image is dominated with camera read noise, which has a Gaussian distribution. To verify the distribution of 2D image, a Kolmogorov–Smirnov test [32] was applied to the 2D elemental image in Fig. 1(b). The polarimetric region was selected from 2D elemental image as a region of interest (ROI). The result shows that the test accepted the null hypothesis at significance level of 5%. Furthermore, in the following derivations for the degree of linear polarization, we assume S₃=0 (i.e. no circular polarization). Therefore, only S_i [i = 0, 1, 2] will be considered and the probability density function of polarimetric region in the DoP image for both the 2D and 3D cases can be written as [33]:

(7)$$f(P) = \sum\limits_{j = 0}^\infty {\sum\limits_{k = 0}^\infty {\frac{{\left( {{e^{ - \frac{\delta }{2}}}{{\left( {\frac{\delta }{2}} \right)}^j}} \right)\left( {{e^{ - \frac{\gamma }{2}}}{{\left( {\frac{\gamma }{2}} \right)}^k}} \right) \times 2{P^{1 + 2j}}{{({1 + {P^2}} )}^{ - 3/2 - j - k}}}}{{j!k!B({1 + j,1/2 + k} )}}} } ,$$

where P represents the DoP of the image and the non-centrality parameters δ =∑²_i=1 (µ_i /σ)² and γ = (µ₀ /σ)², where i is the index of Stokes parameter. B(.) is the beta function, j and k are the variables for summation operator. The details of this derivation and approach are described in Appendix A. In the 3D case, the mean and variance of the 3D polarimetric image are derived by the following equations:

(8)$$\mu _z^\theta (x,y) = \frac{1}{{M \times N}}\sum\limits_{m = 0}^{M - 1} {\sum\limits_{n = 0}^{N - 1} {\left( {I_{m,n}^\theta \left( {x - \frac{{m \times {L_x} \times {p_x}}}{{{c_x} \times z/f}},y - \frac{{n \times {L_y} \times {p_y}}}{{{c_y} \times z/f}}} \right) + \varepsilon } \right)} } ,$$

(9)$${\mathop{\rm var}} _z^\theta (x,y) = \frac{1}{{{{({M \times N} )}^2}}}\sum\limits_{m = 0}^{M - 1} {\sum\limits_{n = 0}^{N - 1} {{\mathop{\rm var}} \left( {I_{m,n}^\theta \left( {x - \frac{{m \times {L_x} \times {p_x}}}{{{c_x} \times z/f}},y - \frac{{n \times {L_y} \times {p_y}}}{{{c_y} \times z/f}}} \right) + \varepsilon } \right)} } .$$

The intensity value µ_z^θ(x,y) at pixel (x,y) is averaged from the 2D elemental images [10]. All the parameters in Eq. (8) and Eq. (9) are the same as those in Eq. (5).

For experimental measurement of histograms, the regions of interest containing the polarizing element in the visible range 2D DoP image shown in Fig. 3(g), and the 3D DoP image shown in Fig. 3(h). The experimental histograms and computed theoretical PDFs are compared in Fig. 6. In Fig. 6(a), the red curve shows the theoretically derived PDF of the 2D DoP using Eq. (7), where µ₀ = 0.35, µ₁= 0.158, µ₂ = 0.026 and σ² = 1.5${\times} $10⁻³. These values were computed from the experimental 2D polarimetric image. For the theoretically derived PDF of the 3D DoP [Fig. 6(b)], the mean and variance of 3D reconstructed images were computed using Eq. (8) and Eq. (9) respectively, then the non-centrality parameters were computed using the mean and the square root of the variance of 3D reconstructed image. Finally, the computed theoretical PDF of the 3D DOP is compared with the experimental histogram [Fig. 6(b)].

Fig. 6. Theoretically derived (red curve) PDF, and experimentally measured (histogram in blue) PDF of polarization of (a) 2D DoP image [see Fig. 3(g)], and (b) 3D DoP image [see Fig. 3(h)] in low light condition.

Download Full Size | PDF

The experimental results agree well with the theoretically derived model. The differences may be due to the finite number of samples used to compute the histograms, and the assumption of statistical independence of elemental images in obtaining the theoretical results. The smaller standard deviation of PDF of 3D image confirms that 3D DoP has less noise compared to the 2D DoP image. Therefore, 3D polarimetric InIm can improve the DoP image quality over 2D DoP. This analysis agrees with the prior experimental results in section 4.1, which showed that the 3D DoP image may outperform the 2D DoP image in terms of SNR.

5. Conclusion

In this paper, passive 3D polarimetric integral imaging in the visible range and LWIR range have been compared in low light conditions. In order to extract the polarimetric information using a visible imaging sensor in photon-starved conditions, a mathematical model for signal restoration of elemental images is used. The total variation (TV) denoising is used for noise reduction of the 2D and 3D DoP images in the visible range. Quantitative comparison between polarimetric 3D integral imaging using a low noise visible band sCMOS camera and an LWIR band camera was presented. The polarimetric signature of a 3 mm-sized object was detected with the visible range camera and was barely detectable by the LWIR camera in low light conditions. The visible range camera similarly outperforms the LWIR range camera in detection of a polarimetric signature in low light conditions based on the calculated SNR for each camera. Finally, theoretically derived and experimentally measured probability distribution functions of the 2D and 3D DoP images were presented and compared. This comparison shows agreement between the experimental results and the theoretically derived models. To the best of our knowledge, this is the first report comparing the polarimetric imaging performance between visible range and IR range sensors under photon-starved conditions. Future work may examine the integration of the preprocessing steps using a joint numerical procedure rather than a piece-wise strategy with the integral imaging reconstruction algorithm for optimized performance.

Appendix A: theoretical statistical distribution of DoP image

In low light illumination conditions, the images recorded at different polarizing angles are corrupted with camera read noise, which has the Gaussian distribution. The Stokes parameters are the addition and subtraction of two orthogonal polarimetric images. Therefore, the calculated Stokes parameters are read noise dominated with Gaussian distribution with same variance and different means. The experimentally measured distribution (histogram) of Stokes parameters (S₀, S₁ and S₂) in photon starved condition without any processing is shown in Fig. 7. The histograms in Fig. 7 show the Stokes parameters follow the Gaussian distribution with nearly same variance and different means.

Fig. 7. Experimentally measured distribution of Stokes parameters S₀ (histogram in blue), S₁ (histogram in green) and S₂ (histogram in red).

Download Full Size | PDF

Assuming I^0° and I^90° are independent of each other, the variances of S₀ and S₁ are defined as,

(10)$$\sigma _{{S_0}}^2 = \sigma _{{S_1}}^2 = \sigma _{{I^{0^\circ }}}^2 + \sigma _{{I^{90^\circ }}}^2,$$

where $\sigma _{{S_0}}^2$and $\sigma _{{S_1}}^2$are the variances of Stokes parameters S₀ and S₁, respectively and $\sigma _{{I^{0^\circ }}}^2$and $\sigma _{{I^{90^\circ }}}^2$are the variances of two orthogonal polarimetric 2D elemental images. Experimentally, the variances of Stokes parameters are found to be approximately equal (see Fig. 7).

The Stokes parameters for the polarimetric area of a scene follow the Gaussian distribution with non-zero mean S_i ∼N (µ_i, σ²) [i = 0, 1, 2].

Let us consider,

(11)$$P = \textrm{DoP} = \sqrt {\frac{{S_1^2 + S_2^2}}{{S_0^2}}} = \sqrt {\frac{{\frac{1}{2}\left( {\frac{{S_1^2}}{{{\sigma^2}}} + \frac{{S_2^2}}{{{\sigma^2}}}} \right)}}{{S_0^2/{\sigma ^2}}}} \sqrt 2 ,$$

where $\frac{{S_1^2}}{{{\sigma ^2}}} + \frac{{S_2^2}}{{{\sigma ^2}}} \sim \chi _{2,\delta }^2$ is the non-central chi-squared distribution with two degrees of freedom and the non-centrality parameter δ =∑²_i=1 (µ_i /σ)². $\frac{{S_0^2}}{{{\sigma ^2}}} \sim \chi _{1,\gamma }^2$ is the non-central chi-squared distribution with one degree of freedom and non-centrality parameter γ = (µ₀ /σ)².

Let $Q = \frac{{\frac{1}{2}\left( {\frac{{S_1^2}}{{{\sigma^2}}} + \frac{{S_2^2}}{{{\sigma^2}}}} \right)}}{{S_0^2/{\sigma ^2}}} \sim {F_{2,1,\delta ,\gamma }},$ is the doubly non-central F-distribution with non-centrality parameters δ =∑²_i=1 (µ_i /σ)² and γ = (µ₀ /σ)². The numerator and denominator of doubly non-central F-distribution are statistically independents.

Therefore, the DoP becomes $P = \sqrt 2 \sqrt Q $ and $\frac{{dQ}}{{dP}} = P.$

The probability density function of doubly non-central F-distribution [33] is given as

(12)$$f(Q) = \sum\limits_{j = 0}^\infty {\sum\limits_{k = 0}^\infty {\frac{{\left( {{e^{ - \frac{\delta }{2}}}{{\left( {\frac{\delta }{2}} \right)}^j}} \right)\left( {{e^{ - \frac{\gamma }{2}}}{{\left( {\frac{\gamma }{2}} \right)}^k}} \right) \times {2^{1 + j}}{Q^j}{{({1 + 2Q} )}^{ - 3/2 - j - k}}}}{{j!k!B({1 + j,1/2 + k} )}}} } .$$

The probability density function of P can be obtained by substituting the value of Q in above equation and using the transformation of variable,

(13)$$f(P) = \frac{{dQ}}{{dP}}f(Q),$$

(14)$$f(P) = \sum\limits_{j = 0}^\infty {\sum\limits_{k = 0}^\infty {\frac{{\left( {{e^{ - \frac{\delta }{2}}}{{\left( {\frac{\delta }{2}} \right)}^j}} \right)\left( {{e^{ - \frac{\gamma }{2}}}{{\left( {\frac{\gamma }{2}} \right)}^k}} \right) \times 2{P^{1 + 2j}}{{({1 + {P^2}} )}^{ - 3/2 - j - k}}}}{{j!k!B({1 + j,1/2 + k} )}}} } .$$

Funding

U.S. Department of Education (GAANN Fellowship); Air Force Office of Scientific Research (FA9550-18-1-0338); Office of Naval Research (N000141712405, N00014-17-1-2561).

Acknowledgments

We wish to thank Hamamatsu Photonics K. K. for the C11440-42U camera. B. Javidi acknowledges support under Air Force Office of Scientific Research (FA9550-18-1-0338) and Office of Naval Research (N000141712405, N00014-17-1-2561).

Disclosures

The authors declare no conflicts of interest.

References

1. G.P. Konnen, Polarized light in nature (Cambridge University, 1985).

2. L. B. Wolff, “Polarization-based material classification from specular reflection,” IEEE Trans. Pattern Anal. Mach. Intell. 12(11), 1059–1071 (1990). [CrossRef]

3. X. Xiao, B. Javidi, G. Saavedra, M. Eismann, and M. Martinez-Corral, “Three-dimensional polarimetric computational integral imaging,” Opt. Express 20(14), 15481–15488 (2012). [CrossRef]

4. M. Born and E. Wolf, Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light (Cambridge University, 1999).

5. J. L. Pezzaniti, D. Chenault, K. Gurton, and M. Felton, “Detection of obscured targets with IR polarimetric imaging”,” Proc. SPIE 9072, 90721D (2014). [CrossRef]

6. J. S. Tyo, B. M. Ratliff, J. K. Boger, W. T. Black, D. L. Bowers, and M. P. Fetrow, “The effects of thermal equilibrium and contrast in LWIR polarimetric images,” Opt. Express 15(23), 15161–15167 (2007). [CrossRef]

7. S. Komatsu, A. Markman, A. Mahalanobis, K. Chen, and B. Javidi, “Three-dimensional integral imaging and object detection using long-wave infrared imaging,” Appl. Opt. 56(9), D120–D126 (2017). [CrossRef]

8. H. Navarro, M. Martínez-Corral, G. Saavedra, A. Pons, and B. Javidi, “Photoelastic analysis of partially occluded objects with an integral-imaging polariscope,” J. Disp. Technol. 10(4), 255–262 (2014). [CrossRef]

9. A. Carnicer and B. Javidi, “Polarimetric 3D integral imaging in photon starved conditions,” Opt. Express 23(5), 6408–6417 (2015). [CrossRef]

10. X. Shen, A. Carnicer, and B. Javidi, “Three-dimensional polarimetric integral imaging under low illumination conditions,” Opt. Lett. 44(13), 3230–3233 (2019). [CrossRef]

11. F. A. Sadjadi, “Passive three-dimensional imaging using polarimetric diversity,” Opt. Lett. 32(3), 229–231 (2007). [CrossRef]

12. G. Lippmann, “Epreuves reversibles donnant la sensation du relief,” J. Phys. 7(1), 821–825 (1908). [CrossRef]

13. N. Davies, M. McCormick, and L. Yang, “Three-dimensional imaging systems: a new development,” Appl. Opt. 27(21), 4520–4528 (1988). [CrossRef]

14. H. Arimoto and B. Javidi, “Integral Three-dimensional Imaging with digital reconstruction,” Opt. Lett. 26(3), 157–159 (2001). [CrossRef]

15. F. Okano, H. Hoshino, J. Arai, and I. Yuyama, “Real-time pickup method for a three-dimensional image based on integral photography,” Appl. Opt. 36(7), 1598–1603 (1997). [CrossRef]

16. M. Martinez-Corral, A. Dorado, J. C. Barreiro, G. Saavedra, and B. Javidi, “Recent advances in the capture and display of macroscopic and microscopic 3D scenes by integral imaging,” Proc. IEEE 105(5), 825–836 (2017). [CrossRef]

17. A. Stern and B. Javidi, “Three-dimensional image sensing and reconstruction with time-division multiplexed computational integral imaging,” Appl. Opt. 42(35), 7036–7042 (2003). [CrossRef]

18. E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” Computat. Models Vis. Process. 1, 3–20 (1991).

19. J. Liu, D. Claus, T. Xu, T. Keßner, A. Herkommer, and W. Osten, “Light field endoscopy and its parametric description,” Opt. Lett. 42(9), 1804–1807 (2017). [CrossRef]

20. G. Scrofani, J. Sola-Pikabea, A. Llavador, E. Sanchez-Ortiga, J. C. Barreiro, G. Saavedra, J. Garcia-Sucerquia, and M. Martinez-Corral, “FIMic: design for ultimate 3D-integral microscopy of in-vivo biological samples,” Biomed. Opt. Express 9(1), 335–346 (2018). [CrossRef]

21. J. Arai, E. Nakasu, T. Yamashita, H. Hiura, M. Miura, T. Nakamura, and R. Funatsu, “Progress overview of capturing method for integral 3-D imaging displays,” Proc. IEEE 105(5), 837–849 (2017). [CrossRef]

22. M. Yamaguchi, “Full-parallax holographic light-field 3-D displays and interactive 3-D touch,” Proc. IEEE 105(5), 947–959 (2017). [CrossRef]

23. J. S. Jang and B. Javidi, “Three-dimensional synthetic aperture integral imaging,” Opt. Lett. 27(13), 1144–1146 (2002). [CrossRef]

24. K. He, J. Sun, and X. Tang, “Single image haze removal using dark channel prior,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2011). [CrossRef]

25. C. Tang, Y. Wang, H. Feng, Z. Xu, Q. Li, and Y. Chen, “Low-light image enhancement with strong light weakening and bright halo suppressing,” IET Image Process. 13(3), 537–542 (2019). [CrossRef]

26. X. Dong, G. Wang, Y. Pang, W. Li, J. Wen, W. Meng, and Y. Lu, “Fast efficient algorithm for enhancement of low lighting video,” IEEE Int. Conf. Muitimed. Expo 7(10), 1–6 (2011). [CrossRef]

27. B. Tavakoli, B. Javidi, and E. Watson, “Three dimensional visualization by photon counting computational integral imaging,” Opt. Express 16(7), 4426–4436 (2008). [CrossRef]

28. L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Phys. D 60(1-4), 259–268 (1992). [CrossRef]

29. A. Stern, D. Aloni, and B. Javidi, “Experiments with Three-dimensional integral imaging under low light levels,” IEEE Photonics J. 4(4), 1188–1195 (2012). [CrossRef]

30. “ORCA fusion Hamamatsu,” https://www.hamamatsu.com/resources/pdf/sys/SCAS0138E_C14440-20UP_tec.pdf.

31. M. Martínez-Corral and B. Javidi, “Fundamentals of 3D imaging and displays: a tutorial on integral imaging, light-field, and plenoptic systems,” Adv. Opt. Photonics 10(3), 512–566 (2018). [CrossRef]

32. F. J. Massey, “The Kolmogorov-Smirnov Test for Goodness of Fit,” J. Am. Stat. Assoc. 46(253), 68–78 (1951). [CrossRef]

33. W. G. Bulgren, “On Representations of the Doubly Non-Central F Distribution,” J. Am. Stat. Assoc. 66(333), 184–186 (1971).

	Visible	LWIR
Focal length (f)	50mm	11mm
Diameter of lens (D)	40mm	9.2mm
Sensor size	682(H) × 682(V)	320(H) × 240(V)
Pixel size	19.5 × 19.5 $μ$ m	17 × 17 $μ$ m
Reconstruction depth (z)	1.2m	1.2m
Camera pitch (H,V)	30 mm, 30mm	30 mm, 30mm
Elemental images	25 (5H × 5 V)	25 (5H × 5 V)
F-number (N = f/D)	1.25	1.2

	No processing		Dark channel modeling		TV denoising		TV denoising + dark channel modeling
	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image
SNR	1.13	5.13	1.14	5.35	14.82	20.28	15.81	23.25

	No processing		Dark channel modeling		TV denoising		TV denoising + dark channel modeling
	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image
SNR	6.37	8.24	5.16	6.70	6.10	7.81	4.94	5.93

	Visible	LWIR
Focal length (f)	50mm	11mm
Diameter of lens (D)	40mm	9.2mm
Sensor size	682(H) × 682(V)	320(H) × 240(V)
Pixel size	19.5 × 19.5 $μ$ m	17 × 17 $μ$ m
Reconstruction depth (z)	1.2m	1.2m
Camera pitch (H,V)	30 mm, 30mm	30 mm, 30mm
Elemental images	25 (5H × 5 V)	25 (5H × 5 V)
F-number (N = f/D)	1.25	1.2

	No processing		Dark channel modeling		TV denoising		TV denoising + dark channel modeling
	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image	2D DoP image	3D DoP image
SNR	1.13	5.13	1.14	5.35	14.82	20.28	15.81	23.25

Three-dimensional polarimetric integral imaging in photon-starved conditions: performance comparison between visible and long wave infrared imaging

Abstract

1. Introduction

2. Theoretical background of degree of polarization (DoP)

3. Experimental methods

4. Results and discussion

4.1 DoP calculation in low light using visible and LWIR range cameras

4.2 Summary of comparison between 3D visible and 3D LWIR polarimetric imaging systems

4.3 Statistical analysis of 2D and 3D DoP images

5. Conclusion

Appendix A: theoretical statistical distribution of DoP image

Funding

Acknowledgments

Disclosures

References

Cited By

Figures (7)

Tables (3)

Equations (14)

Optics Express