1. Introduction

Conventional imaging only exploits the direct (line of sight) photons emitted or reflected from an object. In a typical scene, light from an illumination source is reflected from an object of interest to the camera, which measures the direct, single-bounce photons. The camera provides valuable information about the shape, color, and location of the object of interest as long as that the object is located within the field-of-view of the camera lens. Here, we consider the problem of extracting information from indirect photons, which are the photons that undergo multiple bounces and follow an indirect pathway to the camera. Examples of indirect photons include photons that bounce off an object of interest and then reflect on a wall or other object before reaching the camera. Such reflections can be specular, diffuse, or both. Accurate separation of indirect photons that come from multiple pathways, if possible, can provide valuable information of hidden objects that are outside the field of view and thus considered invisible in conventional imaging techniques.

Depending on the available prior information, separation of direct and indirect photons can be performed with little, partial, or full information of the object of interest and reflected surfaces with decreasing difficulty. When there is little to no prior information available, blind source separation methods such as the independent component analysis (ICA) [1] are typically employed for separation. ICA has been successfully applied to audio signal separation in the so-called ‘cocktail party’ problem [2–4], image separation problems such as astrophysical component separation [5], electroencephalographic data collection [6] as well as for feature extraction and noise removal. Robustness of blind source separation, such as ICA, is highly dependent on the correlation of the underlying components, and therefore, its performance is limited by the inherent noise in the different components [7, 8]. Other relevant separation approaches include user-assisted separation [9], reflection removal using ghosting cues [10], and machine learning [11] methods, which require a training image library. On the other hand, when partial prior information is available, the fidelity of separation can often be improved by exploiting a physical model of the scene [12, 13], thereby reducing the sensitivity to correlation of the underlying components [7]. Existing approaches make use of the polarization effect on the reflecting light and assume a prior knowledge of the medium of the reflection surface to calculate the reflectance and transmittance. This knowledge includes the surface being flat and uniform and statistical independence of transmitting and reflecting object [12–14]. The surface flatness and uniformity requirements are typically satisfied in scenarios like a reflection from a window or floor, while the requirement on statistical independence cannot be generally applied to all scenes, for example, the separation of objects that are similar in shape and intensity [8].

In this work, we demonstrate a robust physics-based model approach that exploits spectral and polarization information to achieve real-time, pixel-wise image separation of direct and indirect photons for indoor and outdoor scenes. Polarization information tends to be largely uncorrelated from intensity, spectral, and coherence information, and it has been commonly employed in remote sensing applications where surface features like material, roughness, and shading play a more important role [15]. Direct measurement of polarization images, which can include all or partial components of the Stokes vector, can be measured in real-time by using an imaging polarimeter [16–18]. Advantages of our technique include (1) good separation for a variety of scenes, (2) generalization to various angles and surfaces by patch-wise analysis, (3) separation of both black and white and color images, (4) applicability to reflection from transparent or opaque semi-glossy objects, and (5) ability to estimate surface reflection angle, i.e. 3D information about the reflector.

2. Methods

2.1 Transmission and reflection on a double-surfaced window

The bidirectional reflectance distribution function (BRDF) describes the reflectance of a surface for any given incoming and outgoing angle. This function can be decomposed into three components: specular reflection (SR), directional diffuse reflection (DDR), and uniform diffuse reflection (UDR) [19, 20]. Among the three components, SR can be explicitly modeled with different polarization components described by the Fresnel equations; UDR is Lambertian and therefore, unpolarized. At the outgoing angle of SR, the DDR component is equivalent to the SR component, which is attenuated solely in its intensity (Appendix) and consequently indistinguishable using polarization information. Without loss of generality, we consider the BRDF is comprised of only the SR and UDR components [Fig. 1(a)].

 

Fig. 1 (a) The BRDF (green) consisting of the SR (red) and the UDR (orange) component. (b) The scene with object T and object R. (c) Patch-wise separation of an outdoor scene. In (b), ϕ is incident angle, and θ is the polarization orientation. In c, $d_{sc}$ is the distance from the camera to the plane of incidence (the window), and $h_{cam}$ is the height of the camera.

Download Full Size | PDF

We consider a scene with a thin transparent medium and two objects: object T behind the medium and object R in front of the medium [Fig. 1(b)]. We assume that the light reflected from both objects arises from UDR and is therefore not polarized. To quantitatively characterize the polarization property of light, we employ the Stokes vector representation of an optical field. The Stokes vector of light from object T and object R is defined as: $S^{Tr}={(S_0^{Tr},0,0,0)}^T\text{and }{S}^{Re}={(S_0^{Re},0,0,0)}^T$ immediately after the reflection from the objects. We model the medium as a double-surfaced window that has two parallel flat surfaces (thickness t) with constant refractive index n. The light from object T consists of direct photons transmitted through the medium with an incident angle of ${\varphi }^{Tr}$, and the light from object R consists of indirect photons reflected once from the surface with an incident angle of ${\varphi }^{Re}$ or reflected multiple times inside the medium [Fig. 2(a)]. The incidence angle of the direct and indirect photons on a given pixel (i,j) on the detector array is ${\varphi }^{Tr}={\varphi }^{Re}\equiv {\varphi }_{ij}$, and varies across the object. The polarization effect of the medium in transmission and reflection is modeled by the corresponding Mueller matrices $T^w\left(\varphi ,n\right)$ and $R^w(\varphi ,n)$ as functions of the transmittance $T_{s,p}\equiv T_{s,p}(\varphi ,n)$ and reflectance $R_{s,p}\equiv R_{s,p}(\varphi ,n)$ of the air-medium surface:

 

Fig. 2 (a) A block diagram shows the calculation of transmitted and reflected Stokes images from data taken by imaging polarimetry. (b) The separation of light from object T and object R is performed by finding $\varphi$ that minimizes the metric function. (c) Illustration of the edge overlap and mutual information in evaluation of the correlation between images. In edge overlap, the object T (red) is a toy car, and the object R (green) is four wooden blocks with letters. Both the edges of the blocks and the letters are detected through a Canny edge detector. In mutual information, the objects are the same, and the comparison of their histograms is shown.

Download Full Size | PDF

2.2 Polarimetric imaging and separation of light

We calculate the Stokes vectors of the direct and the indirect optical beams that reach the camera $S^{cam}=(S_0^{cam},S_1^{cam},S_2^{cam},S_3^{cam})$ using Mueller calculus. In this calculation, we assume the refractive index of the surface, n, is known a priori so that $T_{s,p}$ and $R_{s,p}$ are functions of the incident angle alone. Inaccuracy in the knowledge of the refractive index value adds a perturbation to the separation results (Appendix). As the Stokes vectors of the beams leaving the medium are defined in the local coordinate on the medium, which is not necessarily aligned with the local coordinate on the camera, a general two-dimensional (2D) rotation matrix is inserted for coordinate transformation from the medium to the camera:

To back-calculate the Stokes parameters of object T and R [Fig. 2(b)], we define two diattenuation terms for window transmission and window reflection [12]: $D^{Tr}\left(\varphi \right)=(T_s^w\left(\varphi \right)-T_p^w\left(\varphi \right))/(T_s^w\left(\varphi \right)+T_p^w(\varphi ))$ and $D^{Re}\left(\varphi \right)=(R_s^w\left(\varphi \right)-R_p^w\left(\varphi \right))/(R_s^w\left(\varphi \right)+R_p^w(\varphi ))$, which are equal to the degree of linear polarization (DoLP) of the transmitting and reflecting beam. The recovered intensities of object T and object R, $S_0^{Tr,Re}(\varphi )$, are only functions of angle $\varphi$

2.3 Two metric functions of incident angle estimation

To estimate the incidence angle $\varphi$ required for image separation, we define a metric to quantify the cross correlation between the separated images $S_0^{Tr}(\varphi )$ and $S_0^{Re}(\varphi )$ [Fig. 2(b)]. In most real-world scenarios, object T and object R are unrelated to each other, so $\varphi$ can be estimated by minimizing the cross correlation between the two images. One commonly employed metric for incidence angle estimation and image separation is the mutual information (MI). While the cross correlation metric is based on a pixel-wise comparison of images that capture different sub-regions of the same object [21, 22] and has been applied to estimate image shifts and distortion, the MI metric is based on image statistics and is applied to evaluate two overlapping grayscale images [12]. The MI metric evaluates the statistical similarity (or distance) between two statistical distributions (histograms) defined on the intensities of the two objects. A second metric function, which we formulate in this work, is a variation of the cross-correlation metric and is defined based on the edge map of the two separated images instead of the intensity images. We are motivated to employ image edge information, for it is well-known in both image analysis and image understanding that edge information encodes significant symbolic information about the scene. For example, for a band limited image which is irreducible as polynomial, it is possible to fully reconstruct the image only given the edge information [23]. We define the edge overlap (EO) cross correlation metric, evaluated on each of the RGB color channels separately, to equal the number of pixels at the same location where both images have an edge present. For the edge detection itself, we apply the Canny edge detector [24–27] with an optimized threshold that minimizes the number of false edges. The mathematical definition of the two aforementioned metrics functions is as follows:

2.4 Patch-wise separation and physical calculation of the incident angle

In a scene where the size of object R is comparable to its distance to the camera, variation of incident angle across the surface of the medium needs to be considered [13]. Assuming the surface is smooth, $\varphi$ is a continuous and smooth function of the positions on the surface $x_{phys},y_{phys}$ and therefore, can be modeled as approximately constant over a small region. We thus divide the surface of the medium into multiple non-overlapping patches, each with a slightly different incident angle as shown in Fig. 1(c). For each patch, we estimate the local incident angle and perform the image separation. To evaluate the fidelity of the incident angle estimate, we calculated the incident angle for each patch based on the physical measurement of various distances and size of objects (e.g. glass window pane) comprising the scene. This patch-wise separation can be applied to analysis of both grayscale images and color images, often with increasing accuracy in the latter case. In the scenario where different parts of the object have significant spectral variations, errors can result from over/under exposure and low signal-to-noise ratio in one or more spectral channels. Such errors can be reduced or eliminated by selecting the same area in a complementary color channel or by averaging estimates from different color channels.

To calculate the incident angle on each patch using the measured distances in the scene ($d_{sc},h_{cam}$), we mapped positions $(m,n)$ in an image of the scene [Fig. 1(c)] into positions in physical space $(x_{phys},y_{phys})$ by a 2D collinear transformation, assuming the reflection surface is flat or can be approximated as piecewise flat. The physical position corresponding to a specific position in the image of the scene is $x_{phys}=\left(a_1\cdot m+b_1\cdot n\right)/\left(m+b_0\cdot n+d_0\right)$, and $y_{phys}=\left(a_2\cdot m+b_2\cdot n\right)/\left(m+b_0\cdot n+d_0\right)$, where $a_{1,2},b_{1,2}$ and $b_0,d_0$ are constants to be determined by fitting. Three positions in the physical scene are required to fit all the parameters, and in our experiment, we used six positions to improve the accuracy of fitting. Implementation of the collinear transformation followed the perspective transformation described in Reference [28]. The incident angle was physically computed from the measured distances as ${\varphi }_{phys}={\cos }^{-1}\left(d_{sc}{\left(x_{phys}^2+{(y_{phys}-h_{cam})}^2+d_{sc}^2\right)}^{-1/2}\right)$.

2.5 Analysis of signal-to-noise ratio of reflected light having both diffuse and specular components

Reflection from a flat surface, such as a mirror, obeys the law of reflection, where the angle of incidence is equal to the angle of reflection. However, for glossy surfaces, the reflected energy, $I_{ref}$, is made up of a polarized SR component, $I_{SR}$, and a non-polarized UDR component, $I_{UDR}$.

In general, $I_{SR}$ has both s and p polarized light, and the modulation depth is given by $(I_s-I_p)/2$, with a shape close to a cosine square wave as a function of the polarizer’s angle per Malus’ law. $I_{UDR}$ can be considered generally uncorrelated or weakly dependent on $I_{SR}$ and therefore can be treated here as a source of background noise in the calculation of the reflected image from $I_{ref}$. The reflected energy in diffuse reflection is assumed to be proportional to the incident energy, I, by a constant factor of α.

The s and p polarized components of $I_{SR}$ is a function of the reflectances, $R_{s,p}^w$, and I. The SNR and SR component are then given by

3. Results and discussions

3.1 Experimental setup

For indoor scenes in the visible spectrum, as shown in Fig. 3, the transmission and reflection objects were illuminated with an incandescent lamp (5500K, 900 lumens) and a pico-projector, respectively. To avoid saturation in the measurement, illumination sources were not part of the scene. The pico-projector, Model AAXA P2 Jr., was operated under low brightness mode to project a 5500 K uniform white image. For outdoor scenes, the illumination sources were the sun, as shown in Fig. 4, and the fluorescent lamps inside the building, as shown in Fig. 5.

 

Fig. 3 Separation of indoor scenes. (a) An overlapping indoor scene consisting of a toy car, four letter blocks and a piece of glass with an incident angle of $61.9°$. The separated object T and object R images are shown for the EO (b) and MI (c) metric functions. (d) An indoor scene consisting of four letter blocks and a piece of glossy printer paper with an incident angle of $53.1°$. The separated object T and object R images are shown for the EO (e) and MI (f) metric functions. Incident angle estimation of (a) and (d) using the EO (g,h) and MI (i,j) metric functions are plotted.

Download Full Size | PDF

 

Fig. 4 Patch-wise separation of trees from a window. (a) A polygon shape (green) is used to mask the window in the input image. Only the unmasked part is considered in the incident angle estimation. The metric functions for the EO method (b) and MI method (c) are plotted for different incident angles. Separation result by minimizing the EO (d, e) and MI (f, g) metric functions are shown.

Download Full Size | PDF

 

Fig. 5 A hallway scene consisting of fluorescent lights, an exit sign on the ceiling, and glossy floor. (a) The scene, with object Rs in green boxes and their reflection in yellow boxes. Separation of exit sign from glossy floor by minimizing the EO (b) and of the MI (c) metric functions, and separation of light from glossy floor by minimizing the EO (d) and of the MI (e) metric functions are shown. In addition, incident angle estimation of the exit sign with EO (f), MI (g) and of the fluorescent lights with EO (h) and of MI (i) are plotted.

Download Full Size | PDF

Static images in the visible spectrum were acquired using a Sony DSLR-A350 camera with a Sigma 17-35 mm aspherical lens. A linear polarizer (HOYA 72 mm linear polarizing filter) and a circular polarizer (for right-handed circular polarization: HOYA 72 mm circular polarizing filter; for left-handed circular polarization: B + W 72 mm HTC KSM circular polarizer) were placed in front of the camera and rotated manually to measure different polarization states. Static images in the long wave infrared spectrum were taken using a Seek Thermal CompactPRO thermal camera. A ZnSe based linear polarizer (THORLABS WP50H-Z holographic wire grid polarizer) was used in front of the camera and was rotated manually for different measurements. The glass cover in Fig. 6 was heated with an infrared light bulb (RubyLux NIR-A infrared bulb). Object R was illuminated with a StudioPRO standard 45W photo fluorescent spiral daylight light bulb.

 

Fig. 6 Comparison of the separation results for different incident angles. (a) Values of the estimated incident angles ${\varphi }_{EO,MI}$ calculated using the EO and MI methods. Both angles are compared to the measured incident angles ${\varphi }_{meas}$. (b) Overlapping images for incident angles of $48°$, $60°$ and $66°$. (c) The EO and MI metric functions are plotted as a function of the incident angle ($\varphi =60°$ scene). (d) The separated images using the EO (top) and MI (bottom) methods are shown for measured incident angles of $48°$, $60°$ and $66°$.

Download Full Size | PDF

Videos in near infrared were taken using a customized IMPERX ICL-B1620W-KC000 CCD camera with combination of a zoom lens (Computar H6Z0812 C-Mount) and a close-up lens (Vivitar 49 mm close-up lens). A 760 nm bandpass filter and a micro polarizer array of $2\times 2$ elliptical micro polarizers [29] were mounted in front of the CCD sensor to form a full-Stokes division of focal plane imaging polarimeter [30]. Resolution of the polarimeter was $1608\times 1208\times 14$ bit, and the frame rate was 12 frames per second. Video in the visible spectrum was taken by a commercial PolarCam camera from 4D Technology using the same lens system as the near infrared. This division of focal plane imaging polarimeter utilized linear wire grid micro-polarizer and had a resolution of $2400\times 1800\times 16$ bit and a frame rate of 18 frames per second. The calibration of the cameras and computation of the Stokes images are described in reference [30].

3.2 Separation results in the visible spectrum

We tested our proposed photon separation method both indoors and outside. For the indoor scene, we considered a glass from a common photo frame as the reflection (indirect)/transmission (direct) surface. Object T and object R were illuminated individually. We assume that the glass is BK7 material ($n=1.52$). While the dispersion effect between RGB channels can be included, our calculation indicates that the reflectance change is smaller than 3.5% in the visible range (400 – 700 nm). The measured incident angle is $61.9°$. Minimizing $EO\left(\varphi \right)$ yields incident angles of $62°$, $67°$ and $66°$ for RGB channels, while minimizing $MI(\varphi )$ leads to incident angle estimates of $42°$, $38°$ and $35°$ for RGB channels respectively. Residual exists in separations using both metrics, especially in separated object R image. We attribute the residual in part to the underestimation of the refractive index (Appendix).

Next, we quantified the fidelity of the separated images using reference individual images of the objects T and R that were taken separately. The individual image of object T (object R) was taken with the two illumination sources on and a black scattering cardboard inserted between object R (object T) and the glass window. We employed the median absolute deviation (MAD) metric for normalized images to quantify the fidelity of the separated images. Compared with the mean squared error (MSE), MAD is more resilient to outliers in an image and thus is more robust. The mathematical form of the MAD measure in our experiments is $MAD=\frac{1}{3}{\displaystyle \sum }_{r,g,b}{\displaystyle \sum }_{pixel}\left|S_{0;r,g,b}^{-exp}-S_{0;r,g,b}^{-ref}\right|$, where $S_{0;r,g,b}^{-exp,ref}$ denotes the intensities of the separated/reference image after linear normalization to a new dynamic range of (0,1). The MAD evaluations are summarized in Table 1 and show that separation using EO metric outperforms the MI metric.

Table 1. Comparison of the image quality using the EO and MI methods. MAD of the indoor and outdoor scenes using the EO and MI methods.

View Table

We then replaced the glass surface by a glossy printer paper for an indoor diffuse surface separation experiment. For this experiment, the scene was illuminated with a single source. This results in a slightly polarized incident light onto the coating-paper surface; nevertheless, our technique provides good separation of object R. Coated paper has an effective refractive index around 1.3 [31]. The measured incident angle for this scene is $53.1°$. Minimizing the $EO\left(\varphi \right)$ metric yields incident angle estimates of $55°$, $54°$ and $54°$ (RGB), while minimizing the $MI(\varphi )$ metric yields incident angle estimates of $52°$, $53°$ and $52°$ (RGB). Both metrics yield incident angle estimates that are close to the measured value, and yet the saturation of colors in the separated object R image from the MI method is higher than that of the real object. To evaluate the separated image fidelity, we inserted a cardboard between the color blocks and the glossy paper and took the individual image of the surface as the reference image. Under this configuration, the paper was weakly yet uniformly illuminated by the ambient light. The MAD image fidelity metric for object T shows that minimizing the $EO\left(\varphi \right)$ metric gives a result that is closer to ground truth (Table 1).

Our outdoor scene is centered on large glass window panes of a library situated next to a cluster of trees. Because the window panes are the only region of interest, the raw image is masked to eliminate other parts of the scene and thus, it is not included in the metric evaluation [Fig. 4(a)]. As the size of the trees is comparable to their-distance to the camera, we employ a patch-wise separation approach. Note that the accuracy of incident angle estimate reduces for patches that contain the masked area due to fewer sampling points. For each patch, we compute the incident angle estimate using the same approach as the indoor scene separation. We fit a second order polynomial to the incident angle estimates of different patches and obtain a smooth map of incident angles across the entire scene. As shown in Fig. 4(f), a shift in color can be observed in the separation results obtained by minimizing $MI(\varphi )$ but not in the results based on minimizing $EO\left(\varphi \right)$ [Fig. 4(d)]. Furthermore, compared with non-patch-wise separation approach, the incident angles estimated using the EO metric are closer to the actual incident angles from physical measurements; the average incident angle deviation using patch-wise separation is $5.1°$, while that of non-patch-wise separation is $8.6°$. Thus, the patch-wise separation provides a more accurate representation of the outdoor scene.

A hallway scene with glossy floor was chosen for our surface separation experiment in the visible spectrum. The floor acts both as a reflecting surface with a refractive index of 1.65 [32] and as a scattering surface. The measured incident angle for the exit sign is $52.7°$. Minimizing the $EO\left(\varphi \right)$ metric yields $52°$, $54°$, $53°$ (RGB), while minimizing the $MI(\varphi )$ metric provides $45°$, $75°$, $36°$ (RGB). Furthermore, the measured incident angle for the fluorescent lamp is $51.1°$. We obtain $52°$, $53°$, $52°$ (RGB) incident angle estimates by minimizing $EO\left(\varphi \right)$ and with $54°$, $54°$, $55°$ (RGB) by minimizing $MI(\varphi )$. Though Figs. 5(d) and 5(e) are visually identical, Fig. 5(i) shows that the curves of $MI(\varphi )$ are almost flat in the range of $40°-60°$. This flatness render the metric function value vulnerable to outliers in captured images. To evaluate fidelity of the separated images, we assumed that the specific area of the floor is black and uniform in color. Results from the incident angle estimation and MAD show that separation obtained using the EO metric is closer to the actual physical scene.

Table 1 summarizes the MAD in various scenes in the experiments. All the reference images were taken under the same conditions (ISO, shutter speed, f-number, etc.) as the overlapping images. A smaller MAD corresponds to a higher fidelity in the recovered image. In all four experiments, the EO method demonstrates a smaller MAD and is considered a more truthful image separation method.

3.3 Separation results in the long-wave infrared spectrum

We also performed an experiment in the long-wave infrared (LWIR) spectral band, where the object acts as the source of radiation. The scene consists of a glass plate in front of a paper-made ‘OSC’ pattern as the object T and an incandescent light bulb as the object R. The paper is heated by another incandescent bulb placed behind the glass plate, providing a thermal patterned background. The goal of this experiment is to separate the thermal pattern of the object R from the background thermal pattern of the object T. We adjusted the glass plate orientation to setup three different incident angles at $48°$, $60°$ and $66°$. The actual angles ${\varphi }_{meas}$ are measured from the scene and compared with the angles estimated using the EO method, ${\varphi }_{EO}$, and MI methods, ${\varphi }_{MI}$ [Fig. 6(a)]. In all three angles, the EO method provides a closer estimation of ϕ than the MI method. Incidentally, the MI method gives a close estimation in the $60°$ scene. A plot of MI versus ϕ shows that the curve is flat for a large range [Fig. 6(c)], and thus the method is sensitive to noise.

As shown in Fig. 6(d), an incorrect estimation of the angle of incident can lead to deterioration in the quality of the separated image. When the incident angle is $48°$ and the MI method yields a rather poor underestimation of ${\varphi }_{MI}=24°$, the residual is readily seen in the separated object R image. When the incident angle is $66°$ and the MI method leads to an overestimation of ${\varphi }_{MI}=80°$, the overlapping area is dark and the object R can be seen in the separated object T image. This is consistent with the analysis section for an overestimated incident angle (Appendix).

For glossy surfaces in the LWIR spectrum, the separation experiments were performed using two types of reflector surface, a glossy paper surface and a white paint surface on dry wall (BEHR Premium Plus Ultra Pure White Eggshell Zero VOC Interior Paint) in the LWIR spectrum. The surfaces were chosen for their different diffuse and specular reflection components. The object R consists of a halogen lamp behind a paper mask in the shape of the letter A, the logo for the University of Arizona, and the incident angle is fixed at $76°$.

Figure 7(a) shows the relationship between SNR and $I_{SR}/I_{ref}$ using Eqs. (10) and (11). SNR is a monotonically decreasing function of $I_{SR}/I_{ref}$. Paper coating has a larger refractive index in LWIR than white paint, leading to a higher SNR. The refractive indices for the paper coating and white paint are calculated from data in [33] and [34] for the wave number range of 700–1400 cm⁻¹ (7–14 μm), which is the nominal detectable range of the LWIR camera. For precipitated calcium carbonate (PCC) which makes up the paper coating, the refractive index is calculated by first averaging the reflectances over o-ray and e-ray assuming that the calcite particulates are randomly orientated. This assumption is also applied to the TiO₂ particulates in the white paint in which the reflectances along and perpendicular to the c-axis are averaged. The measured values for $I_{SR}/I_{ref}$ for the two materials are also plotted on the two curves. The SNR for image separation with the glossy paper (white paint) as a reflector is estimated to be 3.4 (0.56) respectively.

 

Fig. 7 Separation with diffuse surfaces. Results of image separation and SNR are shown for reflectors that have both specular and diffuse components in LWIR. (a) The SNR is plotted as a function of $I_{SR}/I_{ref}$ for glossy paper ($n=1.77$) and white paint ($n=1.50$). (b) Experimental setup for separation with diffuse surface is shown. Object R is a lamp (${\varphi }_{in}={\varphi }_{out}=76°$). The camera is positioned at the plane of incidence. Separated images from reflection of a glossy paper are calculated using EO (c) and MI (d) as the metric function. Separated images from reflection of a white paint surface are calculated using EO (e) and MI (f) as the metric function.

Download Full Size | PDF

Figures 7(c) and 7(f) show the results of the image separation. The recovered image of the object R from the reflection of glossy paper surface has a higher fidelity than the corresponding recovered image from the reflection of the white paint surface. This is consistent with the calculated SNR and, in general, the quality of the recovered image degrades with decreasing $I_{SR}/I_{ref}$. For comparison, results using both the EO method and MI method are shown. For both types of surfaces, the EO method is more robust against noise than the MI method and has less residual in the recovered object R images.

A set of indoor and outdoor videos in visible and near infrared spectral bands were also analyzed (Visualization 1, Visualization 2, Visualization 3, and Visualization 4). These videos, including the original and the separated scenes, are available in the supplementary materials. Although the analysis is not performed in real time, the results demonstrate that real time acquisition and separation of overlapping scenes is possible and that our proposed separation technique can be applied to visible and near infrared spectrum.

4. Conclusion

The accuracy of image separation is affected primarily by the polarization of the light reflected from the original objects, the accuracy of the refractive index (medium) and the incident angle estimates, and the BRDF of the transparent or semi-glossy reflector, i.e. strength of the UDR component due to diffuse surface scattering. A detailed analysis of the image separation fidelity with respect to refractive index and incident angle is described in Appendix. The assumption that the light from the two objects is unpolarized does not hold under all circumstances, for example, when the object surface is optically smooth, which gives rise to strong SR, i.e. glass and polished ceramics, or when the light source itself is polarized, i.e. computer screens. Nevertheless, our technique can be generalized to arbitrary polarization, if we can independently estimate the polarization of the light coming from the objects.

An inaccurate refractive index estimate can lead to inaccuracy in $R_{s,p}(\varphi )$ and subsequently to artifacts (residuals and shifts in color) in the separated images. However, we show that for materials commonly-used in daily life, indices range from $n=1.3$ [31] (precipitated calcium carbonate, PCC) to $n=1.79$ [35] (SF11 glass). This inaccuracy does not affect the separated object T image significantly, and it has no effect when $\varphi ={\varphi }_B$, the Brewster angle. In addition, minimization of the same metric by treating the value of the refractive index as an optimization variable may further improve the quality of the object R image. Meanwhile, inaccuracies in incident angle estimate can also lead to artifacts in the separated images. The diattenuation term $D^{Tr}\left(\varphi \right)$ is a monotonically decreasing function of the incident angle. Therefore, when recovering the object R image, either more intensity is removed if the incident angle is underestimated or less intensity is removed if the incident angle is overestimated. This conclusion applies to object T, image separation, when both the estimated incident angle and the actual incident angle are below ${\varphi }_B$, while more (less) intensity that is removed corresponds to an overestimated (underestimated) incident angle when both the estimated and the actual incident angles are above ${\varphi }_B$. Therefore, compared with the MI metric based separation method, the EO metric typically reduces the artifacts in the separated images with a more accurate estimation of $\varphi$.

In conclusion, our separation technique is robust and can be applied in real-time in both indoor and outdoor environments. Objects that are not in the field of view can be imaged and measured in visible, near infrared and long wave infrared spectrum. The ideal choice of the spectrum is determined by the properties of the reflecting surface, and the ideal wavelength range corresponds to where the SNR of light received from the reflector is the highest.

Appendix

A1 Justification of directional diffuse reflection as attenuated specular reflection

In this section, we show that for a given reflection angle ${\varphi }_r$, the polarization effect of the directional diffuse reflection (DDR) component is equal to a specular reflection (SR) component that is attenuated in the intensity, that is, the Stokes vector of DDR ($S^{DDR}$) is equal to the Stokes vector of SR ($S^{SR}$) scaled by a real scalar. We describe how the reflectance of DDR is related to the Fresnel coefficients, and how $S^{DDR}$ is related to $S^{SR}$. Our theoretical framework is identical to that in the reference [19]. Relevant equations are reproduced here for completeness. Figure 8 shows a schematic to facilitate understanding of the notations used in the derivation.

 

Fig. 8 Schematic of monochromatic incident light and its reflected light on a surface. (a) the incident electric field $p$ and its decomposition into a transverse electric (TE, ${\widehat{s}}_i$) component and a transverse magnetic (TM, ${\widehat{p}}_i$) component ($p=c_s{\widehat{s}}_i+c_p{\widehat{p}}_i$). The red ellipse shows the trajectory of $p$ of an arbitrary polarized monochromatic light as a function of time. (b) the local coordinate system on a surface $(x,y,z)$ and the parameters of the incident/reflected field (subscripted with i and r respectively). Other parameters are the wave vectors (${\widehat{k}}_{i,r}$), the incident and reflected angles (${\varphi }_{i,r}$), the solid angle of the incident light ($\text{d}{\omega }_i$), and the surface normal ($\widehat{n}$). To avoid confusion between the local coordinate system of the reflection surface and the coordinate system of the light rays, we use $(x,y,z)$ exclusively for the Cartesian coordinates on the surface, and ${\widehat{s}}_{i,r},{\widehat{p}}_{i,r}$ for the TE and TM modes of the light rays. In addition, ${\varphi }_i={\varphi }_r\equiv \varphi$ for SR. (c) the definition of a bisecting unit vector (${\widehat{n}}_b$).

Download Full Size | PDF

First, we consider the reflection from a single surface using Stokes vectors and Mueller matrices. The light incident on the surface has a general polarization state ($S={(S_0,S_1,S_2,S_3)}^T$). For an uncoated surface, the Mueller matrix for reflection, $R\left(\varphi \right)$, is a function of the surface reflectances $R_{s,p}\equiv R_{s,p}\left(\varphi \right)$, which depends on the incident angle [36]:

(10)$$R(\varphi )=\frac{1}{2}\left(\begin{array}{cccc}{R}_s+R_p& R_s-R_p& 0& 0\\ R_s-R_p& R_s+R_p& 0& 0\\ 0& 0& -2\sqrt{R_s{R}_p}& 0\\ 0& 0& 0& -2\sqrt{R_s{R}_p}\end{array}\right).$$

When $\varphi \ne {\varphi }_B$, where ${\varphi }_B$ is the Brewster angle of the surface, $R_p>0$ and the polarization effect of the surface reflection can be described by a ratio of the two reflectances $r=R_s/R_p$. In addition, as $S_0$ is generally greater than 0, the incident Stokes vector can be written as $S_0{(1,s_1,s_2,s_3)}^T$ with $s_{1,2,3}=S_{1,2,3}/S_0$. The Stokes vector of the reflected light can therefore be expressed as:

(11)$$S\text{'}=\frac{1}{2}\left[\left(R_s+R_p\right)S_0+\left(R_s-R_p\right)S_1\right]s\text{'},$$

where the polarization effect is embedded in the normalized Stokes vector $s\text{'}$, defined as:

(12)$$s\text{'}={\left(\begin{array}{cccc}1& \frac{(r-1)+(r+1)s_1}{(r+1)+(r-1)s_1}& \frac{-2\sqrt{r}{s}_2}{(r+1)+(r-1)s_1}& \frac{-2\sqrt{r}{s}_3}{(r+1)+(r-1)s_1}\end{array}\right)}^T,$$

where $T$ refers to transpose operation. Thus, the normalized Stokes vector of the reflected light $s^{\prime }$ is a function of the ratio of reflectance ratio $r$ and the normalized Stokes vector of incoming light. Two identical normalized Stokes vectors represent the same polarization state on the Poincare sphere.

In general, the reflectance has contributions from both DDR and SR components of the BRDF. In the BRDF theory, the bidirectional reflectivity (BR), denoted by ${\rho }_{s,p}$, characterizes the reflecting powers for the s and p modes respectively. The reflectance ratio $r$ is related to the bidirectional reflectivity as $r={\rho }_s/{\rho }_p$.

We consider first the BR of SR [19],

(13)$${\rho }_{s,p}^{SR}=\frac{{\left|F_{s,p}\left(n_{\lambda }\right)\right|}^2\cdot \exp \left[-g\left(\sigma ,\lambda ,{\varphi }_{i,r}\right)\right]\cdot S\left(\sigma ,\tau ,{\varphi }_{i,r}\right)}{\cos \left({\varphi }_i\right)\cdot d{\omega }_i}\cdot \Delta ,$$

where $F_{s,p}\left(n_{\lambda }\right)$ are the Fresnel reflection coefficients for the s and p modes, $g\left(\sigma ,\lambda ,{\varphi }_{i,r}\right)$ is a function of the effective surface roughness $\sigma$ and the wavelength $\lambda$, $n_{\lambda }$ is the wavelength-dependent refractive index, and $S\left(\sigma ,\tau ,{\varphi }_{i,r}\right)$ is the shadowing function related to the autocorrelation length $\tau$ of the surface. Here, $\Delta$ is 1 inside a light cone of $\text{d}{\omega }_i$, and 0 elsewhere. In Eq. (13), the dependence of polarization is only expressed through the Fresnel coefficients,

(14)$$r^{SR}={\left|\frac{F_s(n_{\lambda })}{F_p(n_{\lambda })}\right|}^2.$$

Next we consider the BR for DDR [19],

(15)$${\rho }_{s,p}^{DDR}=\frac{{\mathcal{F}}_{s,p}\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)\cdot S\left(\sigma ,\tau ,{\varphi }_{i,r}\right)}{\cos \left({\varphi }_i\right)\cdot \cos \left({\varphi }_r\right)}\cdot \frac{{\tau }^2}{16\pi }\cdot \mathcal{M}\left(\tau ,g,v_{x,y}\right),$$

where the function ${\mathcal{F}}_{s,p}\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)$ is a function of the bisecting unit vector ${\widehat{n}}_b$ [Fig. 8(c)] and the incident field $p$, $\mathcal{M}\left(\tau ,g,v_{x,y}\right)$ is an infinite summation where $v_{x,y}$ is the projection of change in wave number, $v=k(k_r-k_i)$, in the tangential plane. In Eq. (15), the dependence of polarization occurs only due to ${\mathcal{F}}_{s,p}\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)$, and thus the bidirectional reflectivity ratio for diffuse reflection is

(16)$$r^{DDR}=\frac{{\mathcal{F}}_s\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)}{{\mathcal{F}}_p\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)}$$

The mathematical forms of ${\mathcal{F}}_{s,p}\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)$ are

(17)$${\mathcal{F}}_s\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)=\delta \cdot {\left|c_s{M}_{ss}+c_p{M}_{sp}\right|}^2,$$

(18)$${\mathcal{F}}_p\left({\widehat{n}}_b,{\widehat{n}}_b,p\right)=\delta \cdot {\left|c_s{M}_{ps}+c_p{M}_{pp}\right|}^2,$$

where $\delta =\delta \left(\lambda ,{\widehat{k}}_{i,r},\widehat{z}\right)$ is related to the geometry and scale parameters of reflection and the $M$ matrices are

(19)$$M_{ss}=F_s\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right)+F_p\left({\widehat{s}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{s}}_r\cdot {\widehat{k}}_i\right),$$

(20)$$M_{sp}=-F_s\left({\widehat{s}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right)+F_p\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{s}}_r\cdot {\widehat{k}}_i\right),$$

(21)$$M_{ps}=F_s\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{s}}_r\cdot {\widehat{k}}_i\right)-F_p\left({\widehat{s}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right),$$

(22)$$M_{pp}=F_s\left({\widehat{s}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{s}}_r\cdot {\widehat{k}}_i\right)+F_p\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right).$$

For ${\varphi }_i={\varphi }_r$, the s component is always perpendicular to the unit wave vectors ${\widehat{s}}_{i,r}\cdot {\widehat{k}}_{i,r}=0$, and therefore,

(23)$$M_{ss}=F_s\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right),$$

(24)$$M_{sp}=M_{ps}=0,$$

(25)$$M_{pp}=F_p\left({\widehat{p}}_i\cdot {\widehat{k}}_r\right)\left({\widehat{p}}_r\cdot {\widehat{k}}_i\right),$$

and $r^{DDR}$ can be written as:

(26)$$r^{DDR}={\left|\frac{c_s(\lambda )}{c_p(\lambda )}\right|}^2\cdot {\left|\frac{F_s(\lambda )}{F_p(\lambda )}\right|}^2.$$

In general, $r^{DDR}$ and $r^{SR}$ are wavelength-dependent, as the Fresnel coefficients and the TE/TM decomposition factors depend on wavelengths. At a single wavelength, the incoming light traces a polarization ellipse [Fig. 8(a)] and has a defined polarization state with unity degree of polarization, $DoP=1$. For light that is not monochromatic, the polarization state of light can be described by the Stokes vector. We define a minimum wavelength range $\Delta \lambda$ centered at ${\lambda }_0$ for the incoming light’s spectrum: [${\lambda }_0-\Delta \lambda /2,{\lambda }_0+\Delta \lambda /2$] so that the incoming light can be treated as unpolarized:

(27)$$\left|s_1\right|=\frac{\left|S_1\right|}{S_0}=\frac{\left|{\displaystyle {\int }_{{\lambda }_0-\Delta \lambda /2}^{{\lambda }_0+\Delta \lambda /2}\left({\left|c_s(\lambda )\right|}^2-{\left|c_P(\lambda )\right|}^2\right)d\lambda }\right|}{{\displaystyle {\int }_{{\lambda }_0-\Delta \lambda /2}^{{\lambda }_0+\Delta \lambda /2}\left({\left|c_s(\lambda )\right|}^2+{\left|c_P(\lambda )\right|}^2\right)d\lambda }}<\epsilon ,$$

where $ϵ$ is infinitesimally small. This leads to the following inequality:

(28)$$\frac{1-\epsilon }{1+\epsilon }<\frac{{\displaystyle {\int }_{{\lambda }_0-\Delta \lambda /2}^{{\lambda }_0+\Delta \lambda /2}{\left|c_s(\lambda )\right|}^{2}d\lambda }}{{\displaystyle {\int }_{{\lambda }_0-\Delta \lambda /2}^{{\lambda }_0+\Delta \lambda /2}{\left|c_p(\lambda )\right|}^{2}d\lambda }}<\frac{1+\epsilon }{1-\epsilon },$$

$r^{DDR}$ and $r^{SR}$ are defined by their averaged values over the spectrum:

(29)$$r^{DDR}=\frac{1}{\Delta \lambda }{\displaystyle \underset{{\lambda }_0-\Delta \lambda /2}{\overset{{\lambda }_0+\Delta \lambda /2}{\int }}{\left|\frac{c_s(\lambda )}{c_p(\lambda )}\right|}^2\cdot {\left|\frac{F_s(\lambda )}{F_p(\lambda )}\right|}^2}\cdot d\lambda ,$$

(30)$$r^{SR}=\frac{1}{\Delta \lambda }{\displaystyle \underset{{\lambda }_0-\Delta \lambda /2}{\overset{{\lambda }_0+\Delta \lambda /2}{\int }}{\left|\frac{F_s(\lambda )}{F_p(\lambda )}\right|}^2}\cdot d\lambda .$$

Using inequalities (28) we get,

(31)$$r^{DDR}<\frac{1}{\Delta \lambda }\frac{1+\epsilon }{1-\epsilon }{\displaystyle \underset{{\lambda }_0-\Delta \lambda /2}{\overset{{\lambda }_0+\Delta \lambda /2}{\int }}{\left|\frac{F_s(\lambda )}{F_p(\lambda )}\right|}^2}\cdot d\lambda .=r^{SR}+O_1(\epsilon ),$$

(32)$$r^{DDR}>\frac{1}{\Delta \lambda }\frac{1-\epsilon }{1+\epsilon }{\displaystyle \underset{{\lambda }_0-\Delta \lambda /2}{\overset{{\lambda }_0+\Delta \lambda /2}{\int }}{\left|\frac{F_s(\lambda )}{F_p(\lambda )}\right|}^2}\cdot d\lambda .=r^{SR}-O_2(\epsilon ),$$

where $O_1\left(ϵ\right)=ϵ\cdot 2r^{SR}/\left[\Delta \lambda (1-ϵ)\right]$ and $O_2\left(ϵ\right)=ϵ\cdot 2r^{SR}/\left[\Delta \lambda (1+ϵ)\right]$ are infinitesimals of the same order as $ϵ$, so $r^{DDR}$ and $r^{SR}$ can be treated as equal in the wavelength range $[{\lambda }_0-\Delta \lambda /2,{\lambda }_0+\Delta \lambda /2]$. Per Eq. (12), this shows that the normalized Stokes vector of SR and DDR components are the same, and thus their Stokes parameters only differ by a real multiplicative constant.

A2 Mueller matrices of transmission and reflection on a double-surfaced window

In this section, we derive the Mueller matrices for transmission and reflection on a double-surfaced window and show their relationships with the refraction and reflection on a single surface. For an uncoated single surface, the Mueller matrix for refraction is [36]

(33)$$T(\varphi )=\frac{1}{2}\left(\begin{array}{cccc}{T}_s+T_p& T_s-T_p& 0& 0\\ T_s-T_p& T_s+T_p& 0& 0\\ 0& 0& 2\sqrt{T_s{T}_p}& 0\\ 0& 0& 0& 2\sqrt{T_s{T}_p}\end{array}\right),$$

where $T_{s,p}=1-R_{s,p}$. $R_{s,p}$ are given by the Fresnel equations, and are functions of the refractive indices on both sides of the surface ($n_{1,2}$), the angle of incidence (${\varphi }_i$) and the angle of refraction (${\varphi }_r=\sqrt{1-{\left(n_1\sin \left({\varphi }_i\right)/n_2\right)}^2}$). One property of the reflectances is that $R_{s,p}^{12}({\varphi }_i)=R_{s,p}^{21}({\varphi }_r)$. In addition, $T_{s,p}^{12}\left({\varphi }_i\right)=T_{s,p}^{21}({\varphi }_r)$ per conservation of energy. Therefore, $T^{12}\left({\varphi }_i\right)=T^{21}\left({\varphi }_r\right)\equiv T(\varphi )$ and $R^{12}\left({\varphi }_i\right)=R^{21}\left({\varphi }_r\right)\equiv R(\varphi )$, where we leave out the subscripts.

The Mueller matrix for transmission through a window is an infinite series of the reflection and refraction Mueller matrices:

(34)$$T^w(\varphi )=T(\varphi )[I+R^2(\varphi )+R^4(\varphi )+\cdots ]T(\varphi ),$$

where I is the identity matrix. It can be shown that $\left[T\left(\varphi \right),R\left(\varphi \right)\right]=0$, so that $T\left(\varphi \right)$ and $R\left(\varphi \right)$ commute and therefore, $T^w\left(\varphi \right)={\displaystyle \sum }_{n=0}^{\infty }{T}^2\left(\varphi \right)R^{2n}\left(\varphi \right)$. Multiplying this equation by $R^2\left(\varphi \right)$ on the right and then subtracting the resulting equation from this equation gives $T^w\left(\varphi \right)\left(I-R^2\left(\varphi \right)\right)=T^2\left(\varphi \right)-T^2\left(\varphi \right)R^{2n}\left(\varphi \right),n\to \infty$, where $I$ is the $4\times 4$ unit matrix.

Next, we show that $\underset{n\to \infty }{\lim }{R}^{2n}\left(\varphi \right)$ is a null matrix. The $4\times 4$ matrix R consists of two $2\times 2$ independent subspaces,

(35)$$R(\varphi )=\left(\begin{array}{cc}{R}_1(\varphi )& 0\\ 0& R_2(\varphi )\end{array}\right).$$

$R_2\left(\varphi \right)$ is a diagonal matrix with eigenvalues less than 1 and evaluates to a null matrix when raised to an infinite power. On the other hand, $R_1\left(\varphi \right)$ can be represented by the similarity transformation $R_1\left(\varphi \right)=UD{U}^{-1}$, where $U$ is a unitary matrix whose columns are the eigenvectors of $R_1\left(\varphi \right)$, and $D$ is a diagonal matrix whose elements are the corresponding eigenvalues $R_{s,p}\left(\varphi \right)$. Therefore, $R_1^{2n}\left(\varphi \right)=U{D}^{2n}{U}^{-1}$. Since $R_{s,p}\left(\varphi \right)<1$, $D^{2n}(n\to \infty )$ is a null matrix.

The matrix $I-R^2\left(\varphi \right)$ has a non-zero determinant, so the inverse matrix ${\left[I-R^2\left(\varphi \right)\right]}^{-1}$ exists. Therefore,

(36)$$T^w(\varphi )=T^2(\varphi ){\left[I-R^2(\varphi )\right]}^{-1},$$

and similarly,

(37)$$R^w(\varphi )=\left[R(\varphi )+T^2(\varphi )R(\varphi )-R^3(\varphi )\right]{\left[I-R^2(\varphi )\right]}^{-1}.$$

We define the reflectance and transmittance of a double-surfaced window as

(38)$$R_{s,p}^w(\varphi )=\frac{2R_{s,p}}{1+R_{s,p}},$$

and $T_{s,p}^w(\varphi )=1-R_{s,p}^w(\varphi )$. Equations (36) and (37) thus partially recover the forms in Eqs. (33) and (10). The Mueller matrices of transmission and reflection on a double-surfaced window are

(39)$$T^w(\varphi )=\frac{1}{2}\left(\begin{array}{cccc}{T}_s^w(\varphi )+T_p^w(\varphi )& T_s^w(\varphi )-T_p^w(\varphi )& 0& 0\\ T_s^w(\varphi )-T_p^w(\varphi )& T_s^w(\varphi )+T_p^w(\varphi )& 0& 0\\ 0& 0& T_{33}^w& 0\\ 0& 0& 0& T_{33}^w\end{array}\right),$$

(40)$$R^w(\varphi )=\frac{1}{2}\left(\begin{array}{cccc}{R}_s^w(\varphi )+R_p^w(\varphi )& R_s^w(\varphi )-R_p^w(\varphi )& 0& 0\\ R_s^w(\varphi )-R_p^w(\varphi )& R_s^w(\varphi )+R_p^w(\varphi )& 0& 0\\ 0& 0& R_{33}^w& 0\\ 0& 0& 0& R_{33}^w\end{array}\right),$$

where

(41)$$T_{33}^w=\frac{4T_s^w(\varphi )T_p^w(\varphi )}{T_s^w(\varphi )+T_p^w(\varphi )},$$

and

(42)$$R_{33}^w=\sqrt{\frac{R_s^w(\varphi )R_p^w(\varphi )}{\left(2-R_s^w(\varphi )\right)\left(2-R_p^w(\varphi )\right)}}\cdot \frac{3R_s^w(\varphi )+3R_p^w(\varphi )-2R_s^w(\varphi )R_p^w(\varphi )-4}{2-R_s^w(\varphi )-R_p^w(\varphi )}.$$

A3 Effect of an inaccurate refractive index on the separated images

In this section, we quantify the inaccuracy in recovering the object T images and object R images resulting from an inaccurate estimate of the refractive index of the reflection surface. Without loss of generality, we assume the refractive index n to be 1.52 (BK7 glass) and study the separated images for an actual refractive index that is in the range $n$ = 1.3 (PCC) to 1.79 (SF11 glass).

Per equations in the manuscript, the refractive index only enters the diattenuation term $D^{Re,Tr}\left(\varphi \right)$. This term is inversely proportional to the amount of intensity to be removed from $S_0^{cam}$ to recover the object T image or object R image. Therefore, we define a ratio of its actual value over its assumed value:

(43)$${\eta }^{Re,Tr}(\varphi )=\frac{D_{actu}^{Re,Tr}\left(\varphi \right)}{D_{assu}^{Re,Tr}\left(\varphi \right)}.$$

When ${\eta }^{Re,Tr}\left(\varphi \right)=1$, the correct amount of intensity is removed from $S_0^{cam}$ to recover the two individual images; a value less/more than 1 indicates that less/more intensity is removed and the existence of residual in the separated images.

We compare four materials with $n$ = 1.3 [31] (PCC), 1.46 [37] (fused silica), 1.60 [35] (F5 glass), and 1.79 [35] (SF11 glass) to the assumed or estimated value $n$ = 1.52 [35] (BK7) at the wavelength of 550 nm. Plots of ${\eta }^{Re,Tr}(\varphi )$ as a function of the incident angle are shown in Fig. 9.

 

Fig. 9 Effect of an inaccurate refractive index on the separated images. (a) ${\eta }^{Tr}(\varphi )$ which determines the recovered object R image is plotted as a function of the incident angle $\varphi$ for different materials. b, ${\eta }^{Re}(\varphi )$ which determines the recovered object T image is plotted as a function of the incident angle $\varphi$ for different materials.

Download Full Size | PDF

Figure 9(a) shows that underestimation (overestimation) of the refractive index causes less (more) intensity being removed when recovering the object R image. In contrast, Fig. 9(b) shows that inaccuracy in the refractive index may not affect the separated object T image if $\varphi$ is close to ${\varphi }_B$. For the studied materials, ${\varphi }_B$ are $52.4°$ (PCC), $55.6°$ (fused silica), $58.0°$ (F5 glass), and $60.8°$ (SF11 glass), which correspond to the $\varphi$ values of the four intersecting points on the ${\eta }_R=1$ line. This is because when $\varphi$ is close to ${\varphi }_B$, $R_p$ is close to 0 and $D^{Re}\left(\varphi \right)$ is close to 1, so the amount of intensity to be removed is independent of the refractive index.

In conclusion, the separated object R image is more affected by an inaccurate refractive index. This effect can be quantitatively evaluated with the relative error in $D^{Re,Tr}\left(\varphi \right)$ ($\left|D_{actu}^{Re,Tr}\left(\varphi \right)-D_{assu}^{Re,Tr}\left(\varphi \right)\right|/D_{assu}^{Re,Tr}\left(\varphi \right)=|{\eta }^{Re,Tr}\left(\varphi \right)-1|$). In the materials that we analyze, $|{\eta }^{Tr}\left(\varphi \right)-1|$ reaches 60% and has a minimum of 40% [Fig. 9(a)], whereas $|{\eta }^{Re}\left(\varphi \right)-1|$ varies from 0 to 30% [Fig. 9(b)].

A4 Effect of inaccurate incident angle estimation on the separated images

In this section, we quantify the effect of inaccurate incident angle estimation on the separated images. Following the last section, we focus on $D^{Re,Tr}\left(\varphi \right)$, the diattenuation terms, and assume that the refractive index is accurate. For the studied materials of various refractive indices, $D^{Re,Tr}\left(\varphi \right)$ are calculated as a function of the incident angle and the values are summarized in Fig. 10.

 

Fig. 10 Effect of inaccurate incident angle estimation on the separated images. (a) the absolute value of $D^{Tr}(\varphi )$, which determines the recovered object R image in (39) is plotted as a function of the incident angle for different materials. (b), $D^{Re}(\varphi )$, which determines the recovered object T image in (39) is plotted as a function of the incident angle for different materials. In addition, a detailed subplot is shown in linear scale. The incident angle where a curve reaches its minimum corresponds to the Brewster angle.

Download Full Size | PDF

Per Fig. 10(a), an underestimation (overestimation) of $\varphi$ results in an overestimation (underestimation) of $D^{Tr}(\varphi )$; therefore, less (more) intensity is removed in recovering the object R image. In contrast, $D^{Re}(\varphi )$ is a decreasing function of the incident angle when $\varphi <{\varphi }_B$, and an increasing function when $\varphi >{\varphi }_B$. One general conclusion is when both the actual and the estimated incident angles are below the Brewster angle, an underestimation (overestimation) of $\varphi$ results in an overestimation (underestimation) of $D^{Re}(\varphi )$, yetwhen both angles are above the Brewster angle, it results in an underestimation (overestimation) of $D^{Re}(\varphi )$. However, when the actual angle is not known or the two angles are not on the same side of the Brewster angle, no general conclusion can be drawn.

A5 Experimental BRDF measurement of rough surfaces

The ratio, $I_{SR}/I_{ref}$, can be determined by measuring the bidirectional reflectance distribution function (BRDF). The BRDF depends on four angles, the azimuthal and polar angles of the source and the detector, with respect to the reflecting surface. Since the reflector materials are isotropic, only the difference in azimuthal angle is considered. In our experiment [Fig. 11(a)], a heat source (a halogen lamp) is placed 5 meters from the reflector surface and can be approximated as a point source. The reflected light is measured by a LWIR camera as a function of the elevation angle of the detector. Figures 11(b) and 11(c) shows the reflected LWIR signals for different angles. A peak signal is located at ${\varphi }_{out}=76°$, which is consistent with the law of reflection. Polynomial fitting is applied to the measured data for points excluding the peak on both sides of the peak to extract the $I_{UDR}$ component. The specular component, $I_{SR}$, is calculated by subtracting the measured data and the polynomial fitting of $I_{UDR}$. $I_{SR}$ is found to peak at around ${\varphi }_{out}=76°$ and is generally negligible at other angles. $I_{SR}$ is fitted with a Gaussian function. All the components are plotted as a function of angle. Our measurement shows that the glossy paper has lower diffuse reflection and higher specular reflection compared with the white paint. The values for $I_{SR}/I_{ref}$ can be calculated from the ratio of the LWIR signal at the peak and the corresponding diffuse reflection. $I_{SR}/I_{ref}$ is 0.83 for the glossy paper and 0.51 for the white paint respectively.

 

Fig. 11 BRDF measurement for diffuse surface separation. (a) Experimental setup for BRDF measurement is shown. Reflected light is measured at long-wave infrared. The light source is fixed at $76°$, corresponding to the experimental configuration in the manuscript. The camera is positioned at the plane of incidence, and the angle of the camera, ${\varphi }_{out}$, is varied. The reflected light as a function of angle is plotted for (b) glossy paper surface and (c) white paint surface. The diffuse and specular components of the reflected light, along with the sum of the two components, are also shown.

Download Full Size | PDF

Funding

National Science Foundation (NSF 1607358); DARPA Revolutionary Enhancement of Visibility by Exploiting Active Light-fields (REVEAL HR0011-16-C-0026).

Acknowledgment

We thank Prof. Rongguang Liang for loan of the PolarCam.

References and links

1. A. Hyvärinen and E. Oja, “Independent component analysis: algorithms and applications,” Neural Netw. 13(4-5), 411–430 (2000). [PubMed]  

2. E. C. Cherry, “Some experiments on the recognition of speech, with one and with two ears,” J. Acoust. Soc. Am. 25, 975–979 (1953).

3. S. Haykin and Z. Chen, “The cocktail party problem,” Neural Comput. 17(9), 1875–1902 (2005). [PubMed]  

4. A. J. Simpson, G. Roma, and M. D. Plumbley, “Deep karaoke: Extracting vocals from musical mixtures using a convolutional deep neural network,” in International Conference on Latent Variable Analysis and Signal Separation, (Springer, 2015), pp. 429–436.

5. D. Maino, A. Farusi, C. Baccigalupi, F. Perrotta, A. J. Banday, L. Bedini, C. Burigana, G. De Zotti, K. M. Górski, and E. Salerno, “All-sky astrophysical component separation with Fast Independent Component Analysis (FASTICA),” Mon. Not. R. Astron. Soc. 334, 53–68 (2002).

6. S. Makeig, A. J. Bell, T.-P. Jung, and T. J. Sejnowski, “Independent component analysis of electroencephalographic data,” Adv. Neural Inf. Process. Syst. 8, 145–151 (1996).

7. Y. Y. Schechner, “Inversion by P4: polarization-picture post-processing,” Philos. Trans. R. Soc. Lond. B Biol. Sci. 366(1565), 638–648 (2011). [PubMed]  

8. H. Farid and E. H. Adelson, “Separating reflections from images by use of independent component analysis,” J. Opt. Soc. Am. A 16(9), 2136–2145 (1999). [PubMed]  

9. A. Levin and Y. Weiss, “User assisted separation of reflections from a single image using a sparsity prior,” IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1647–1654 (2007). [PubMed]  

10. Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition, (IEEE, 2015), pp. 3193–3201.

11. M. Zibulevsky and B. A. Pearlmutter, “Blind source separation by sparse decomposition in a signal dictionary,” Neural Comput. 13(4), 863–882 (2001). [PubMed]  

12. Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization and statistical analysis of scenes containing a semireflector,” J. Opt. Soc. Am. A 17(2), 276–284 (2000). [PubMed]  

13. N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 209–221 (2014). [PubMed]  

14. K. Xu, S. Liu, W. Sun, Z. Ma, Z. Li, Q. Yu, and G. Li, “Design and fabrication of a monolithic optoelectronic integrated Si CMOS LED based on hot-carrier effect,” IEEE J. Sel. Top. Quantum Electron. 22, 70–77 (2016).

15. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45(22), 5453–5469 (2006). [PubMed]  

16. V. Gruev, R. Perkins, and T. York, “CCD polarization imaging sensor with aluminum nanowire optical filters,” Opt. Express 18(18), 19087–19094 (2010). [PubMed]  

17. G. Myhre, W.-L. Hsu, A. Peinado, C. LaCasse, N. Brock, R. A. Chipman, and S. Pau, “Liquid crystal polymer full-stokes division of focal plane polarimeter,” Opt. Express 20(25), 27393–27409 (2012). [PubMed]  

18. W. L. Hsu, G. Myhre, K. Balakrishnan, N. Brock, M. Ibn-Elhaj, and S. Pau, “Full-Stokes imaging polarimeter using an array of elliptical polarizer,” Opt. Express 22(3), 3063–3074 (2014). [PubMed]  

19. X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH, (ACM Press, 1991), pp. 175–186.

20. K. Xu, “Current-voltage characteristics and increase in the quantum efficiency of three-terminal gate and avalanche-based silicon LEDs,” Appl. Opt. 52(27), 6669–6675 (2013). [PubMed]  

21. W. K. Pratt, “Correlation techniques of image registration,” IEEE Trans. Aerosp. Electron. Syst. 10, 353–358 (1974).

22. A. J. Fitch, A. Kadyrov, W. J. Christmas, and J. Kittler, “Fast robust correlation,” IEEE Trans. Image Process. 14(8), 1063–1073 (2005). [PubMed]  

23. S. R. Curtis and A. V. Oppenheim, “Reconstruction of multidimensional signals from zero crossings,” J. Opt. Soc. Am. A 4, 221–231 (1987).

24. J. Canny, “A computational Approach to Edge Detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986). [PubMed]  

25. M. Jacob and M. Unser, “Design of steerable filters for feature detection using canny-like criteria,” IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 1007–1019 (2004). [PubMed]  

26. P. Bao, L. Zhang, and X. Wu, “Canny edge detection enhancement by scale multiplication,” IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1485–1490 (2005). [PubMed]  

27. W. Rong, Z. Li, W. Zhang, and L. Sun, “An improved CANNY edge detection algorithm,” in IEEE International Conference on Mechatronics and Automation (IEEE, 2014), pp. 577–582.

28. G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library (O'Reilly Media, 2008).

29. W. L. Hsu, K. Balakrishnan, M. Ibn-Elhaj, and S. Pau, “Infrared liquid crystal polymer micropolarizer,” Appl. Opt. 53(23), 5252–5258 (2014). [PubMed]  

30. W. L. Hsu, J. Davis, K. Balakrishnan, M. Ibn-Elhaj, S. Kroto, N. Brock, and S. Pau, “Polarization microscope using a near infrared full-Stokes imaging polarimeter,” Opt. Express 23(4), 4357–4368 (2015). [PubMed]  

31. E. J. Elton and J. S. Preston, “Effective refractive index and the porosity of coated paper,” (Surfoptic, 2007).

32. G. Ghosh, “Dispersion-equation coefficients for the refractive index and birefringence of calcite and quartz crystals,” Opt. Commun. 163, 95–102 (1999).

33. L. L. Long, M. R. Querry, R. J. Bell, and R. W. Alexander, “Optical properties of calcite and gypsum in crystalline and powdered form in the infrared and far-infrared,” Infrared Phys. 34, 191–201 (1993).

34. R. Gonzalez, R. Zallen, and H. Berger, “Infrared reflectivity and lattice fundamentals in anatase TiO2s,” Phys. Rev. B 55, 7014 (1997).

35. SCHOTT, “Interactive Abbe-Diagram” (2016), retrieved http://www.us.schott.com/advanced_optics/english/knowledge-center/technical-articles-and-tools/abbe-diagramm.html?us-origin=en-US#.

36. R. A. Chipman, Mueller Matrices, in Handbook of Optics, M. Bass, ed. (McGraw-Hill Education, 2010).

37. I. H. Malitson, “Interspecimen comparison of the refractive index of fused silica,” J. Opt. Soc. Am. 55, 1205–1209 (1965).