1. INTRODUCTION

When incident light interacts with a surface, the surface’s properties (shape, color, refractive index, roughness, etc.) are encoded into the plenoptic function [1] of the reflected and refracted light [2]. The plenoptic function includes spatial, angular, spectral, and polarization degrees of freedom of the optical field. A set of such light reflection and refraction from the surface can be used to form an image of a scene through an imaging system, which in turn provides the basis for inferring various properties of the surface comprising the scene [3]. In conventional imaging, typically only direct photons that are emitted and/or reflected from an object that is inside the line of sight of the imaging aperture are exploited; thus, only the properties of the source (object that emits light) and/or the surface properties of the reflector (object that reflect light) are studied. For example, an image of a tree under sunlight represents the spatial distribution of the diffuse reflection coefficient over the tree itself. Meanwhile, indirect photons that are reflected off other objects before being reflected by an object inside the field of view (FoV) of the camera contain information of those hidden objects that are not perceivable in conventional imaging.

To separate indirect photons from direct ones, prior work in this area has focused on the separation of transmission and reflection on a transparent window, where both the physical approach (using polarization information) [4,5] and the computational approach (using independent component analysis) [6,7] have been demonstrated to recover useful information. Post-processing has been applied to improve image extraction by decreasing the correlation between the transmitted and reflected images [8]. Meanwhile, transparent windows induce the unique problem of ghost reflections on the captured image, and several techniques have been developed that either chose to eliminate ghost images as artifacts [9] or take advantage of them to recover better transmitted images [10]. On the other hand, opaque reflective surfaces that also reflect and scatter light (e.g., surfaces on marble tiles, wood furniture, painted walls) have received relatively little attention [5,11,12], although they are more common in real-world scenarios than transparent media. Important issues such as surface scattering, volume scattering, and non-shift-invariant point spread functions (PSFs) arise during the reflection from these surfaces, and different physical models and more sophisticated post-processing are necessary to compute the separated images.

While prior work on separation has mainly dealt with reflection from an unpolarized light source, polarized light sources are very common, especially in display devices. Due to the increasing usage of liquid crystal displays (LCDs), the screens of many electronic devices emit predominantly linearly polarized light. Moreover, some displays, such as the iPhone screens, emit elliptically polarized light and are designed to be visible through polarized sunglasses when the screen is rotated.

In this work, we demonstrate the separation of the reflection of polarized sources from transmitted light incident on a transparent window as well as the reflection of polarized sources from surface textures upon opaque reflective surfaces. First, we assume the Stokes vectors of different sources, ${\mathbf{S}}^{\mathrm{src}}$, are polarized. Second, based on Mueller calculus, we derived a physical model of the Stokes vector of light captured by the camera, ${\mathbf{S}}^{\mathrm{cam}}$, originating from multiple types of reflective media. Third, for different reflective surfaces, a closed form of ${\mathbf{S}}^{\mathrm{src}}$ was derived from ${\mathbf{S}}^{\mathrm{cam}}$ by minimizing the correlation of reflected photons and the refracted and scattered photons based on a well-defined pixel-wise metric function over the different spectral channels (e.g., red, green, and blue). We then apply Stokes imaging polarimetry to measure the components of the Stokes vector at each spatial location (pixel) in the image, which provides a full characterization of the polarization information in a scene. Our technique has several distinct advantages: (1) good separation result for different real-life scenes for a variety of display devices, (2) applicability to both grayscale and color images, (3) measurement of reflection from transparent or opaque semi-glossy objects, and (4) the ability to estimate the angle of reflection of the surface, including the orientation of the surface and the light source.

2. PRINCIPLE

A general schematic of the optical measurement process is shown in Fig. 1. A light source, such as an LCD screen, emits polarized light incident on a reflective medium. The light is reflected by the medium, passes through a polarization analyzer, and is captured by an imaging sensor. The interaction of the light with the medium yields photons in the light rays that undergo specular reflection, surface scattering, or volume scattering. Meanwhile, the medium may also transmit light in the spectrum of interest (in our case the visible spectrum), and the transmitted component can also be subsequently reflected, for example, as photons from a picture behind a glass window. Ambient light, e.g., light from an overhead lamp, generally acts as unwanted background or noise in the separation, and it is typically unpolarized and uniform in terms of spatial/angular distribution. In the discussion of the separation methods below, we refer to all photon contributions to the captured image that do not result from specular reflection as “ambient,” and we do not discern further between the various “ambient” components. A robust separation method should provide good separation results, even when the ambient component is not modeled in detail because in most cases, little is known about this component compared with the specular reflection component.

 

Fig. 1. Schematic of the scene is shown. Light emitted from a display device (image of a camera man) propagates to the reflector (gray) at an incident angle of $\varphi$, is reflected by the reflector, passes through the polarization analyzer (blue), and is detected by an imaging sensor. The imaging lens is not shown. Local coordinates of the display device in $x$, $y$ are rotated by an angle of $\chi$ into the incident local coordinates $s_{\mathrm{in}},p_{\mathrm{in}}$ on the reflector, and the exit local coordinates $s_{\mathrm{out}},p_{\mathrm{out}}$ on the reflector are rotated by an angle of $\theta$ into the local coordinates $m$, $n$ of the sensor. The horizontal direction of the polarization analyzer (in yellow) aligns with $m$.

Download Full Size | PDF

A. Stokes Vector of Display Devices

In this section, we compute the Stokes vector of different display devices. The Stokes vector is $4\times 1$ column vector, $\mathbf{S}={(S_0,S_1,S_2,S_3)}^T$, where $S_0$ is the irradiance, and the superscript $T$ denotes the transpose. Linearly polarized sources can be represented by ${(S_0,-S_0,0,0)}^T$. The source is assumed to be initially polarized along the vertical direction, as most LCDs are. Other displays, such as the iPhone screen, can be approximately described by a Stokes vector with a circular polarization component ${(S_0,s_1{S}_0,0,S_0)}^T$, where $s_1$ is the projection of ${\mathbf{S}}^{\mathrm{src}}$ along the 0°/90° direction on the Poincarè sphere. The validity of this approximation can be shown as follows.

The iPhone screen is roughly right-handed circularly polarized at a single wavelength, the designed wavelength ${\lambda }_0\approx 550\mathrm{nm}$. The screen can be modeled as a quarter-wave plate (QWP) placed on top of a linearly polarized source. The phase difference between the fast and slow axes of the QWP is ${\mathrm{OPD}}_{\mathrm{QWP}}=\mathrm{\Delta }n·d_{\mathrm{QWP}}$, where $\mathrm{\Delta }n$ is the difference of refractive indices between the two axes, and $d_{\mathrm{QWP}}$ is the physical thickness of the QWP. For a QWP operating at ${\lambda }_0$, ${\mathrm{OPD}}_{\mathrm{QWP}}=(m+1/4){\lambda }_0(m\in Z)$. From Jones calculus, the electric field of light emitted from the screen, ${\mathbf{E}}^{\mathrm{src}}$, is an integral of the Jones vector under each wavelength,

For numerical calculation of the integral, we focus on the visible spectrum (${\lambda }_1=400\mathrm{nm}$, ${\lambda }_2=700\mathrm{nm}$). The change of $n$, i.e., dispersion, is assumed to be small relative to the absolute change of $\lambda$. Direct evaluation of Eq. (1) yields the electric field instead of a Jones vector, as the latter is defined for a single wavelength. For a more intuitive result, the Jones vector being integrated is converted first into its corresponding Stokes vector and then integrated. For a general electric field $E_{x,y}$ that oscillates in the $xy$ plane, the relationship between the Stokes vector and the electric field is

For cases where the spectrum is peaked around ${\lambda }_0$, in other words, $A(\lambda )$ is largely concentrated around ${\lambda }_0$, the sinusoidal functions inside the two integrals can be approximated by their first-order terms (first and second terms in the Taylor series expansion) as $\cos (\pi {\lambda }_0/2\lambda )\sim \pi (\lambda -{\lambda }_0)/2{\lambda }_0$ and $\sin (\pi {\lambda }_0/2\lambda )\sim 1$. The cosine term’s expansion is of the order of $O(\lambda -{\lambda }_0)$, which is one order smaller than the sine term’s expansion. We therefore define $s_1={\int }_{400\mathrm{nm}}^{700\mathrm{nm}}{A}^2(\lambda )\pi (\lambda -{\lambda }_0)/2{\lambda }_0·\mathrm{d}\lambda$, and the iPhone screen can be approximated as an elliptically polarized source described by $S_0{(1,s_1,0,1)}^T$. Note that this model violates the normalization property of the Stokes vector to the second order of $\lambda -{\lambda }_0$: $\sum _{i=1,2,3}{S}_i^2/S_0^2=1+s_1^2=1+O[{(\lambda -{\lambda }_0)}^2/{\lambda }_0^2]>1$.

This approximation generally underestimates the $s_1$ component and overestimates the $s_3$ component because the sine term in $s_1$ is approximated by a linear term, and the cosine term in $s_3$ is approximated by 1. Deviation from the approximation becomes noticeable for wavelengths far from ${\lambda }_0$. The degree of circular polarization, $\mathrm{DoCP}=S_3/S_0$, is 1 if the light is right-handed circularly polarized. When the source emits only 400 nm light, the DoCP of the source reaches a minimum of 0.83. This deviation has been observed in our experiments and is discussed in Section 3.

B. Reflection, Scattering, and Refraction on Reflective Media

When light from the source encounters the reflective media, it undergoes various processes depending on the physical properties of the media. In this section, we introduce the model for ${\mathbf{S}}^{\mathrm{cam}}$, and then explain in detail all the processes and their corresponding derivations;

We consider three cases of reflecting media. The first case considers a single-surfaced opaque media, such as marble tiles. A portion of light is reflected on the surface, while the rest transmits through the surface and is subsequently volumetrically scattered or absorbed. The ratio of reflection and refraction is determined by Fresnel equations, while the portion of light that undergoes volume scattering depends on the porosity and exact chemical composition of the material and can be unknown to us. We note that volume scattering is assumed to create unpolarized light [13]. Both the reflected and a portion of the scattered light are captured by the camera. Therefore, in this case, the Mueller rotation matrix due to reflection can be expressed as [14]

In the second case, we consider coated opaque media, such as surfaces of wood furniture. The media has two interfaces—the air–coating interface and the coating–substrate interface. The physical processes on the media resemble those in the single-surface opaque media, except that the volume scattering is now replaced by surface scattering. The surface scattering often partially depolarizes the incident light and produces partially polarized light, whose properties depend on the polarized bidirectional reflectance distribution function (pBRDF) of the surface/interface, for example, at the latex–wood interface of the wood furniture. The Mueller matrix BRDF of a general depolarizer is [16]

For the third case, we consider transparent windows as reflective media [5]. The Mueller matrix is

For the above studies, we focus on discerning photons that come from specular reflection from those from refraction/scattering, while recovering some of the geometrical information of the scene, such as the angles $\chi ,\varphi ,\theta$. If the reflective media is in direct line of sight, the orientation of the media and the angle $\theta$ can be determined. If the rim of the screen is visible, the orientation of a screen and the angle $\chi$ can be calculated by line detection algorithms using the Hough transform.

C. Estimation of the Source’s Stokes Vector

To solve ${\mathbf{S}}^{\mathrm{src}}$ in terms of ${\mathbf{S}}^{\mathrm{cam}}$, we list below the three sets of equations corresponding to the three cases discussed in Section 2.B. Both ${\mathbf{S}}^{\mathrm{src}}$ and ${\mathbf{S}}^{\mathrm{amb}}$ are functions of the incident angle $\varphi$ and can be obtained separately by solving the linear equations. As the source and the ambient light component/surface texture are generally uncorrelated, the correlation between the two should ideally reach a minimum for the correct $\varphi$ corresponding to the physical setup of the scene. Note that the reflectance and transmittance are in fact functions of $\varphi$, and they determine the diattenuation of the reflection/transmission Mueller matrices. In Section 2.D, we describe the metric function used to calculate the correlation. The metric function is minimized to estimate the value of $\varphi$. In this section, we focus on deriving solvable linear equations between the captured image and the source. In general, the equations are nonlinear, but we show here that the problem can be approximated by linear equations.

For the first case of the single-surfaced opaque media,

For the second case, we consider the model for coated surfaced media with a circularly polarized source,

For transparent windows [5],

D. Metric Function

We use the edge overlap (EO) [5] as the metric function to evaluate the correlation between photons from specular reflection and from refraction/scattering. By first performing edge detection on the separated images and then computing their pixel-wise correlation, the incident angle $\varphi$ that minimizes the metric function is found. This angle is considered to be the optimal value. The mathematical formula of EO is

For each color channel, EO is computed for a set of trial $\varphi$s, and an optimal value that minimizes EO is estimated as ${\varphi }_{R,G,B}^{\mathrm{est}}$. The three RGB values ${\varphi }_{R,G,B}^{\mathrm{est}}$ are averaged as ${\varphi }^{\mathrm{est}}=\frac{1}{3}\sum _{i=R,G,B}{\varphi }_i^{\mathrm{est}}$, and the final separated images are obtained by substituting in ${\varphi }^{\mathrm{est}}$ into Eqs. (11)–(13) for all color channels.

3. RESULTS AND DISCUSSION

A. Experimental Setup

Static images were acquired using a Sony DSC-RX10M3 camera with a ZEISS Vario-Sonnar $T^{*}$ 2.4–4/8.8–220 lens. Polarizing filters were attached in front of the camera lens and rotated manually to achieve different polarization modulation of the incoming light. The linear polarizer was a HOYA 72 mm linear polarizing filter, which is fixed in front of the camera and can be manually rotated for different polarization modulations. The corresponding orientations of the polarizer are 0°, 45°, 90°, and 135° with respect to its horizontal direction shown in Fig. 1. The polarization properties of the filter were characterized beforehand with an Axometrics AxoScan polarimeter, and the Stokes images of the scene were computed with the conventional data reduction method [18,19].

The LCD screen, acting as a linearly polarized source, was a Dell AS501 monitor [Fig. 2(a)] and the screen of a Toshiba Protègè laptop [Fig. 3(a)]. The screen that acted as a circularly polarized source was an Apple iPhone 6 cell phone [Fig. 4(a)]. Screen resolutions were $1280\times 800$ for both linearly polarized sources, and $750\times 1334$ for the circularly polarized source. For all experiments, ambient light was provided by the same fluorescent light tube with the same brightness overhead. The refractive indices of marble, glass, and latex coating are assumed to be 1.65, 1.52, and 1.4, respectively.

 

Fig. 2. Reflection of the monitor screen from a marble tile is shown. (a) Overlap of the image of the screen contents and the tile is denoted by a red rectangle. (b) Separated screen contents and (c) separated tile texture are shown. (d) Reflection-removed image emphasizing the continuity of irradiance over the boundary of the overlapping region is also shown.

Download Full Size | PDF

 

Fig. 3. Reflection of the laptop screen (displaying ISO 12233 test chart) from a glass-covered picture is shown. (a) Overlap of the image of the screen contents and the picture is denoted by a red rectangle. (b) Separated screen contents and (c) separated picture are shown. (d) Reflection-removed image showing discontinuity of irradiance over the boundary of the overlapping region is also shown.

Download Full Size | PDF

 

Fig. 4. Reflection of a cell phone screen (displaying a login screen with the username and the password) from a wood sample (overlapping region shown in red rectangle) is shown. (a) Image of the screen contents is reflected from the wood surface. (b) PSF is measured with a mirror as the reflection surface. (c) PSF is measured with the wood surface as the reflection surface. (d) MAD plot is shown comparing the overlapping/separated images before and after Wiener filtering with the mirror-reflected image. (e) Overlapping unfiltered image, (f) overlapping filtered image, (g) separated unfiltered image, (h) separated filtered image, and (i) mirror-reflected image are shown. Images (e)–(i) are rotated and flipped for a better view of the screen contents. (j) Separated wood surface texture image is shown. (k) Reflection-removed image shows a discontinuity of irradiance and color over the boundary of the overlapping region.

Download Full Size | PDF

B. Separation Results

To study opaque media with a single surface, a piece of marble tile was used as the reflective medium. The monitor displayed a 1951 USAF resolution target. This target was used as a general object rather than the purpose of studying the modulation transfer function (MTF) after the separation, and as such, it was not displayed at its designed resolution. For this case, $\theta$ was $-90°$ and $\chi$ was calculated to be 134°. $\varphi$ was measured to be 63° and recovered through EO-based correlation algorithm to be 60°. From Eq. (11), the screen contents and tile texture were separated as shown in Figs. 2(b) and 2(c). The model of single reflection and volume scattering is found to work well, and there is no visible artifact in the separated images, especially in terms of the irradiance continuity over the boundary of the screen as shown in Fig. 2(d).

For the case of transparent window media, the glass plate from a photo frame was used on top of a cat photograph. The laptop screen showed an ISO 12233 test chart, which was not displayed at its designated resolution. For this case, $\theta$ was $-90°$ and $\chi$ was calculated to be 79°. $\varphi$ was measured to be 47° and was recovered through algorithm to be 46°. The screen contents and picture were separated using Eq. (12), and the results are shown in Figs. 3(b) and 3(c). Clear separation of the test chart and the cat picture are observed. The assumption of the printer paper as a perfect depolarizer does not consider the fact that reflected light from the printer paper is also weakly polarized. This leads to a visible discontinuity in irradiance over the boundary of the screen in the separated image as shown in Fig. 3(d).

For the case of coated single-surfaced media with a circularly polarized source, a wood sample coated with latex acted as the reflective medium. The cell phone screen showed a login interface with the username “U ARIZONA” and the password “OSC” as shown in Fig. 4(i). For this case, $\theta$ was 0°, and $\chi$ was calculated to be $-1°$. $\varphi$ was measured to be 54° and was estimated by algorithm to be 55°. The screen contents and wood surface texture were separated using Eq. (13), and the results are shown in Figs. 4(g) and 4(j). The separated screen image is bluish due to the approximation on the spectral response of the camera sensor. As discussed in Section 2.A, the $s_1$ component of the source’s blue channel is underestimated among the three color channels. Since $S_1=S_0·s_1$ is recovered with relatively high accuracy, the irradiance, $S_0$, of the blue channel is overestimated, resulting in the bluish tone.

The separated contents on the screen were barely recognizable. In order to improve the quality of the images, the point spread function (PSF) of the reflective medium was measured. An $8\times 8$ white square was displayed on the screen under the same geometry, and reflected images were taken using a mirror and a wood sample as the reflective media. The PSFs of the mirror and wood sample are shown in Figs. 4(b) and 4(c). The standard Wiener filter (see Appendix A) using the measured PSF are applied to both the overlapping image and the separated image, and the results are compared with the ideal image as shown in Fig. 4(i). Fidelity was measured with the mean absolute deviation (MAD), $\mathrm{MAD}=1/3\sum _{r,g,b}\sum _{i,j}|{\overline{I}}_{i,j}-{\overline{I}}_{\text{ideal};i,j}|$, where ${\overline{I}}_{i,j}$ and ${\overline{I}}_{\text{ideal};i,j}$ are the normalized to-be-compared and ideal images with each color channel normalized to (0, 1). As shown in Fig. 4(d), the lowest-MAD separated filtered image has an MAD below 0.16 at an SNR of $\sim 5$, and the separated filtered image quality as shown in Fig. 4(h) has improved significantly with better visibility after application of the Wiener filter, though the noise induced by scattering is not fully eliminated and the recovered image remains blurry. Note that the original image displayed on the screen is mainly in black and white. The color shift of the recovered images [Figs. 4(e)–4(h)] is caused by the assumption on the spectral response of the color filter in the DSLR camera’s sensor (Section 2.C) and on the assumption of the circular polarization state of the display device (Section 2.A).

4. CONCLUSION

In the last section, we discuss the various sources of error in our technique and how they can be potentially mitigated. Two major sources of inaccuracy in the image separation process are the fully depolarizing model for a general rough surface/interface, Eq. (8), and the assumed transmission spectrum of the camera’s Bayer filters. In the former case, the error can be reduced by using a more comprehensive model that incorporates (1) the pBRDF of the surface/interface, (2) an accurate dimension of the display and its distance to the reflector, which define the incoming rays of the pBRDF, and (3) the aperture size of the camera and its distance to the reflector, which define the outgoing rays of the pBRDF. In practice, information from (2) and (3) are not known a priori. Time-of-flight methods and lightfield imaging can likely provide the necessary depth information in the scene; this information can be utilized to improve the image separation process.

The Bayer filters in the camera sensor generally have wide spectral response ($>100\mathrm{nm}$ spectral width with $\sim 50\%$ quantum efficiency) and color crosstalk. Thus, the Taylor expansion discussed in Section 2.B is not a very accurate approximation. This error can be reduced by using an imaging spectro-polarimeter that has a narrow spectral response for each color channel.

Finally, the presence of ghost images can degrade the recovered image quality. Multiple reflections in the glass cover or in the latex coating induce ghost images that overlap the original image. The visibility of the ghost images is discussed in the Appendix, and our analysis shows that the ghost image can be ignored in certain cases when the thickness of the coating is less than the product of the camera’s magnification and its pixel size.

In conclusion, in this work, we have successfully demonstrated a method to recover information from display devices that are not in direct line of sight by using imaging polarimetry. Our technique can be applied to various types of displays and reflective surfaces. The recovered image can further be improved by Wiener filtering or other more sophisticated nonlinear de-convolution filtering techniques, which require a prior or post characterization of the reflective surface.

APPENDIX A: ANALYSIS OF GHOST IMAGES

In this section, we analyze the effect of ghost images due to the finite thickness of the multi-layer reflective medium. We study the propagation of a beam of light incident on a multi-layer structure shown in Fig. 5. This model applies to the reflection from a glass cover (Fig. 3) and from a latex-painted wood surface (Fig. 4). The first layer or coating can be the glass cover or latex paint, and the substrate can be air or wood. In general, the irradiance of beams that reflect more than once inside the coating (higher-order ghost reflections) is much smaller than the two reflected beams (beam 0 and 1) shown in Fig. 5. Consider the combination of a typical transparent dielectric material ($n_2\cong 1.5$) and a typical metallic material ($n_3\cong 2.5$), when a beam of unpolarized light (mixture of $s$ and $p$ light with equal amplitude) encounters the surface with an incident angle of 45°; the reflected flux ratios of beam 0, beam 1, and a three-time-reflected beam are 5.1%, 5.8%, 0.02%, respectively. On the other hand, when the substrate is air, the same ratios are 5.3%, 4.4%, 0.04%, respectively. In both cases, higher-order reflections are significantly less compared with the first two reflections.

 

Fig. 5. Schematic shows the ghost image from multiple reflections. The incident beam splits into two beams on the air–coating interface. Beam 0 reflects off the interface toward the camera. Beam 1 transmits through the interface, reflects off the coating–substrate interface, and transmits toward the camera. Only the first-order ghost reflection (beam 1) is shown. Beams 0 and 1 are separated spatially by $d$ at the air–coating interface. $\varphi$ is angle of incidence, ${\varphi }^{\prime }$ is angle of refraction, and $t$ is the coating thickness.

Download Full Size | PDF

We next consider why the shift, $d$, between the image (resulted from beam 0) and the ghost image (resulted from beam 1), which is induced by the nonzero thickness of the coating ($t$), is not always observable. We compare $d$ with the size of the pixel, $p$, of the imaging sensor. For a coating surface such as latex on wood, $t\le 200\mathrm{\mu m}$, and $d$ is thus less than $2·t·\sin ({\varphi }_C)\le 267\mathrm{\mu m}$, where ${\varphi }_C=41.8°$ is the critical angle for $n_2=1.5$. In imaging of an LCD screen, the size of the screen is approximately 100 times the size of its image on the sensor. Thus, $d\le 2.67\mathrm{\mu m}\sim 1$ pixel dimension for a typical commercial digital single-lens reflex camera. The ghost image is hardly observable in this configuration (Fig. 4). On the other hand, the thickness of a typical photo frame cover is approximately 4 mm, which results in approximately 20 pixels of displacement between the ghost image and the image (Fig. 3).

APPENDIX B: APPLICATION OF WIENER FILTERING IN THE IMAGE RECOVERY

Wiener filtering is a common method of image restoration for a spatially invariant system [20]. Our system can be approximated as spatially invariant because spatial variance of PSFs is mainly caused by (1) motion blur of two objects with different velocities, (2) spatially varying scattering, and (3) off-axis aberrations of the imaging system. In the experiment, the display device was stationary, and the off-axis aberrations of the camera were negligible, because the image quality is mainly limited by the pixel size of the sensor. We assumed the scattering property of the wood surface was uniform. Iterative methods that focus on spatially variant PSFs can be utilized to account for the non-uniformity of the surface’s scattering property and to improve the filtering process [21].

The inputs of the Wiener filter are the OTF of the reflecting medium, the OTF of the ideal system, both of which can be characterized beforehand, and the SNR of the measurement. The Wiener filter that includes the PSF of the camera, i.e., the ideal PSF, and the reflecting medium are given by

(B1)$$I_{\mathrm{filt}}=\mathrm{IFT}[\frac{{\mathrm{OTF}}_{\mathrm{surf}}^{*}}{{|{\mathrm{OTF}}_{\mathrm{surf}}|}^2+{\mathrm{SNR}}^{-1}}\times {\mathrm{OTF}}_{\text{mirror}}\times \mathrm{FT}(I_{\text{unfilt}})]$$

where $I_{\text{unfilt}/\mathrm{filt}}$ are the images before and after Wiener filtering, FT, IFT denote the Fourier transform and inverse Fourier transform, ${\mathrm{OTF}}_{\mathrm{surf}/\text{mirror}}$ are the Fourier transform of PSFs taken with the reflecting medium (wood surface) and with a flat mirror as the reflector. The SNR is defined to be the ratio of the power spectrum of the signal and the noise; its value is determined by the object and the ambient light and can be determined by estimating the specular reflected and diffuse reflection light components. Using the refractive index of latex and the estimated incident angle, SNR is estimated to be around 10. This value provides an improved image separation and is consistent with the MAD calculation using the reference image of the screen that is taken with a mirror as the reflector.

Funding

National Science Foundation (NSF) (1607358); Defense Advanced Research Projects Agency (DARPA) Revolutionary Enhancement of Visibility by Exploiting Active Light-fields (REVEAL HR0011-16-C-0026).

Acknowledgment

Y. D. thanks Dr. Bofan Song for discussions on image processing and Dr. Ori Katz for suggesting the experiment.

REFERENCES

1. M. Levoy and P. Hanrahan, “Light field rendering,” in 23rd Annual Conference on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42.

2. J. S. Tyo, D. L. Goldstein, D. B. Chenault, and J. A. Shaw, “Review of passive imaging polarimetry for remote sensing applications,” Appl. Opt. 45, 5453–5469 (2006). [CrossRef]  

3. B. M. Ratliff and J. S. Tyo, “Moving towards more intuitive display strategies for polarimetric image data,” Proc. SPIE 10407, 104070E (2017). [CrossRef]  

4. Y. Y. Schechner, J. Shamir, and N. Kiryati, “Polarization and statistical analysis of scenes containing a semireflector,” J. Opt. Soc. Am. A 17, 276–284 (2000). [CrossRef]  

5. Y. Ding, A. Ashok, and S. Pau, “Real-time robust direct and indirect photon separation with polarization imaging,” Opt. Express 25, 29432–29453 (2017). [CrossRef]  

6. H. Farid and E. H. Adelson, “Separating reflections from images by use of independent component analysis,” J. Opt. Soc. Am. A 16, 2136–2145 (1999). [CrossRef]  

7. A. M. Bronstein, M. M. Bronstein, M. Zibulevsky, and Y. Y. Zeevi, “Sparse ICA for blind separation of transmitted and reflected images,” Int. J. Imaging Syst. Technol. 15, 84–91 (2005). [CrossRef]  

8. N. Kong, Y. W. Tai, and J. S. Shin, “A physically-based approach to reflection separation: from physical modeling to constrained optimization,” IEEE Trans. Patt. Anal. Mach. Intell. 36, 209–221 (2014). [CrossRef]  

9. Y. Diamant and Y. Y. Schechner, “Overcoming visual reverberations,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2008), pp. 1–8.

10. Y. Shih, D. Krishnan, F. Durand, and W. T. Freeman, “Reflection removal using ghosting cues,” in IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2015), pp. 3193–3201.

11. T.-P. Wu and C.-K. Tang, “Separating specular, diffuse, and subsurface scattering reflectances from photometric images,” in European Conference on Computer Vision (ECCV) (Springer, 2004), pp. 419–433.

12. B. Lamond, P. Peers, A. Ghosh, and P. Debevec, “Image-based separation of diffuse and specular reflections using environmental structured illumination,” in IEEE International Conference on Computational Photography (IEEE, 2009), pp. 1–8.

13. X. D. He, K. E. Torrance, F. X. Sillion, and D. P. Greenberg, “A comprehensive physical model for light reflection,” in SIGGRAPH (ACM, 1991), pp. 175–186.

14. R. A. Chipman, “Mueller matrices,” in Handbook of Optics (McGraw-Hill, 2010), Vol. 1.

15. E. Hecht, Optics (Pearson Education, 2016).

16. S.-Y. Lu and R. A. Chipman, “Interpretation of Mueller matrices based on polar decomposition,” J. Opt. Soc. Am. A 13, 1106–1113 (1996). [CrossRef]  

17. Thorlabs, “CMOS Cameras: USB 2.0 and USB 3.0,” (2017), retrieved https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=4024.

18. W. L. Hsu, J. Davis, K. Balakrishnan, M. Ibn-Elhaj, S. Kroto, N. Brock, and S. Pau, “Polarization microscope using a near infrared full-Stokes imaging polarimeter,” Opt. Express 23, 4357–4368 (2015). [CrossRef]  

19. W. L. Hsu, G. Myhre, K. Balakrishnan, N. Brock, M. Ibn-Elhaj, and S. Pau, “Full-Stokes imaging polarimeter using an array of elliptical polarizer,” Opt. Express 22, 3063–3074 (2014). [CrossRef]  

20. M. Sonka, V. Hlavac, and R. Boyle, Image Processing, Analysis, and Machine Vision (Cengage Learning, 2014).

21. J. G. Nagy and D. P. O’Leary, “Restoring images degraded by spatially variant blur,” SIAM J. Sci. Comput. 19, 1063–1082 (1998). [CrossRef]