New strategy for high-dimensional single-pixel imaging

Xianye Li; Yongkai Yin; Wenqi He; Xiaoli Liu; Qijian Tang; Xiang Peng

doi:10.1364/OE.442520

1. Introduction

Single-pixel imaging (SPI) has become a familiar imaging modality after the development in past decade [1], which can be further classified as computational correlation imaging (CCI) [2] and compressive sensing imaging (CSI) [3] depending on their sampling strategies. In CCI, an orthogonal basis, such as Hadamard basis or Fourier basis [4], is usually employed to achieve structured illumination, and an intensity correlation operation is applied to realize computational reconstruction of the image in physical scene. In contrast, CSI usually utilizes a designed or random measurement basis for measuring. The theory of compressive sensing is used to realize image reconstruction from under-sampled detections [5]. A common feature in all these imaging modalities is that only one single-pixel detector (SPD) is employed to capture high-dimensional scenes. Due to its minimum detection requirement and equipment costs, SPI has drawn widely attentions as a potential technology in infrared imaging [6], terahertz imaging [7], optical communication [8], quantitative phase imaging [9], and three-dimensional (3D) imaging [10–12]. However, a set of projected patterns are usually required to code the targets into single-pixel detection sequence. This procedure will seriously restrict the efficiency of SPI owing to the limited modulation speed of spatial light modulator (SLM). The fastest SLM, digital micro-mirror device (DMD), has only about 22 kHz binary refresh rate, it is almost impossible for this refresh rate to achieve high-resolution real-time imaging. Different with the CCI and CSI using structured illumination, raster scan (RS) only employs one scan mirror to realize SPI, instead of modulation array, which can be considered as a particular CCI modality whose measurement basis is an identity matrix [13,14]. This arrangement can effectively reduce the equipment cost of SPI, while increasing the imaging speed. The fastest scan mirror based on the microelectromechanical system (MEMS) has already realized spot scan with FHD resolution (1080p) and 60 Hz refresh rate. Such a fast-scanning speed provides potential possibilities for SPI in real-time imaging. Besides, RS employs a laser beam to realize dot scanning, rather than a expanded structured illumination, which avoids the energy loss during the beam expanding and pattern coded procedures. Therefore, RS has higher light intensity utilization than the conventional structured illumination SPI [15].

For 3D imaging, SPI aims to address the issue of detection limitations in traditional 3D imaging systems. It is also reported that the correlation imaging is superior to traditional imaging in terms of noise immunity [15]. The current single-pixel 3D imaging solutions are most derived from existing 3D imaging schemes, which can be divided as photometric stereo [16], Time-of-flight (TOF) [17] and fringe projection [18]. Sun, et al. firstly investigated 3D imaging employing CCI together with shape from shading (SFS), where four SPDs are required to realize 3D reconstruction [10]. However, the robustness of photometric stereo is usually influenced by the characteristics of objects and the homogeneity of illuminations, the accuracy of photometric stereo is hardly to meet the requirement in high-precision measurement. TOF is another 3D imaging solutions which has been introduced in SPI [11]. To realize high measurement precision, TOF always requires a high-speed detector to continually measure the received time of reflected photons, which will further restrict the imaging speed of single-pixel 3D imaging system. Zhang, et al. proposed a compressible single-pixel Fourier imaging, they employed a periodical grating to encode the 3D scenario in single pixel detection sequence and Fourier transform profilometry (FTP) to achieve 3D imaging [12]. Ma, et al. further developed the grating coding scheme in SPI, they realized the grating coding utilizing a programable DMD and acquired 3D images using phase-shifting technique [19]. The shifted grating can provide better robustness and higher accuracy in 3D reconstruction, but also reduces the imaging speed. Therefore, considering the limitation of imaging speed for SPI, the scheme using single fringe pattern, such as FTP scheme, may be a more suitable solution for high-speed 3D SPI.

In this paper, we begin with the analysis of the limitation of different single-pixel sampling strategies. To realize high-speed imaging, we adopt RS technique for illumination, a periodic grating for light wave modulation, and single-pixel detector(s) for data detection. To establish our experiment layout for the high-dimensional SPI, a periodic binary grating is placed before the SPD to realize fringe encoding, and the FTP technique is employed to recover 3D information from single modulated fringe pattern. In traditional FTP system, the angle between the projection direction and the detection viewpoint is necessary. Therefore, the shadow area is usually existed and restricts the practical imageable area [20]. To tackle this problem, we extend the imageable area to cover the whole projection field by introducing the second SPD, which detect the modulated light from two different angles, respectively. Then, an algorithm for information fusion is proposed to merge the range images reconstructed from two SPDs. In this matching procedure, a modified total variation (TV) based criterion is proposed to remove the height errors and improve the accuracy of depth information [21]. To acquire color texture, we further introduce the third SPD such that there are three SPDs in proposed approach: two for range images acquisition and one for texture acquisition. Three bandpass filters (red, green and blue waveband) are placed in front of three SPDs, respectively, to collect the color information. Furthermore, the shading from shape technique is applied to switch viewpoints based on known 3D information [22], resulting in a colorful 3D model after the implementation of image fusion. In the whole process, only three SPDs are employed to realize wide-field, colorful, and high-resolution 3D imaging.

The contributions of this paper are presented as follows:

(1). A single-pixel 3D imaging scheme is realized based on the RS and FTP modalities, which perform in good precision and has been verified with 3D measurement.

(2). A TV based criterion is proposed to merge the height information captured from two different angle, which takes full advantage of the reciprocity of SPI and can expand the field of view to the whole illumination area.

(3). A textures fusion method is proposed inspired by the shape from shading theory, which significantly reduces the chromatic aberration caused by the detection parallax.

This work is organized as follows. Section. 2 describes the basic experimental arrangement, provides the calibration scheme and verification results for fundamental 3D imaging. Section. 3 shows the implementation of detection viewpoints extending. Section. 4 describes the strategies of acquisition and fusion for color information. Section. 5 gives the conclusions and discussions.

2. Basic scheme of single-pixel 3D imaging based on RS and FTP techniques

SPI is an emerging computational imaging scheme which encode the spatial image to time-varying detection sequence through a set of spatial modulation patterns. However, the efficiency of pattern encoding will be seriously limited by the refresh rate of SLM. The refresh rate of SLM is desired to approach or exceed million hertz to realize high-resolution dynamic SPI, even though the compressive imaging scheme is employed. It is hard to build a SLM with such high refresh rate, more than terabite pattern data will be read and displayed within one second, even so the cost may far exceed a comparable area-array camera, which obviously reduces the advantages of SPI. Optical scan imaging can be considered as a special computational correlation imaging which sampling basis is an identity matrix. The scan mirror has distinct advantages in the manufacturing costs and refresh rate. Consider the potential value for RS in the low-cost, real-time, low-light imaging, we last adopt the RS technique in our 3D imaging scheme.

For 3D information acquisition, we employ a binary grating to encode the height information and adopt FTP technique to realize 3D reconstruction. FTP is a classic technique which encode the height information into the phase distribution of single sinusoidal fringe pattern as

(1)$$\boldsymbol{I_o} (x,y)=a+\boldsymbol{R_o}(x,y) \cos [ 2 \pi f_x+ \boldsymbol{\Phi}_0 (x,y)+\boldsymbol{\Phi}_h (x,y)],$$

where $a$ is the background light, $\boldsymbol{R_o}$ is the reflectivity distribution of the target object, and $f_x$ is the frequency of projected pattern. The phase term $\boldsymbol{\Phi }_0$ is the initial phase of reference plane and $\boldsymbol{\Phi }_h$ is the phase term related to the height of the object. To extract the height information, the reference pattern should also be captured as

(2)$$\boldsymbol{I_r} (x,y)=a+\boldsymbol{R_r}(x,y) \cos [ 2 \pi f_x+ \boldsymbol{\Phi}_0 (x,y)],$$

where the $\boldsymbol{R_r}$ is the reflectivity distribution of reference plane. Applying frequency filtering after Fourier transform of $\boldsymbol{I_O}$ and $\boldsymbol{I_r}$, the terms containing phase information can be remained

(3)$$\boldsymbol{G_o} (x,y)=\boldsymbol{R_o} (x,y) \exp \{ i[2 \pi f_x+ \boldsymbol{\Phi}_0 (x,y)+\boldsymbol{\Phi}_h (x,y) ]\},$$

(4)$$\boldsymbol{G_r} (x,y)=\boldsymbol{R_r} (x,y) \exp \{ i[2 \pi f_x+ \boldsymbol{\Phi}_0 (x,y) ]\} .$$

Thus, the phase term $\boldsymbol{\Phi }_h$ can be acquired through

(5)$$\boldsymbol{\Phi}_h = \mathrm{ang}[\boldsymbol{G_o} (x,y)\circ \boldsymbol{G_r} (x,y)^*],$$

where the superscript ’$*$’ represents the complex conjugation operator and the ’$\circ$’ denotes the Hadamard product. The operation ’ang()’ returns the phase angle of complex variable in the interval $[-\pi,\pi ]$. Therefore, the phase unwrapping operation is an essential operation to unwrap the wrapped phase and get the monotonic phase distribution. Then, the height information can be acquired by converting the phase value to height according to the system model. This procedure will be discussed in the subsection of calibration.

2.1 Experimental setup of single-pixel 3D imaging

The primary experimental setup can be depicted as one channel in Fig. 1 (e. g. green channel). In the scan part, a recombination laser is employed as the dot scan light source. We use a mechanical galvanometer (SC30L, TianChuanQi) to implement raster scan operation, which can be easily controlled and triggered using data acquisition card (USB-6356, NI). In the detection part, an imaging lens ( $f$ = 12.5 mm) collects the reflected light and transfers the light through a binary grating (20 lp/mm). Behind the grating, a photodiode (C10439-09, Hamamatsu) is employed to detect the coded light. According to the Helmholtz reciprocity, the photodiode and the grating can be considered as a structured illumination system and the scanner can be taken as an area-array camera. Therefore, the proposed system can be regarded as a FTP system with binary grating modulation. In the conventional FTP scheme, defocus of binary grating is usually applied to produce quasi-sinusoidal pattern. In proposed system, the detection process can be denoted as a convolution between the binary grating and scanning laser spot. Considering the grating we used is only distributed in one dimension, we simplify this two-dimensional convolution in one dimension as shown in Fig. 2(a). This convolution process can be expressed as

(6)$$I(x) = \int_{{-}b}^{b} l(x-\tau) g(x) \,d\tau ,$$

where $l$ represents the distribution of laser point and $g$ is the grating distribution. The width of the grating or detection area is $2b$. The integration from $-b$ to $b$ is just the detection procedure with a SPD. This convolution process can realize quasi-sinusoidal coding from binary grating by controlling the image size of the laser point. Empirically, we set the ratio between the image size of the laser point and grating period to be 0.5 to 1, which can produce the quasi-sinusoidal pattern with high contrast. When this ratio is smaller than 0.5, the laser signal can not get fully coded, as shown in Fig. 2(b). Besides, when this ratio is larger than 1, as shown in Fig. 2(c), the laser signal will be coded with multiple periods of grating, and the contrast of detection fringe will be very low. In our experiment, the diameter of laser beam is about 3 mm, the distance between the imaging lens and the object is about 83 cm. The diameter of laser point imaged on the grating is about 0.0375 mm. This configuration helps to realize a high-contrast quasi-sinusoidal coding through binary grating and performs better than traditional defocus scheme when applied in proposed method.

Fig. 1. The experimental setup of of high-dimensional single-pixel imaging based on raster scan and FTP techniques. (a) Schematic diagram of proposed method. (b) Practical experimental setup.

Download Full Size | PDF

Fig. 2. Schematic diagram of quasi-sinusoidal coding. (a) The convolution procedure. (b) The situation when the ratio between the image size of the laser point and grating period is less than 0.5. (c) The situation when the ratio between the image size of the laser point and grating period is larger than 1.

Download Full Size | PDF

2.2 Calibration

The calibration is a necessary step in 3D imaging for precise measurement. In proposed method, the role of the scanner is to provide the illuminations and the position scanning, therefore, the scanning distortion will influence the reconstruction precision in lateral. The best method is to calibrate the directions of all the scanning light rays, but it may be a very complicated process because that the SPD lacks any spatial resolution, an additional area-array camera is still required to determine the positions of scanning points. According to the Helmholtz reciprocity [23], proposed system can be regarded as a monocular structured light system. Therefore, considering the distance between the two galvo mirrors is much smaller than the scanning distance, the scanner can be simplified as a pin-hole camera, as shown in Fig. 3(a), the relationship between the spatial coordinate and pixel coordinate can be expressed as

(7)$$x = \tan (\theta_x)z= \tan (k_x\Delta \theta) z,$$

(8)$$y = \tan (\theta_y)z= \tan (k_y\Delta \theta) z,$$

where $\theta _x=k_x\Delta \theta$ and $\theta _y=k_y\Delta \theta$ are the scanning angle along the $x$-axis and $y$-axis in the scanning plane, respectively. $\Delta \theta$ is the step of scanning angle which is considered as a constant and controlled by the driving voltage. $k_x$ and $k_y$ are the pixel coordinate in the virtual image plane, as shown in Fig. 3(b). As described in Eq. (7) and (8), the relationship between the spatial coordinate and pixel coordinate is not linear, it means that the lateral distortion is still existed, which is the difference between the scanner and the pin-hole camera. Generally, this distortion can be eliminated through a f-theta scanning lens, but in this work, we just consider this distortion as the radial distortion of the camera lens, and calibrate it using the standard camera calibrator toolbox in Matlab by scanning a chessboard placed on different positions and postures. In a typical calibration experiment, the mean reprojection error is about 0.9 pixels and the calibrated parameters are shown in Table 1. Similarly, the SPD can be calibrated as a projector, considering the grating is static, we adopt the curve fitting method to establish the relationship from the phase to height [24,25]. A standard plane is moved from start point ($z$ = 0 mm) with the step $\Delta z$ = 5 mm, as shown in Fig. 3(c). We measure the phase variation at 16 different positions from the start point ($z$ = 0 mm) to the end point ($z$ = 75 mm) and fit the conversion function from the phase to the height pixel-by-pixel using 3rd-order polynomial model. The fitted curve for the pixel in (300, 300) has been plotted in Fig. 3(d). So far, we have completed the calibration for the whole single-pixel 3D imaging system and established the conversion relationship for phase-$z$ -($x, y$).

Fig. 3. Calibration for the proposed system. (a) The sketch of the scanning system. (b) The pixel coordinate in the virtual image plane. (c) the schematic of the calibration procedure for the SPDs. (d) The fitted curve for the pixel in (300, 300).

Download Full Size | PDF

Table 1. Calibration parameters for the raster scan system.

View Table

2.3 Experimental verification

To verify the feasible of proposed scheme, a foam plastic ball with the radius of 50 mm (shown in Fig. 4(a)) and a standard plane (shown in Fig. 4(b)) are imaged to evaluate the accuracy of 3D reconstructions. In the experiment, the scanning resolution is set as 1000 $\times$ 1000 and the scanning speed is set as 30 kpps (pulses per second), which is the fastest speed of our galvanometer. In each scanning point, 30 intensity values are acquired through the DAQ device (900 ks/s). Two captured images are shown in Fig. 4(c) and (d). The 3D shape of foam plastic ball is derived using FTP scheme and shown in Fig. 4(e). In order to quantitatively analyze the measurement error of proposed system, we fit the point cloud data using a standard sphere function. The radius of fitted sphere is 50.1004 mm, which is only 0.1 mm away from the nominal radius. The residual error between the fitted surface and reconstructed data is plotted in Fig. 4(f), its root-mean-square error (RMSE) is 0.3803 mm. The reconstructed 3D shape of the standard plane is plotted in Fig. 4(g). We also fit its point cloud data using a standard plane function to evaluate the reconstruction error. The corresponding distribution of the residual error is shown in Fig. 4(h), and its RMSE is 0.1134mm. In these two experiments, the RMSE of residual error are both less than 0.4 mm, which demonstrates that our experimental system can achieve accurate 3D reconstruction. One phenomenon should be noticed is that the reconstruction error is larger at the edge area. The change of brightness at the edge can significantly influence the reconstruction quality, this is the drawback of single-frame FTP scheme. The robustness of FTP is much weaker than the multi-frame schemes such as phase shifting profilometry. Besides, we approximatively fit the lateral distortion using a polynomial model, instead to calibrate each ray, which is another factor for influencing the reconstruction precision at the edge area.

Fig. 4. 3D reconstruction and accuracy evaluation. (a) The foam plastic ball. (b)The standard plane. (c), (d) Captured images of the foam plastic ball and the standard plane using proposed experiment system. (e), (g) The reconstruction 3D surface from (e) and (d). (f), (h) The residual distribution between fitted shape and reconstructed data.

Download Full Size | PDF

3. Viewpoint expansion and shape fusion using two SPDs

In proposed SPI FTP system, the angle between the detection and the scanning directions will cause that the detection area usually contains some shadow areas, where the fringe information gets lost in the detection procedure. Besides, the brightness variation in the edge area is also an important factor to influence the accuracy of phase reconstruction. To solve this problem, we introduce another detection channel to realize viewpoint expansion and error reduction. In traditional FTP system, two cameras are also employed to expand the viewpoint and field of view, but the precise registration of point cloud is always necessary to fuse 3D information from two different cameras. In our system, the light modulation device is equivalent to the camera in the traditional system, which means both the fringe images from two detection angles are captured by one camera, therefore the point clouds from the two viewpoints are inherently matched. This attribute omits the procedure of point cloud registration and strengthens the reconstruction accuracy effectively.

The key point of 3D information fusion for our system is that how to distinguish the height error from different viewpoints. We notice that the height error usually becomes larger in the area where the surface normal is nearly perpendicular to the detection viewpoint. Detecting from different viewpoints can effectively reduce this error. We have imaged a plaster human face from two different viewpoints (the left (red) channel and the right (green) channel in Fig. 1) and two reconstructed 3D surfaces are shown in Fig. 5(a) and (b). Obviously, large reconstruction errors are mainly concentrated on the shadow areas, e.g., the edge areas of the nose and face. Due to the different detection directions, the errors on both viewpoints are generally complementary. Therefore, we can complement the error area using the data from the other sides. To extract the areas having large error and determine the directions (the left channel or right channel), we introduce a modified TV criterion to indicate the position where the error occurred. TV is a classic regularization model which has been widely applied for image denoising and can be perceived simply as the summation of the discrete gradient in an image. However, in this work, we ignore the summation operation in the TV and only keep the gradient variation term as

(9)$$\boldsymbol{V}(\boldsymbol{O})=\sqrt {\boldsymbol{D}_x^2 (\boldsymbol{O})+\boldsymbol{D}_y^2 (\boldsymbol{O})},$$

where $\boldsymbol{D}_x(\boldsymbol{O})$ and $\boldsymbol{D}_y(\boldsymbol{O})$ is the discrete gradient of the object $\boldsymbol{O}$ in the $x$ and $y$ direction. In most cases of FTP, it is essential that the surface of object is continuous, which is also the precondition for TV criterion. The difference of gradient variation between the left viewpoint and right viewpoint is calculated and shown in Fig. 5(c) as

(10)$$\Delta \boldsymbol{V}(\boldsymbol{O})=V(\boldsymbol{O}_l)-V(\boldsymbol{O}_r),$$

where $\boldsymbol{O}_l$ and $\boldsymbol{O}_r$ are the reconstructed 3D shape from the left and right viewpoints, respectively. As shown in Fig. 5(c), there is a clear peak where the height error exists. The positive peak values represent that the height errors are existed in the 3D shape from the left viewpoint, and the negative values indicate the right viewpoint (see the areas marked with red and blue curve in Fig. 5(a), (b) and (c)). Meanwhile, when the difference is closed to zero, it can be considered that there is no error in both viewpoints. Therefore, the 3D shape of two viewpoints can be fused as

(11)$$\boldsymbol{O}_f=\left\{\begin{matrix} \boldsymbol{O}_l & \Delta \boldsymbol{V}(\boldsymbol{O})>t \; \\ \boldsymbol{O}_r & \Delta \boldsymbol{V}(O)<{-}t \; \\ (\boldsymbol{O}_l+\boldsymbol{O}_r)/2 & \left |\Delta \boldsymbol{V}(\boldsymbol{O}) \right| \leqslant t \; \ & \; \boldsymbol{R}(\boldsymbol{O})<e \end{matrix}\right.,$$

where the $t$ is a predefined threshold value, which is determined by the density of scanning points and the shape variation of 3D object. $\boldsymbol{R}(\boldsymbol{O})$ is the height difference between two viewpoints and $e$ is also a predefined threshold value. This parameter provides additional constraint to ensure the reconstruction error in both viewpoints is acceptable. In general, calibration errors exist in both viewpoints and cause the height difference between two the two viewpoints. In our experiment, the mean height difference between the two viewpoints is about 0.631 mm. To reduce the influence of calibration error, we take the mean values of two reconstructed surface as the fused surface values when the difference of the gradient variation is smaller than the threshold value. In most cases, fusion with TV based criterion can achieve high accuracy, but sometimes the height errors may exist in the same area of two viewpoints, which will cause the wrong fused results. Considering that two viewpoints have covered the scanning area already, these errors are usually caused by the imperfections in phase unwrapping.

Fig. 5. The results for viewpoint expansion and shape fusion using two SPDs. (a) 3D surface retrieved from left (red) viewpoint. (b) 3D surface retrieved from right (green) viewpoint. (c) The difference of TV between the left and right viewpoints. (d) Fused 3D surface.

Download Full Size | PDF

4. Colorful 3D imaging using three SPDs

To explore more characteristics in the higher dimensions, we introduce the third detection channel (the blue channel shown in Fig. 1) aiming to capture the color information. In our experiment, a colorful plaster human face is imaged to verify the feasibility of proposed scheme. The 3D surface distribution of the object is shown in Fig. 6, which is reconstructed from the fringe images in red and green channels and merged using the method proposed in Section. 3. The ground truth color of the object is shown in Fig. 7(a), which is acquired from three independent measurements using the structure of the blue channel, but we replace the different bandpass filter in each measurement to get the color textures. The grating in the detection channel has been removed aims to get the high-resolution textures. The red, green and blue textures are all captured in the same direction, where the chromatic aberration caused by the detection parallax are not existed. Therefore, we set this result as the ground truth.

Fig. 6. 3D reconstruction results for the colorful plaster human face. (a) The height map. (c) The 3D surface.

Download Full Size | PDF

Fig. 7. Color information extraction and synthesis. (a) The ground truth of colorful object. (b) The directly synthesized color texture. (c) The brightness distribution captured by the red channel. (d) The switched brightness distribution of (c). (e) Retrieved color texture using three SPDs. (f) Retrieved color texture using four SPDs. (g) The 3D surface after texture mapping.

Download Full Size | PDF

To acquire the color textures from single measurement, we place three different bandpass filter in corresponding color channels. However, the grating in the red and green channels are still existed aims to acquire 3D surface. It means that the textures acquired from these two channels still include the fringe information. We employ a low-pass filter to filter out the fringe information and extract surface textures with the low-resolution from the fringe images in red and green channels. Figure 7(c) shows a low-resolution surface texture filtered from the red channel. Combining with the high-resolution texture acquired from blue channel, color information can be retrieved. The directly synthesized color image is shown in Fig. 7(b). The chromatic aberration can be easily noticed, in which more red components on the left and more green components on the right. This is because that the three-color channels are recorded from different viewpoints, the parallax between them will influence the consistency of brightness. In order to reduce the chromatic aberration, we propose a method to switch the detection viewing angle inspired by the SFS. Assuming the object is a standard Lambertian surface, whose scattered light luminance in all direction is the same [10,26]. Under this assumption, the power of the scattered light detected by the SPD depends on the angle of detection viewpoint and the surface normal, the detected intensity of an image pixel can be expressed as

(12)$$\boldsymbol{I}(x,y)=\alpha \boldsymbol{R}(x,y) [\boldsymbol{n}(x,y) \cdot \boldsymbol{d}],$$

where $\alpha$ represents the intensity of incident laser, which can be seen as a constant. $\boldsymbol{R}$ is the reflectivity distribution of of object surface, which depends on its material. $\boldsymbol{n}$ and $\boldsymbol{d}$ are the surface normal and the detection vector respectively. In the previous discussions, we have acquired two low-resolution textures from the red and green channels, therefore, the detection intensity $\boldsymbol{I}$ in Eq. (12) is known. The surface normal can be calculated from 3D shape, which has been measured and shown in Fig. 6. The detection viewpoint can be roughly regarded as the surface normal of brightest point of a homogeneous surface. Hence, one can get the approximate brightness in any detection viewpoint by solving the equations as

(13)$$\left\{\begin{matrix} \boldsymbol{I}_{i}(x,y)=\alpha \boldsymbol{R}(x,y) [\boldsymbol{n}(x,y) \cdot \boldsymbol{d}_i] \\ \boldsymbol{I}_t(x,y)=\alpha \boldsymbol{R}(x,y) [\boldsymbol{n}(x,y) \cdot \boldsymbol{d}_t] \end{matrix}\right.,$$

where the subscript $i$ represents the parameters in the initial detection viewpoint, and the subscript $t$ represents the parameters in the target viewpoint. The switched brightness distribution of Fig. 7(c) is shown in the Fig. 7(d), where the detection viewpoint is switched as the same as the blue channel. One more thing should be noted, there are some dark edges in the switched brightness images (see the areas marked in the red box of Fig. 7(d)), this is caused by the normal errors in the phase jump parts. After switching of detection viewpoint, we try to retrieve the color texture again by using a principal component analysis (PCA) based fusion method [27], and the result is shown in Fig. 7(e). Compared with the result in Fig. 7(b), The chromatic aberration gets reduced obviously, meanwhile more surface textures get presented. Figure 7(g) displays the 3D surface after texture mapping. The mapping relationships between the texture and the 3D surface captured in any viewpoint are naturally coincident, which is a great advantage for SPI. However, limited by the number of detectors, the high-resolution textures only include the information in the blue channel. Therefore, We further introduce the fourth channel with out any bandpass filter to obtain a panchromatic high-resolution texture. After the fusion of color information and gray texture, a new textures image is acquired with both high-resolution and low chromatic aberration, corresponding result is shown in Fig. 7(f). Benefit from the minimum requirement for the detector and the equipment costs, it is much easy to extend the imaging dimensions in SPI by combining more SPDs.

5. Conclusion and discussion

In this work, we discuss the different sampling strategies for SPI utilizing one to three SPDs, and realize a high-dimensional SPI scheme through a raster scanner and a grating modulated SPD. To settle the limitation of detection range and viewpoint, the second SPD is introduced to acquire 3D information from another viewpoint, and a TV based criterion is proposed to fuse the 3D shape captured from these two SPDs. For higher dimensional imaging, we further introduce the third SPD aims to acquire the color information and realize 3D colorful imaging. Inspired by the principle of SFS, A viewpoint switching method is proposed to reduce the chromatic aberration caused by the parallax between different viewpoints. A PCA based image fusion method is employed to combine the textures information in high-resolution and the color distribution in low-resolution. The experimental results with good shape quality and high color fidelity verify the feasibility of proposed method.

To summarize, a wide-field, colorful, high-resolution 3D imaging scheme is proposed only using three SPDs, which has great potential in the low-cost application. It is believed that our study provides inspiration for the application of SPI in high-dimensional imaging. Future study will be focused on the more complex reflection model, which may provide more general method of switching viewpoint, rather than being restricted to the Lambertian surface. Moreover, higher-dimensional imaging scheme, combining with the spectrum, polarization and phase, will also be explored to realize plenoptic SPI.

Funding

National Natural Science Foundation of China (61875137, 62061136005); Sino-German Cooperation Group (GZ 1391, M-0044); China Postdoctoral Science Foundation (2021M692171); Key Laboratory of Intelligent Optical Metrology and Sensing of Shenzhen (ZDSYS20200107103001793).

Disclosures

The authors declare no conflicts of interest.

Data availability

Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.

References

1. M. P. Edgar, G. M. Gibson, and M. J. Padgett, “Principles and prospects for single-pixel imaging,” Nat. Photonics 13(1), 13–20 (2019). [CrossRef]

2. J. H. Shapiro, “Computational ghost imaging,” Phys. Rev. A 78(6), 061802 (2008). [CrossRef]

3. M. F. Duarte, M. A. Davenport, D. Takhar, J. N. Laska, T. Sun, K. F. Kelly, and R. G. Baraniuk, “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag. 25(2), 83–91 (2008). [CrossRef]

4. Z. Zhang, X. Wang, G. Zheng, and J. Zhong, “Hadamard single-pixel imaging versus fourier single-pixel imaging,” Opt. Express 25(16), 19619–19639 (2017). [CrossRef]

5. S. Foucart and H. Rauhut, “An invitation to compressive sensing,” in A mathematical introduction to compressive sensing, (Springer, Berlin, 2013), pp. 1–39.

6. N. Radwell, K. J. Mitchell, G. M. Gibson, M. P. Edgar, R. Bowman, and M. J. Padgett, “Single-pixel infrared and visible microscope,” Optica 1(5), 285–289 (2014). [CrossRef]

7. W. L. Chan, K. Charan, D. Takhar, K. F. Kelly, R. G. Baraniuk, and D. M. Mittleman, “A single-pixel terahertz imaging system based on compressed sensing,” Appl. Phys. Lett. 93(12), 121105 (2008). [CrossRef]

8. F. Devaux, P.-A. Moreau, S. Denis, and E. Lantz, “Computational temporal ghost imaging,” Optica 3(7), 698–701 (2016). [CrossRef]

9. K. Ota and Y. Hayasaki, “Complex-amplitude single-pixel imaging,” Opt. Lett. 43(15), 3682–3685 (2018). [CrossRef]

10. B. Sun, M. P. Edgar, R. Bowman, L. E. Vittert, S. Welsh, A. Bowman, and M. J. Padgett, “3d computational imaging with single-pixel detectors,” Science 340(6134), 844–847 (2013). [CrossRef]

11. M.-J. Sun, M. P. Edgar, G. M. Gibson, B. Sun, N. Radwell, R. Lamb, and M. J. Padgett, “Single-pixel three-dimensional imaging with time-based depth resolution,” Nat. Commun. 7(1), 12010 (2016). [CrossRef]

12. Z. Zhang, S. Liu, J. Peng, M. Yao, G. Zheng, and J. Zhong, “Simultaneous spatial, spectral, and 3d compressive imaging via efficient fourier single-pixel measurements,” Optica 5(3), 315–319 (2018). [CrossRef]

13. M. Harwit, Hadamard transform optics (Elsevier, 2012).

14. C. M. Watts, D. Shrekenhamer, J. Montoya, G. Lipworth, J. Hunt, T. Sleasman, S. Krishna, D. R. Smith, and W. J. Padilla, “Terahertz compressive imaging with metamaterial spatial light modulators,” Nat. Photonics 8(8), 605–609 (2014). [CrossRef]

15. S. Jiang, X. Li, Z. Zhang, W. Jiang, Y. Wang, G. He, Y. Wang, and B. Sun, “Scan efficiency of structured illumination in iterative single pixel imaging,” Opt. Express 27(16), 22499–22507 (2019). [CrossRef]

16. C. Hernandez, G. Vogiatzis, and R. Cipolla, “Multiview photometric stereo,” IEEE Transactions on Pattern Analysis and Mach. Intell. 30(3), 548–554 (2008).

17. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, “Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging,” Nat. Commun. 3(1), 745 (2012). [CrossRef]

18. S. Zhang, “High-speed 3d shape measurement with structured light methods: A review,” Opt. Lasers Eng. 106, 119–131 (2018). [CrossRef]

19. Y. Ma, Y. Yin, S. Jiang, X. Li, F. Huang, and B. Sun, “Single pixel 3d imaging with phase-shifting fringe projection,” Opt. Lasers Eng. 140, 106532 (2021). [CrossRef]

20. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-d object shapes,” Appl. Opt. 22(24), 3977–3982 (1983). [CrossRef]

21. A. Chambolle, “An algorithm for total variation minimization and applications,” J. Math. Imaging Vis. 20(1/2), 163–177 (2004). [CrossRef]

22. V. S. Ramachandran, “Perception of shape from shading,” Nature 331(6152), 163–166 (1988). [CrossRef]

23. T. E. Zickler, P. N. Belhumeur, and D. J. Kriegman, “Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction,” Int. J. Comput. Vis. 49(2/3), 215–227 (2002). [CrossRef]

24. H. Liu, W.-H. Su, K. Reichard, and S. Yin, “Calibration-based phase-shifting projected fringe profilometry for accurate absolute 3d surface profile measurement,” Opt. Commun. 216(1-3), 65–80 (2003). [CrossRef]

25. Z. Cai, X. Liu, X. Peng, Y. Yin, A. Li, J. Wu, and B. Z. Gao, “Structured light field 3d imaging,” Opt. Express 24(18), 20324–20334 (2016). [CrossRef]

26. R. J. Woodham, “Photometric method for determining surface orientation from multiple images,” Opt. Eng. 19(1), 191139 (1980). [CrossRef]

27. T.-M. Tu, S.-C. Su, H.-C. Shyu, and P. S. Huang, “A new look at ihs-like image fusion methods,” Inform. Fusion 2(3), 177–186 (2001). [CrossRef]

New strategy for high-dimensional single-pixel imaging

Abstract

1. Introduction

2. Basic scheme of single-pixel 3D imaging based on RS and FTP techniques

2.1 Experimental setup of single-pixel 3D imaging

2.2 Calibration

2.3 Experimental verification

3. Viewpoint expansion and shape fusion using two SPDs

4. Colorful 3D imaging using three SPDs

5. Conclusion and discussion

Funding

Disclosures

Data availability

References

Data availability

Cited By

Figures (7)

Tables (1)

Equations (13)

Optics Express