Optical 3-D surface reconstruction with color binary speckle pattern encoding

Pei Zhou; Jiangping Zhu; Hailong Jing

doi:10.1364/OE.26.003452

1. Introduction

Structured light projection based optical 3-D measurements [1] have been extensively applied in various domains, typically involving inspection and control of industry quality, 3-D digitalization of cultural relics and historic sites, as well as entertainment industry, due to its characters like full field and on-contact, high accuracy and reliability. Monocular and binocular stereo vision configurations are the commonly used geometric architectures. The former is composed of a camera and a projector, which is more cost-effective than the latter that consists of two cameras and a projector. From the standpoint of practicality, the binocular stereo vision is more convenient in system calibration and is more widely adopted in productization, although it is more vulnerable to occlusions.

In order to expand optical 3-D measurement techniques to a wider range of applications, researchers have carried out extensive researches for improvements of all aspects, including the encoding and decoding of projected structured patterns [2], system calibration [3–6], simplification and extension of measurement principle [7,8], modification of some special application scenarios (dynamic [9] and static objects, objects of partial specular reflection [9–11]), 3-D reconstruction schemes [12], etc.

As for the binocular stereo vision, accurately solving correspondences between two views play a key role in restoring 3-D geometry of tested objects, for this purpose various encoding schemes have been put forward [2]. However, different encoding and decoding strategies specially used differs a lot in measurement performance and application ranges. In general, multiple patterns projection based encoding and decoding schemes are suitable for high accuracy measurement of static objects unless high-performance hardware and optimum 3-D reconstruction frameworks are involved [9, 13] that the movement of object under test can be ignored. Instead, dynamic objects that pay more attention to fast acquisition of 3-D shape instead of accuracy, single-shot pattern projection based approaches [1-2, 14–16] that only a stereo image pair or an image is required to reconstruct a whole 3-D point cloud are preferred. Among them the metrology based on speckle pattern [16–20] is the important one due to its abundant high-frequency features conductive to establish the correspondence between a stereo image pair [7]. Ref [17]. used coherent light through ground glass to generate random speckle patterns for both 3-D mapping of objects and range finding. Ref [18]. adopted a similar approach (laser-illuminated rough surfaces) to encode the tested object and further by cross-correlation of corresponding subimages to search the correspondences. Chen et al [19] proposed using sum of square difference (SSD) based 3D digital image correlation (3D-DIC) for vibration measurement. More detailed and in-depth introduction including basic concepts, theory and applications concerning image correlation for shape measurement or other applications can be found in [20]. Besides, a comprehensive overview of most widely used correlation criteria, i.e., zero-mean normalized cross-correlation (ZNCC) criterion, zero-mean normalized sum of squared difference (ZNSSD) criterion, and parametric sum of squared difference (PSSD_𝑎𝑏) criterion as well as their relations used in digital speckle correlation (DSC) is provided in [21].

The conventional speckle pattern-based 3-D solution is theoretically very allowable for dynamic 3-D measurement since only one image pair is required, but it provides a limited spatial resolution and accuracy although its possibility of using an optimum matching window [22]. The main reason lies in the fact that a single-shot pattern does not give sufficient information to enable the accurate correspondence between two views. Generally, to increase the number of matched points a large subset (matching window) must be used, but meanwhile it can lead to the reduction of spatial resolution and the increment of computation amount as well as mismatching rate. In order to overcome these deficiencies, Wiegmann et al [23] carefully design band-limited statistical patterns (a variant of the conventional speckle) to encode a static human face, and then apply a temporal correlation algorithm with matching window size 1pixel × 1pixel along the temporally sequential stereo images pairs to implement the stereo matching, by this manner which substantially improve the spatial resolution. Meanwhile, Schaffer et al [24,25] put this idea into 3-D measurement of moving scenes. Laser speckle patterns generated by high-speed projection devices continually scan the tested moving object and two synchronized high-speed cameras capture images of the encoded object. In their experiments, an assumption that the moving object is nearly static at capturing the image sequences within ultrashort time interval is made. But in some cases, the assumption is usually not always true. Let’s take the example of fast moving objects. Usually, compensation strategies like iterative reconstruction, parameters-optimized adaptive correlation algorithm and careful adjustment and match between the capture frame rate of cameras and the motion of object are necessary to remedy motion-induced artifacts [26].

Some commercial products like PrimeSense^®’s depth sensor originally focused on games and living room markets also adopts single-shot random pattern with ~30, 000 binary dots, as known “light coding” technology [27], a speckle pattern generation approach based on diffraction optics element (DOE), to code the moving scene. This sensor is later integrated into Microsoft^® Kinect [28] and iPhone X 3D face recognition technique [29]. Intel^® RealSense 3D Camera [30] is also based on the similar scheme.

In order to carry out a trouble–free and reliable 3-D measurement task by commonly available devices without additional compensations and fussy parameters adjustment, in this paper, we devote ourselves to presenting a novel single-shot color binary speckle pattern (CBSP) based full-resolution 3-D measurement scheme with improved spatial resolution and measurement accuracy relative to the traditional BWBSP. The main contributions of this research are focused on: (1) on the basis of established binocular stereo measurement system, we describe in detail the encoding strategy of CBSP by mathematically and physically related with the systematic configuration parameters to be adopted; (2) the temporal–spatial correlation framework is extended to determine the matched points that permits 3-D reconstruction by only a stereo image pair; (3) the experiments quantitatively and qualitatively verify that the proposed approach can provide much better performances than BWBSP in terms of spatial resolution, measurement accuracy and number of reconstructed 3-D points. And we also show its feasibility in measuring a moving object; (4) the advantages and disadvantages are objectively analyzed and discussed.

2. Principle

In this section, we will introduce the related theoretical background as well as our proposed computational framework to perform CBSP-based 3-D measurement method. We will first introduce how the CBSP is designed for theoretically optimum image contrast and measurement accuracy, as well as its association with system configuration parameters. Then we will introduce our proposed computational framework that extends the traditional temporal-spatial correlation algorithm in multiple stereo image pairs to the encoding scheme in our research with only a single stereo image pairs.

2.1 Design of CBSP

We firstly describe how to design the BWBSP that is related with the system configuration parameters. Next, RGB channels of CBSP are coupled with three spatially and temporally independent BWBSPs.

Figure 1(a) shows a binocular stereo measurement system that is composed of two color cameras (square pixel size $Δ δ_{c}$ ), and a LCD commercial projector (square pixel size $Δ δ_{p}$ ) being placed between two cameras. Figure 1(b) gives the geometrical relationship of BWBSP design with the system configuration parameters. The baseline length, namely the distance between optical centers of two cameras, is denoted by B; the BWBSP is projected to encode the surface of tested object (e.g., mask), and its minimum cell is $Δ x_{p s}$ (equal to the minimum cell of CBSP), which is determined by Eq. (1); D_P and D_c are respectively the distances between the optical centers of projector/cameras and the object plane; f_p and f_c are the focal lengths of projector and cameras. We suppose that $Δ x_{p s}$ has m_p (m_p is an integer.) width of square pixel size $Δ δ_{p}$ of projector

Δ x_{p s} = m_{p} Δ δ_{p},

When the BWBSP is vertically projected on the surface of tested object shown in Fig. 1(b), its minimum cell

Δ x_{p s}

will be magnified by the project objective with an approximate pinhole imaging model:

Δ X_{p} \approx Δ x_{p s} \frac{D_{p}}{f_{p}},

The BWBSP distorted by the tested object is then digitally sampled by cameras with m_c (m_c is an integer.) pixels, this imaging relationship can be approximately represented by

Δ X_{p} \cos θ \approx Δ X_{p} \frac{D_{p}}{D_{c}} = Δ X_{c},

From Eq. (3), we can further derive that

Δ X_{c} \approx Δ x_{c} \frac{D_{c}}{f_{c}} = \frac{D_{c}}{f_{c}} (m_{c} Δ δ_{c}) .

Combining Eqs. (1)-(4), we have

Δ x_{p s} \approx \frac{D_{c}^{2} f_{p}}{D_{p}^{2} f_{c}} Δ x_{c},

Or

m_{p} Δ δ_{p} \approx \frac{D_{c}^{2} f_{p}}{D_{p}^{2} f_{c}} (m_{c} Δ δ_{c}) .

In order to meet the sampling theorem and obtain a good image contrast, the range of values allowed for m_c should belong to 3 to 5 [31]. There should be an emphasis on Eqs. (1)-(6) that optical distortions in the process of imaging are omitted instead of a linear model for simplicity.

Fig. 1 Relationship between designs of BWBSP and system parameters. (a) Schematic of binocular stereo measurement system; (b) Geometrical relationships amongst system parameters during imaging.

Download Full Size | PDF

We use MATLAB 2016a software platform in manner of global and local randomness to create BWBSP. The specific procedures are composed of the following major steps:

1. Creating one m × n pixels (equal to the resolution of projector) image of all black (0), then which is segmented into many regions of M × M pixels (M is an integer greater than or equal to 3);
2. Within each region, we randomly select some seed pixels with the MATLAB 2016a “RANDPERM” function;
3. Starting with the seed pixels in each region, region growing by considering the connection (three- or eight-neighborhood) of the pixel space is performed until BWBSP is entirely yielded
4. Steps 1-3 can be regarded as local random encoding. Next, a global scanning work is implemented throughout the image for the sake of ensuring the global randomness. Specifically, the values of pixels in window of c × r, c × 1 and 1 × r (including diagonal pixels of c × r) (here c and r are odd greater than or equal to 3) cannot be all 0 or 255. If so, 42~45% (experimental value) pixels randomly selected from windows will be assigned to 255 or 0.

According to the description aforementioned, we firstly create three temporally and spatially independent BWBSPs of the same image resolution (see Figs. 2(a)-2(c)), then which are coupled with three color channels RGB of one color image to create one CBSP (see Fig. 2(c)). In our designed BWBSPs, the percentage of gray value 255 is roughly 42.7%. Here, bearing in mind that color channels only serve as the encoding carrier without providing any contributions on the stereo matching. The smallest white or black square dot in Figs. 2(a)-2(c)) represents the minimum cell $m_{p} Δ δ_{p}$ of BWBSP. This creating process of one CBSP can be expressed with Eq. (7):

Fig. 2 Creation of CBSP. (a)-(c) three temporally and spatially independent BWBSPs; (d) CBSP.

Download Full Size | PDF

\begin{array}{l} R (B W B S P) \\ G (B W B S P) \\ B (B W B S P) \end{array}} \to RGB : C B S P (B W B S P) .

2.2 3-D reconstruction framework for CBSP

Based on the proposed encoding strategy, a 3-D reconstruction framework is established, as schematically shown in Fig. 3, whose main procedure can be described as below:

Fig. 3 3-D reconstruction framework based on CBSP projection.

Download Full Size | PDF

1. The captured image pair (only one image from each camera, as shown Figs. 3(a) and 3(b)) is firstly rectified to make them be strictly row-aligned (as shown Figs. 3(c) and 3(d)), whose purpose is to reduce the searching space in stereo match from two dimensions to one;
2. Extracting R/G/B images from the rectified image pair (Fig. 3(c) and 3(d)) in step. 1, we can obtain three sub-stereo image pairs (Fig. 3(e) and 3(e’), Fig. 3(f) and 3(f’) and Fig. 3(g) and 3(g’));
3. Combining pure spatial [19] with temporal [32] versions of conventional zero normalized cross correlation (ZNCC) algorithms, we propose a temporal-spatial correlation matching algorithm with a single stereo images pair (essentially, three sub-stereo image pairs from a single stereo images pair) and extend it to search the correspondences between two views. We call it extended temporal-spatial correlation matching algorithm (ETSCMA), as schematically in Fig. 3(h).
More specifically, as shown Fig. 3(h), a rectangular window (Red box) of size w_x*w_y (both w_x and w_y are odd greater than or equal to 3) centered at a pixel (i,j,N) in left rectified view is correlated with all possible windows of the same size within the right rectified view along the same row, until the correlation coefficient C_corr(i,j,i,j + △d,N) surpasses the set threshold C_th. For all pixel pairs (i,j,N) and (i,j + △d,N), the correlation can be computed by [32]
$C_{c o r r} (i, j, i, j + Δ d, N) = \frac{\sum_{t = 1}^{N} \sum_{h = - w_{y} / 2}^{w_{y} / 2} \sum_{l = - w_{x} / 2}^{w_{x} / 2} I_{S}^{L} I_{S}^{R}}{\sqrt{\sum_{t = 1}^{N} \sum_{h = - w_{y} / 2}^{w_{y} / 2} \sum_{l = - w_{x} / 2}^{w_{x} / 2} {(I_{S}^{L})}^{2}} \sqrt{\sum_{t = 1}^{N} \sum_{h = - w_{y} / 2}^{w_{y} / 2} \sum_{l = - w_{x} / 2}^{w_{x} / 2} {(I_{S}^{R})}^{2}}},$ $I_{S}^{L} = I^{L} (i + l, j + h, t) - I_{m}^{L},$ $I_{S}^{R} = I^{R} (i + l, j + Δ d + h, t) - I_{m}^{R},$

The variable △d represents the integer-pixel searching range, i.e., disparity, along the row direction; (i,j) is index of pixel position; N describes the total number of sub-image pairs. In Eqs. (9) and (10), the average intensities of left and right matched windows can be computed by

I_{m}^{L} = \frac{\sum_{t = 1}^{N} \sum_{h = - w_{y} / 2}^{w_{y} / 2} \sum_{l = - w_{x} / 2}^{w_{x} / 2} I (i + l, j + h, t)}{N (w_{y} + 1) (w_{x} + 1)},

I_{m}^{R} = \frac{\sum_{t = 1}^{N} \sum_{h = - w_{y} / 2}^{w_{y} / 2} \sum_{l = - w_{x} / 2}^{w_{x} / 2} I^{R} (i + l, j + Δ d + h, t)}{N (w_{y} + 1) (w_{x} + 1)} .

After obtaining the integer-pixel position (i, j_int, N) (j_int = j + △d) corresponding to the maximum correlation, its sub-pixel position (i, j_subpx, N) can be refined by a five point quadratic curve fitting model:

C_{c o r r} (i, j, i, j + Δ d, N) = a {(j - j_{s u b p x})}^{2} + c, j \in [j_{\max} - 2, j_{\max} + 2],

Where a and c are the fitting coefficients; (i,j_subpx,N) representing the position of the maximum value of fitting curve C_corr(i,j,i,j + △d,N) is regarded as the new matching point in place of (i, j_int, N). By this time, the obtained disparity △d should be updated by

Δ d_{s u b p x} = Δ d + j_{s u b p x} - INT (j_{s u b p x}) .

Where INT(·) represents rounding operation.

Especially important, in order to accelerate the stereo matching using ETSCMA, we firstly determine some fixed-interval sampling points (20pixles for example), and their adjacent ones can be subsequently determined within a smaller searching range until all pixels are matched, which can greatly reduce the computational effort.

4. After obtaining the disparity map, we can reconstruct the 3-D geometry (see Fig. 3(i)) of tested object (e.g. mask) with the calibration parameters of measurement system by triangulation.

3. Experiments and analysis

As shown in Fig. 1(a), we set up a binocular stereo experimental system that is composed of two cameras (Model: IDS 3060CP(color)/3060P (black and white); resolution = 1936pixel × 1216pixel; focal length = 16mm; pixel size = 4.8μm × 4.8μm, i.e., $Δ δ_{c}$ = 4.8μm) and a LCD commercial projector (Model: EPSON CB-X25; resolution = 1024 pixel × 768 pixel; focal length range varies from 18.4 to 22.08mm; pixel size = 12.5μm × 12.5μm, i.e., $Δ δ_{p}$ = 12.5μm), to validate our proposed 3-D mask solution.

The baseline length B between two cameras is about 340mm, and the optimum work distance D_c is about 700mm. Prior to experiments, we project a pure white light field with [R G B]^T = [255 255 255]^T forward the object under test, and the responses of RGB channels of cameras to the surface reflected intensity are adjusted to be basically consistent by parameters adjustment of cameras, allowing that the subsequently separated RGB images have a similar gray range in this fashion.

Based on Eqs. (1)-(6), the geometrical parameters of measurement system are specified by m_c = 3, m_p = 1 with D_p = ~800mm, f_p = 18.4mm, D_c = 700mm and f_c = 16mm. Thus, the minimum cell △x_p_s of designed BWBSP (CBSP) approximately satisfies Eq. (6):

Δ x_{p s} = 1 \times Δ δ_{p} \approx \frac{D_{c}^{2} f_{p}}{D_{p}^{2} f_{c}} (3 \times Δ δ_{c}) .

And the pattern parameterized by Eq. (15) is then projected toward the surface of tested object via the projector and captured by two synchronized cameras.

After obtaining the experimental photographs, morphology operations will be done to extract the region of interest (ROI). ROI is the area where the tested objects are located, and subsequent 3-D reconstruction procedures will be carried out on the selected ROI. An adaptive intensity threshold $I_{T h}^{A v g} (i, j)$ based on the average of R/G/B sub-images $I (i, j, c h)$ ,

I_{T h}^{A v g} (i, j) = \frac{\sum_{c h \in {R, G, B}} I (i, j, c h)}{N}, N = 3.

is used for the image segmentation by a template

\underset{c h \in {R, G, B}}{T e m p l a t e (i, j, c h)} = {\begin{cases} 1, i f I (i . j, c h) \geq I_{T h}^{A v g} (i, j) \\ 0, i f I (i . j, c h) < I_{T h}^{A v g} (i, j) \end{cases} .

The final template can be obtained by

T e m p l a t e (i, j) = \prod_{c h \in {R, G, B}} T e m p l a t e (i, j, c h) .

Figure 4(b) shows ROI of the tested mask in Fig. 4(a) using the template. In this way, foreground and background can be easily separated, and the black regions involving shadow and occlusion which will result in error matching between two view and will be not processed.

Fig. 4 (a) Captured object (mask) image; (b) ROI (template) for 3-D reconstruction.

Download Full Size | PDF

3.1 Accuracy evaluation: ceramic plate and dumbbell gauge

Referring to the Germany guide line VDI VDE2634 Part2 [33], a 140mm × 140mm ceramic standard plate with certified flatness 0.005mm, a dumbbell gauge with sphere diameters D₁ = 50.788 ± 0.005mm, D₂ = 50.779 ± 0.005mm and sphere center distance d = 100.0178mm (see Fig. 5), placed in the position ~800mm away from the projector, are tested for accuracy assessment. All experiments are implemented with the same experimental system and geometric parameters. Note that BWBSP is imaged by two 3060P gray cameras and its remaining physical parameters are the same with the color camera 3060CP. In order to ensure the accuracy of calibration when using 3060P gray and 3060CP color cameras, a mechanical mounting bracket with the same positioning holes are provided that enable them have the same baseline B. Besides, with the calibration method introduced in [36], 9 and 20 calibration images are respectively used that ensure their having similar calibration accuracy. Three patterns (1) CBSP shown in Fig. 2(d), (2) three BWBSPs (TBWBSP) shown in Figs. 2(a)-2(c)) as well as (3) one BWBSP in Fig. 2(a) or 2(b) or 2(c), are respectively projected to compare their measurement accuracy.

Fig. 5 (a) Ceramic standard plate and (b) Gauge.

Download Full Size | PDF

3-D reconstruction framework in section 2.2 with a 9 pixels × 9 pixels matching window and correlation threshold C_th = 0.1 is used. Points located on boundary areas of reconstructed results having large error will be deleted. Beyond that, any post-processing activities like interpolation, filtering and hole filling are not involved. We firstly use the ceramic standard plate as the tested object. Figure 6 shows us the error distribution maps between measured results and fitting planes using least square plane fitting algorithm; Table 1 specifies the statistical results. It is easily observable that the CBSP with standard derivation 0.054mm obviously outperforms the conventional BWBSP with standard derivation 0.069mm but slightly inferior to 0.043mm of TBWBSP. We believe that in view of single-shot image based 3-D reconstruction scheme, a remarkable improvement by CBSP with respect to BWBSP is presented.

Fig. 6 Fitting error maps respectively using (a) CBSP; (b) TBWBSP; (c) BWBSP. Units: mm.

Download Full Size | PDF

Table 1. Error statistics of measuring a ceramic plate (Unit: mm)

View Table | View all tables in this article

Figure 7 visualizes 3-D reconstructions of measuring a dumbbell gauge and corresponding fitting maps by least square sphere fitting algorithm. We also compute several statistical indexes involving the error D₁_err/D₂_err, fitting standard derivation Std. and sphere center distance error d_err, and so on, as shown in Table 2, which indicate that CBSP still has a comparable performance with TBWBSP while the BWBSP performs the worst.

Fig. 7 3-D reconstruction results (left bottom) and fitting derivation maps of a dumbbell gauge (D₁ and D₂), (a) CBSP; (b) TBWBSP and (c) BWBSP.

Download Full Size | PDF

Table 2. Error statistics of measuring a dumbbell gauge. (Unit: mm)

View Table | View all tables in this article

3.2 Object measurement

3.2.1 Stationary object: mask

For further validation of measuring real objects, a surface-complex static object and a waving paper featured with holes are tested. Figures 8(a)-8(b) show the tested mask and corresponding CBSP image distorted by the surface. 3-D reconstruction results of the tested mask using CBSP, TBWBSP and BWBSP are respectively visualized in Figs. 8(c) - 8(e). The regions marked with red box in Figs. 8(c) - 8(e) are enlarged, as shown in Figs. 8(c*) - 8(e*). We are able to clearly observe that the measurement result using CBSP is close to that of TBWBSP, while BWBSP performs worst, accompanied by many incorrect matching points (which are removed for clearer comparison) featured with holes, lots of unsuccessfully reconstructed 3-D points and lack of details.

Fig. 8 (a) photo of tested mask; (b) with CBSP illumination; (c)-(e) 3-D reconstruction results (point cloud) of mask using CBSP, TBWBSP and BWBSP and (e*)-(g*) their partly enlarged views.

Download Full Size | PDF

In principle, our proposal can be also suitable for precise 3-D measurement occasions. Like [23], the authors projected more than 15 spatially and temporally different distribution binary speckle patterns forward the object surface, then used temporal [23,32] or space-time correlation 3-D reconstruction algorithm [26] to establish the correspondence between two views. As a comparison, we also carry out a similar experiment respectively projecting the same number (N) BWBSP and CBSP. The 3-D reconstruction results are given in Fig. 9 with N = 1, 3, 5, 7, 9 (9pixels × 9 pixels matching window and correlation threshold C_th = 0.1). Interestingly, 3-D reconstruction results by CBSP are much better than that of BWBSP when N is less than or equal to 3, while they result in basically similar results in case of N greater than 5. In view of qualitative observation, the reconstruction result, shown in the first image in Fig. 9(a) from left to right appears basically consistent with that of the second image in Fig. 9(b), which agrees well with the tested results of standard components shown in Table 1 and Table 2. Therefore, this phenomenon exhibits its good prospect of CBSP applications in dynamic 3-D measurements with less images and better reconstruction results.

Fig. 9 3-D Reconstructions as a function of projected pattern number N. (a) CBSP; (b) BWBSP.

Download Full Size | PDF

3.2.2 Dynamic object: a moving A4 paper

Finally, we test an A4 paper featured with three artificially made irregular holes and shaken by the author’s hand. When the projector constantly projects one CBSP onto its surface, two synchronized color cameras capture the reflected pattern at 120Hz. In the process, we analyze the first twelve images pairs. Figures 10(a)-10(c) are three sample 3-D distorted CBSP images (right views), and their point clouds are respectively visualized in Figs. 10(a*) - 10(c*). These results indicate that the moving paper can be well recovered by our proposed CBSP-based 3-D reconstruction framework. Especially the feature of three holes is also successfully recovered. The associated video can be found in Visualization 1. It is worth to note that this video is played repeatedly 5 times at frame rate of 20Hz for better visualization purpose. This experiment has clearly demonstrated the success of CBSP-based 3-D measurement solution in measuring a dynamic object.

Fig. 10 3-D reconstructions of shaking paper (associated video Visualization 1). (a)-(c) captured pattern images (right view) respectively at t = 0, 1/20, 1s/10 and corresponding point clouds (a*)-(c*) rendered by reseda and illumination for purpose of better visualization.

Download Full Size | PDF

4. Discussion

This paper reports a novel structured light encoding method and associated 3-D reconstruction framework. Our research presented in this article mainly focuses on measuring a moving object using single-shot pattern illumination with improved accuracy relative to the traditional BWBSP, which have also been quantitatively verified via the accuracy evaluations of standard components. Certainly, it can be also applied into static scenarios with many CBSPs [23] as the provided qualitative experiments in Fig. 9 that show advantages of our solution over traditional BWBSP-based 3-D reconstruction scheme. However, still there are several aspects need to be further discussed.

Regarding the choice of matching window size, large window-image in the measurement of shape will function as a low-pass filter and suppress rapid variations in the measured field [32], leading to reduction in spatial resolution or loss of high frequency details, as well as increase of calculation cost. In this sense, a choice of large or small matching window may be not absolute, a suitable size needs to attempt in view of statistics for balancing both reliability and spatial resolution [32]. This topic will be in investigation elsewhere.

The random error in the peak correlation position using Eq. (8) is evaluated by the conclusion in [32], that is

e_{s t d} = a \frac{σ^{2}}{(w + 1) \sqrt{N}} \sqrt{\frac{1 - C_{c o r r}}{C_{c o r r}}}

Where a is a dimensionless constant, σ is the average speckle size, (w + 1) (here w_x = w_y) is the side of the square window-image used, and C_corr is the obtained correlation value at the peak position, N is the number of stereo image pairs being involved in correlation calculation. According to Eq. (10), we can see that the random error e_std is only in proportion to

\sqrt{N}

if one stereo image pair yields the same C_corr and the same parameters in correlation calculation are selected.

Next, we turn to analyze the experimental results of testing a ceramic plate in Fig. 6 and Table 1. The random error (Std.) from BWBSP is about 1.6 (0.069/0.043 = 1.6046 ≈ ) times as large as the random error from TBWBSP under the same calculation parameters, which is agree well with the conclusion in Eq. (19). However, as shown Fig. 7 and Table 2, the random errors using BWBSP to measure the diameters of a dumbbell gauge are about 3 times as large as do CBSP and TBWBSP. Of great surprise, CBSP yields an approximate performance that of TBWBSP, meaning that a better application prospect of the former in dynamical scenes. Specifically, Refs [25, 34] provide in-depth and detailed investigations that can be as evidences to explain the conclusion, and how other parameters like correlation, speckle size, window size, to affect the accuracy, are also found in [34,35].

We also have noticed that CBSP is still slightly inferior to TBWBSP (See Fig. 6 and Table 1, in Fig. 7 and Table 2) in measurement, the reasons should be mainly twofold: on the one hand, the pixels interpolation lead to a gray average and reduction of resolution; on the other hand, the detection ranges of color cameras are worsen due to color coupling (three BWBSPs are coupled into RGB channels), which in turn lead to a lower signal-to-noise ratio as opposed to a BWBSP image from gray cameras of the same resolution. Therefore, the R/G/B images separated from a captured CBSP image are not equivalent to three BWBSP images.

A closer comparison between CBSP and BWBSP from Figs. 8(a) and 8(c) or Figs. 9(a) and Fig. 9(b) with N = 1 by testing a shape-complex mask, we can further see that CBSP increases the spatial resolution, reliability and precision with the same parameters of correlation calculation. This demonstrates advantages of our proposed solution over the traditional BWBSP in the application of dynamic scenes.

There are disadvantages of the CBSP-based 3-D measurement technique. Firstly, light-colored surface objects are selected for test. In the presence of colorful objects the measurement effect could be discounted since the physical property will override the projected color encoding pattern. Nevertheless, we have confirmed that only the R/G/B images separated from the captured CBSP image are not in saturation, different responses and color crosstalk amongst them have almost no influence on the result in measuring the same object whether or not color correction is used [36]. In this regard, it differs from [37] that a rigorous color correction is necessary. Another disadvantage lying in the projection manner of pattern since it needs a more costly digital color projector in comparison with pure optical ones in [24–26]. The authors believe that an improved approach can be inspired from [38] as an alternative.

5. Summary

A novel single-shot color pattern encoding strategy and 3-D reconstruction framework for measuring both static and moving objects based on binocular stereo vision principle are presented. The proposed ETSCMA is successfully applied to a single stereo image pair based 3-D reconstruction work. Random errors of measuring a standard plane and diameters of a dumbbell gauge have been shown that the CBSP method respectively yields approximately $\sqrt{3}$ and 3 times as good as does BWBSP. This trend can be also found from the experiment of testing a shape-complex mask that the CBSP increases the spatial resolution, reliability and precision in case of the same calculation parameters. We also demonstrate the capability of measuring a moving object characterized by spatially separately holes without any motion compensation and parameter adjustment. Currently, CBSP is still not recommended to color rich objects.

Funding

Full-time Postdoctoral Research and Development Funds of Sichuan University (2017SCU12023); China Post-doctoral Special Funds (2016T90847); China's Post-doctoral Science Fund (2015M572472); National Major Instrument Special Fund (2013YQ490879).

References and links

1. J. Geng, “Structured-light 3D surface imaging: a tutorial,” Adv. Opt. Photonics 3(2), 128–160 (2011). [CrossRef]

2. J. Salvi, J. Pagès, and J. Batlle, “Pattern codification strategies in structured light systems,” Pattern Recognit. 37(4), 827–849 (2004). [CrossRef]

3. Z. Y. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). [CrossRef]

4. J. Heikkila and O. Silven, “A four-step camera calibration Procedure with implicit image correction,” Proc. IEEE Comput. Soc. Conf. Comput. Vision Pattern Recognition, 1106–1112 (1997).

5. Z. Jia, J. Yang, W. Liu, F. Wang, Y. Liu, L. Wang, C. Fan, and K. Zhao, “Improved camera calibration method based on perpendicularity compensation for binocular stereo vision measurement system,” Opt. Express 23(12), 15205–15223 (2015). [CrossRef] [PubMed]

6. Z. Zhang, S. Huang, S. Meng, F. Gao, and X. Jiang, “A simple, flexible and automatic 3D calibration method for a phase calculation-based fringe projection imaging system,” Opt. Express 21(10), 12218–12227 (2013). [CrossRef] [PubMed]

7. S. Zhang, Handbook of 3-D Machine Vision: Optical Metrology and Imaging (CRC Press, Taylor & Francis Group, 2013).

8. J. Dai, Y. An, and S. Zhang, “Absolute three-dimensional shape measurement with a known object,” Opt. Express 25(9), 10384–10396 (2017). [CrossRef] [PubMed]

9. Y. Wang, J. I. Laughner, I. R. Efimov, and S. Zhang, “3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique,” Opt. Express 21(5), 5822–5832 (2013). [CrossRef] [PubMed]

10. C. Jiang, T. Bell, and S. Zhang, “High dynamic range real-time 3D shape measurement,” Opt. Express 24(7), 7337–7346 (2016). [CrossRef] [PubMed]

11. B. Salahieh, Z. Chen, J. J. Rodriguez, and R. Liang, “Multi-polarization fringe projection imaging for high dynamic range objects,” Opt. Express 22(8), 10064–10071 (2014). [CrossRef] [PubMed]

12. X. Huang, J. Bai, K. Wang, Q. Liu, Y. Luo, K. Yang, and X. Zhang, “Target enhanced 3D reconstruction based on polarization-coded structured light,” Opt. Express 25(2), 1173–1184 (2017). [CrossRef] [PubMed]

13. S. Zhang, D. Van Der Weide, and J. Oliver, “Superfast phase-shifting method for 3-D shape measurement,” Opt. Express 18(9), 9684–9689 (2010). [CrossRef] [PubMed]

14. M. Takeda and K. Mutoh, “Fourier transform profilometry for the automatic measurement of 3-D object shapes,” Appl. Opt. 22(24), 3977–3982 (1983). [CrossRef] [PubMed]

15. Z. H. Zhang, “Review of single-shot 3D shape measurement by phase calculation-based fringe projection techniques,” Opt. Lasers Eng. 50(8), 1097–1106 (2012). [CrossRef]

16. S. Ettl, O. Arold, Z. Yang, and G. Häusler, “Flying triangulation--an optical 3D sensor for the motion-robust acquisition of complex objects,” Appl. Opt. 51(2), 281–289 (2012). [CrossRef] [PubMed]

17. J. García, Z. Zalevsky, P. García-Martínez, C. Ferreira, M. Teicher, and Y. Beiderman, “Three-dimensional mapping and range measurement by means of projected speckle patterns,” Appl. Opt. 47(16), 3032–3040 (2008). [CrossRef] [PubMed]

18. T. Fricke-Begemann, “Three-dimensional deformation field measurement with digital speckle correlation,” Appl. Opt. 42(34), 6783–6796 (2003). [CrossRef] [PubMed]

19. Z. Chen, X. Zhang, and S. Fatikow, “3D robust digital image correlation for vibration measurement,” Appl. Opt. 55(7), 1641–1648 (2016). [CrossRef] [PubMed]

20. M. A. Sutton, J. J. Orteu, and H. Schreier, Image Correlation for Shape, Motion and Deformation Measurements: Basic Concepts, Theory and Applications (Springer US, 2009).

21. B. Pan, H. Xie, and Z. Wang, “Equivalence of digital image correlation criteria for pattern matching,” Appl. Opt. 49(28), 5501–5509 (2010). [CrossRef] [PubMed]

22. B. Pan, H. Xie, Z. Wang, K. Qian, and Z. Wang, “Study on subset size selection in digital image correlation for speckle patterns,” Opt. Express 16(10), 7037–7048 (2008). [CrossRef] [PubMed]

23. A. Wiegmann, H. Wagner, and R. Kowarschik, “Human face measurement by projecting bandlimited random patterns,” Opt. Express 14(17), 7692–7698 (2006). [CrossRef] [PubMed]

24. M. Schaffer, M. Grosse, B. Harendt, and R. Kowarschik, “High-speed three-dimensional shape measurements of objects with laser speckles and acousto-optical deflection,” Opt. Lett. 36(16), 3097–3099 (2011). [CrossRef] [PubMed]

25. M. Schaffer, M. Grosse, and R. Kowarschik, “High-speed pattern projection for three-dimensional shape measurement using laser speckles,” Appl. Opt. 49(18), 3622–3629 (2010). [CrossRef] [PubMed]

26. B. Harendt, M. Grosse, M. Schaffer, and R. Kowarschik, “3D shape measurement of static and moving objects with adaptive spatiotemporal correlation,” Appl. Opt. 53(31), 7507–7515 (2014). [CrossRef] [PubMed]

27. https://en.wikipedia.org/wiki/PrimeSense.

28. https://developer.microsoft.com/en-us/windows/kinect.

29. https://www.apple.com/cn/iphone-x/.

30. https://realsenseapp.intel.com/.

31. G. Lionello and L. Cristofolini, “A practical approach to optimizing the preparation of speckle patterns for digital-image correlation,” Meas. Sci. Technol. 25(10), 107001 (2014). [CrossRef]

32. M. Sjödahl and P. Synnergren, “Measurement of shape by using projected random patterns and temporal digital speckle photography,” Appl. Opt. 38(10), 1990–1997 (1999). [CrossRef] [PubMed]

33. VDI/VDE 2634 Blatt 2: 2002–08 Optische 3D-Messsysteme; Systeme mit flachenhafter Antastung. Berlin: Beuth Verlag.

34. M. Sjödahl, “Accuracy in electronic speckle photography,” Appl. Opt. 36(13), 2875–2885 (1997). [CrossRef] [PubMed]

35. P. Lutzke, M. Schaffer, P. Kühmstedt, R. Kowarschik, and G. Notni, “Experimental comparison of phase-shifting fringe projection and statistical pattern projection for active triangulation systems,” Proc. SPIE 8788, 878813 (2013). [CrossRef]

36. P. Zhou, J. P. Zhu, X. Y. Su, H. L. Jing, and X. Zhang, “Three-dimensional shape measurement using color random binary encoding pattern projection,” Opt. Eng. 56(10), 104102 (2017). [CrossRef]

37. P. S. Huang, Q. Hu, F. Jin, and F. P. Chiang, “Color-encoded digital fringe projection technique for high-speed 3-D surface contours,” Opt. Eng. 38(6), 1065–1071 (1999). [CrossRef]

38. L. Yu and B. Pan, “Full-frame, high-speed 3-D shape and deformation measurements using stereo-digital image correlation and a single color high-speed camera,” Opt. Lasers Eng. 95, 17–25 (2017). [CrossRef]

Method	Max. Err ⁽¹⁾	Avg. Err⁽²⁾
CBSP	﹢0.233/-0.215	0.00084	0.054
CBSP	﹢0.233/-0.215	﹢0.043/-0.042	0.054
TBWBSP	﹢0.212/-0.171	−0.00055	0.043
TBWBSP	﹢0.212/-0.171	﹢0.033/-0.034	0.043
BWBSP	﹢0.697/-0.711	0.00051	0.069
BWBSP	﹢0.697/-0.711	﹢0.054/-0.053	0.069

Method	D₁ _err / D₂ _err ⁽¹⁾	D_ierr/ D_i*100%(i = 1,2)⁽²⁾	Std⁽³⁾	d_err ⁽⁴⁾	d_err/d*100%⁽⁵⁾
CBSP	0.286	0.56%	0.072	0.237	0.24%
CBSP	0.245	0.48%	0.074	0.237	0.24%
TBWBSP	0.219	0.43%	0.082	0.203	0.20%
TBWBSP	0.293	0.58%	0.094	0.203	0.20%
BWBSP	1.071	2.11%	0.225	1.084	1.1%
BWBSP	1.371	1.54%	0.228	1.084	1.1%

Method	Max. Err ⁽¹⁾	Avg. Err⁽²⁾
CBSP	﹢0.233/-0.215	0.00084	0.054
CBSP	﹢0.233/-0.215	﹢0.043/-0.042	0.054
TBWBSP	﹢0.212/-0.171	−0.00055	0.043
TBWBSP	﹢0.212/-0.171	﹢0.033/-0.034	0.043
BWBSP	﹢0.697/-0.711	0.00051	0.069
BWBSP	﹢0.697/-0.711	﹢0.054/-0.053	0.069

Method	D₁ _err / D₂ _err ⁽¹⁾	D_ierr/ D_i*100%(i = 1,2)⁽²⁾	Std⁽³⁾	d_err ⁽⁴⁾	d_err/d*100%⁽⁵⁾
CBSP	0.286	0.56%	0.072	0.237	0.24%
CBSP	0.245	0.48%	0.074	0.237	0.24%
TBWBSP	0.219	0.43%	0.082	0.203	0.20%
TBWBSP	0.293	0.58%	0.094	0.203	0.20%
BWBSP	1.071	2.11%	0.225	1.084	1.1%
BWBSP	1.371	1.54%	0.228	1.084	1.1%

Optical 3-D surface reconstruction with color binary speckle pattern encoding

Abstract

1. Introduction

2. Principle

2.1 Design of CBSP

2.2 3-D reconstruction framework for CBSP

3. Experiments and analysis

3.1 Accuracy evaluation: ceramic plate and dumbbell gauge

3.2 Object measurement

3.2.1 Stationary object: mask

3.2.2 Dynamic object: a moving A4 paper

4. Discussion

5. Summary

Funding

References and links

Supplementary Material (1)

Cited By

Figures (10)

Tables (2)

Equations (19)

Optics Express