Perspective projection model for prism-based stereovision

Xiaoyu Cui; Yue Zhao; Kahbin Lim; Teresa Wu

doi:10.1364/OE.23.027542

1. Introduction

Stereovision is an important branch of computer vision. It aims to recover the shape, size or location of an object from two or more images taken from different viewpoints by a non-contact measuring way. There are a variety of classification methods for stereovision. According to the number of cameras used, it can be divided into multi-camera based method and single-camera based method.

Multi-camera based method employs two or more cameras to capture images from different views. It is currently used for space exploration [1], panoramic reconstruction [2], robot navigation or other volume static target measurements and surface reconstruction [3]. But the multi-purpose visual measurements still have some inherent problems: 1)due to multiple cameras which are associated, there were high cost; 2) In order to obtain a high accuracy, the distance of baseline must be long enough, resulting in large system size; 3) the working condition between multiple cameras cannot completely consistent, thus affecting the measurement accuracy; 4) cameras capture time cannot be synchronized accurately, which will cause the errors in dynamic target measurement. To overcome these limitations, the concept of single-camera stereo vision has been proposed and attracts many researchers.

The single-camera stereovision system can be classified into two categories. One is depth recovery by exploiting some known cues such as the environment or the movement of the camera [4–6 ]. The other is depth recovery by the reflective or refractive properties of optical devices such as mirrors [7] or prisms. Among these methods, prism-based stereovision has the simplest system structure. With the use of only one camera, it could reduce the cost and eliminate the multi-camera synchronization problem automatically. The system is more compact which can be used in some applications with space constraint. These advantages enable the system with a good potential application that has developed rapidly in recent decades.

In this paper, a novel perspective projection model for prism-based stereovision is proposed. Different from previous studies, an affine transformation matrix is proposed to establish the relationship between object point and its virtual image. Then, the perspective projection model of prism based stereovision can be expressed directly by taking the affine matrix into a pinhole camera model. The methods for model parameter calibration, distortion correction and 3D reconstruction were also provided. Our proposed model can connect the prism imaging with homography between the model plane and its image, not only it can express the projection procedure of prism based stereovision, but it also can solve the 3D reconstruction problem with high precision.

2. Lecture review

Lee and Kweon [8] first proposed prism based single-lens stereovision system in 2000. In their study, a ray tracing method was used to calculate the relationship between object point and its image point. This groundbreaking method changed the scope of application of prism in common optical system, which lay a foundation for prism as an image lens. However, they ignored the nonlinear distortion of the image and derived transformation model by approximate calculation, which affected the accuracy of 3D reconstruction.

Lim and Xiao [9] introduced some new understanding about the system by proposing the concept of a “virtual camera.” They assume that one image captured by this system is divided into two halves which is equivalent to two images captured using a two camera system. This idea converted the single-lens stereovision system using a prism into a conventional stereovision system using two cameras, which enables the system to be more easily understood and implemented. Furthermore, they extended this system from binocular to multi-ocular [10], and proposed epipolar rectification method [11], stereo matching [12] and error analysis method [13], Since then, the concept of virtual camera has been developed and used widely.

There were many significant advantages compared with mufti-camera stereovision systems, such as potential cost-effectiveness, compactness, low computational cost, and elimination of the complicated cameras synchronization process. These multitude of benefits extend the scope of application of the system. Thus, researchers were interested to devote attention to study this system [14]. extended single prism to prism microarray, which made the system smaller. After that [15], designed an optical lens by making used of a four-ocular prism. In order to solve the problem of non-linear distortion [16], proposed a distortion rectifying method by adding another auxiliary camera [17]; used a concave mirror for discrepancy compensating. In order to increase the accuracy of the system [18], analyzed the color aberration of the prism and concluded that with the increase of prism angle, the nonlinear magnification and color aberration will become severely undesirable [19]; derived a transformation matrix which can express the relationship between an object point and its image by geometrical optics, and a position estimation method of multi-ocular prism was proposed in [20]. Recently [21], improved the method in [19] and enhanced the precision of this kind of stereovision system.

With the improvement of the theoretical system, prism-based stereovision has been applied to many fields [22]. designed a free-form prism for the microscopic measurement, with which a complete three-dimensional image can be captured at one time [23]; applied this method to particle image velocity measurement [24]; focused on the method for recognizing partially occluded objects; and [25] integrated prism to smart phone and developed an app which can implement 3D reconstruction in phone.

With the efforts of so many researches, the prism based single-camera stereovision develops rapidly. It has experienced the crossover from approximate evaluation [8] to accurate calculation [11,19,21 ]; from two-views [8,9 ] to multi-views [10,15,20 ]; form theoretical study [8–20 ] to various applications [21–25 ]. The methods which have been proposed include the main processing flow of stereovision, such as calibration methods [8–10, 19–21 ], stereo matching algorithms [11,12 ] and 3D reconstruction techniques [14,15,21–23 ]. However, there is still no one complete model which can denote the projection procedure of an object point from world coordinate to image coordinate. On the basis of the former researches, we introduced a perspective projection model for prism-based stereovision, with which the homography between the model plane and its image can be expressed and calculated easily.

3. Virtual image formed by prism

In this section, the prism is considered as an optical lens, and its imaging model as well as imaging procedure will be derived so that the refraction of light by prism could be expressed through an object point and its virtual image.

3.1 Refraction of an arbitrary plane

As shown in Fig. 1 , Π is an arbitrary plane with equation $n_{x} x + n_{y} y + n_{z} z + d = 0$ in 3D space and separates two different media with refractive index n and n’. $P = {[x_{0}, y_{0}, z_{0}]}^{T}$ is a point in front of this plane. PB is a line segment which is perpendicular to plane Π. PC is an incident ray intersecting Π at point C, and the back extension line of its refracted ray intersects PB at point $P' = {[x', y', z']}^{T}$ . According to Shell’s law, it is clear that α is equal to the incident angle, and α’ is equal to the angle of refraction.

Fig. 1 Ray refraction of an arbitrary plane.

Download Full Size | PDF

The relationship between point P and P’ can be expressed as below [19].

[\begin{matrix} x' \\ y' \\ z' \end{matrix}] = [\begin{matrix} q n_{x}^{2} + 1 & q n_{x} n_{y} & q n_{x} n_{z} \\ q n_{x} n_{y} & q n_{y}^{2} + 1 & q n_{y} n_{z} \\ q n_{x} n_{z} & q n_{y} n_{z} & q n_{z}^{2} + 1 \end{matrix}] [\begin{matrix} x_{0} \\ y_{0} \\ z_{0} \end{matrix}] + q d [\begin{matrix} n_{x} \\ n_{y} \\ n_{z} \end{matrix}]

Equation (1) can be rewritten as

P' = A P + t

According to Eq. (2), a point refracted through an arbitrary plane can be expressed by an affine transformation matrix A and a translation vector t. Moreover, matrix A also can be regard as a scaling matrix along the N direction with the scale factor q.

3.2 Imaging model of multi-ocular prism

A multi-ocular prism used in a stereovision system is composed of one back plane and multi inclined planes. If we do not consider the occlusion, the number of views in an image plane is equal to the number of inclined planes. So no matter how many inclined planes there are of a prism, if we captured one point in the image plane, the ray sending from the object point in 3D space will project to this image point after two refraction through the prism. From Eq. (1), a 3D point and its virtual images by the two refraction of prism can be expressed by combining these two processes together.

For the sake of simplicity, only light refraction of binocular prism in the horizontal (O−XZ) plane is discussed. As shown in Fig. 2, P is an object point in 3D space, P’ is the virtual image point of P by the refraction of backplane of prism, P_l and P_r are virtual image points of P’ by the refraction of left and right inclined plane respectively. From Eq. (2), the virtual image point P’, P_l and P_r can be expressed as

Fig. 2 Virtual image point formed by prism.

Download Full Size | PDF

P' = A_{B} P + t_{B}

{\begin{matrix} P_{l} = A_{L} P' + t_{L} \\ P_{r} = A_{R} P' + t_{R} \end{matrix}

Substituting Eq. (3) into Eq. (4), we have,

{\begin{matrix} P_{l} = A_{L} A_{B} P + A_{L} t_{B} + t_{L} = A_{L B} P + t_{L B} \\ P_{r} = A_{R} A_{B} P + A_{R} t_{B} + t_{R} = A_{R B} P + t_{R B} \end{matrix}

The imaging model of a bi-prism can be represented by Eq. (5). The virtual images of any object points can be regarded as the combination of an affine transformation and a translation, which also could be expanded to multi-ocular prism.

4. Perspective projection model for prism based stereovision

A coordinate system attached to a pinhole camera, whose origin O coincides with the pinhole, and a bi-prism is placed in front of the image plane. Without loss of generality, the position of the prism is discretionary on the premise that the two virtual images could be captured by camera image plane synchronously, as shown in Fig. 3 .

Fig. 3 Image projection in camera reference. (a) side view; (b) top view.

Download Full Size | PDF

4.1 Perspective projection model

A 3D point is denoted by $P = {[X, Y, Z]}^{T}$ . A 2D point is denoted by $p = {[u, v]}^{T}$ . If we use $\tilde{p}$ and $\tilde{P}$ to denote the homogeneous coordinates: $\tilde{P} = {[X, Y, Z, 1]}^{T}$ and $\tilde{p} = {[u, v, 1]}^{T}$ , the relationship between a 3D point P and its image projection p is given by

s \tilde{p} = M_{int} [R_{e} t_{e}] \tilde{P}

where s is an arbitrary scale factor, [R_e t_e], called the extrinsic parameters, is the rotation matrix and the translational vector which relate the world coordinate system to the camera coordinate system, and M_int is the camera intrinsic matrix. Then the object point in camera coordinates can be expressed as

{\tilde{P}}^{C} = [\begin{matrix} R_{e} & t_{e} \\ 0^{T} & 1 \end{matrix}] \tilde{P} = M_{e x t} \tilde{P}

As shown in Fig. 3, points P_l and P_r are virtual images of P which formed by prism in camera coordinate, and p_l and p_r are projection points of P on the image plane. From Eqs. (5), (6) and (7) , we have

{\begin{matrix} {\tilde{P}}_{l} = [A {}_{L B} t_{L B}] {\tilde{P}}^{C} = [A {}_{L B} t_{L B}] M_{e x t} \tilde{P} = M_{L B} M_{e x t} \tilde{P} \\ {\tilde{P}}_{r} = [A {}_{R B} t_{R B}] {\tilde{P}}^{C} = [A {}_{R B} t_{R B}] M_{e x t} \tilde{P} = M_{R B} M_{e x t} \tilde{P} \end{matrix}

The perspective projection model of bi-prism based stereovision system can be acquired by substituting Eq. (8) into Eq. (6),

{\begin{matrix} s {\tilde{p}}_{l} = M_{int} M_{L B} M_{e x t} \tilde{P} \\ s {\tilde{p}}_{r} = M_{int} M_{R B} M_{e x t} \tilde{P} \end{matrix}

So far, the projection model of prism based stereovision can be regarded as the combination of an intrinsic matrix, a scale-translate matrix and a rotate-translate matrix. The scaling factor of the matrix $M_{L B}$ and $M_{R B}$ will be discussed in the next subsection.

4.2 Scaling factor

From Eqs. (1) and (9) , an image point can be determined by an affine transformation (with scaling factor q) and a translation. Therefor the parameter q will affect the results of perspective projection. In fact, when the prism is considered as an optical lens, the position of the virtual images will be changed according to different viewpoint, which is also closely related to scaling factor q.

According to section 3.1, the scaling factors for matrix M_RB in Eq. (9) can be defined as

q_{R} = \frac{\tan α_{1}}{\tan α_{2}} - 1, q_{B} = \frac{\tan α_{3}}{\tan α_{4}} - 1

where

q_{L}

and

q_{B}

express the scaling factors of affine matrices

A_{L}

and

A_{B}

. The incident ray enters the bottom of the prism at angle

α_{1}

(with the bottom surface normal). The ray exits from the top surface with angle

α_{2}

, and angle

α_{3}

is the refraction angle at the bottom surface,

α_{4}

is the incident angle to the top surface. According to Snell’ law, Eq. (10) can be written as

q_{R} = \frac{n \cos α_{1}}{\cos α_{2}} - 1, q_{B} = \frac{\cos α_{3}}{n \cos α_{4}} - 1

where n denote the relative refraction index of the prism. If the unit vectors are

\vec{O I_{1}} = V_{1}

,

\overset{⇀}{I_{1} I_{2}} = V_{2}

and

\overset{⇀}{I_{2} P} = V_{3}

, let Nr be the surface normal of the right plane, N_b be the surface normal of the backplane of the prism, from Eq. (11) we have

q_{R} = \frac{n V_{2} \cdot N_{r}}{V_{1} \cdot N_{r}} - 1, q_{B} = \frac{V_{3} \cdot N_{b}}{n V_{2} \cdot N_{b}} - 1

By making use of the vector model of Snell’s law, the scaling factor q_r and q_b are given by(the detailed deducing can be find in Appendix A):

{\begin{matrix} q_{R} = {(\frac{n^{2} - 1}{k^{2}} + 1)}^{\frac{1}{2}} - 1 = f_{R} (k) \\ q_{B} = n \cos θ (q_{R} + 1) k - 1 = f_{B} (k) \end{matrix}

where

k = V_{1} \cdot N_{l}

. The same results can be derived for q_R. From the above results, we conclude: 1) different points of the image plane have different scaling factor; 2) the affine matrices which can translate different points in 3D space to their virtual images have different scaling factors. In other words, different object points have different projection matrix, which lead to image distortion.

5. Parameter calibration and 3D reconstruction of prism based stereovision

5.1 Model parameter calibration

From Eq. (9) we know that the projection process from an object point to the image plane by prism can be formulated as the combination of an intrinsic matrix, an affine matrix and an extrinsic matrix. Since the intrinsic matrix of the camera can be easily calculated by single camera calibration, the calibration problem becomes to find the affine matrix and the extrinsic matrix. In Euclidean coordinates, the projection model of bi-prism based stereovision system can be written as

{\begin{matrix} s {\tilde{p}}_{l} = M_{L} \tilde{P} \\ s' {\tilde{p}}_{r} = M_{R} \tilde{P} \end{matrix}

where M_L and M_R are given by the following equations:

{\begin{matrix} M_{L} = M_{int} M_{L B} M_{e x t} \\ M_{R} = M_{int} M_{R B} M_{e x t} \end{matrix}

They can be regarded as the projection matrix and be calculated by normal single camera calibration method, such as Direct Linear Transformation. Then we can decompose the extrinsic matrix and acquire the following equations:

{\begin{matrix} H_{L} = M_{int}^{- 1} M_{l} = M_{L B} M_{e x t} = [A_{L} A_{B} R_{e} A_{L} A_{B} t_{e} + A_{L} t_{B} + t_{L}] \\ H_{R} = M_{int}^{- 1} M_{r} = M_{R B} M_{e x t} = [A_{R} A_{B} R_{e} A_{R} A_{B} t_{e} + A_{R} t_{B} + t_{R}] \end{matrix}

The solution of Eq. (16) can be refined through maximum likehood inference. The detailes deducing as well as the initial guess can be found in Appendix B.

5.2 Distortion correction

As discussed in section 4.2, the scaling factor of projection model is determined by the position of 3D object point, so different object points have different projection matrix, this will lead to the image distortion problem. The form of distortion is determined by the parameter k shown in Eq. (13). According to Fig. 3, if we set $V_{1} = (x, y, f)$ and $N_{r} = (n_{r x}, n_{r y}, n_{r z})$ , then k can be written as

k = V_{1} \cdot N_{r} = \frac{x}{\sqrt{x^{2} + y^{2} + f^{2}}} \cdot n_{_{r x}} + \frac{y}{\sqrt{x^{2} + y^{2} + f^{2}}} \cdot n_{_{r y}} + \frac{f}{\sqrt{x^{2} + y^{2} + f^{2}}} \cdot n_{_{r z}}

After simplification, we have

(n_{r x}^{2} - k^{2}) x^{2} + (n_{r y}^{2} - k^{2}) y^{2} + 2 n_{r x} n_{r y} x y + 2 f n_{r x} n_{r z} x + 2 f n_{r y} n_{r z} y + f^{2} n_{r z}^{2} - k^{2} f^{2} = 0

Δ = {(2 n_{r x} n_{r y})}^{2} - 4 (n_{r x}^{2} - k^{2}) (n_{r y}^{2} - k^{2}) = 4 k^{2} (n_{r x}^{2} + n_{r y}^{2} - k^{2})

As

\cos \frac{π}{4} \leq k \leq \cos θ

Therefore,

Δ < 0

Thus, the form of distortion is a series of elliptic curves. We use the radial distortion to approximately express the distortion caused by prism, which can be written as

{\begin{matrix} \hat{u} = u + (u - u_{0}) [k_{1} r^{2} + k_{2} r^{4}] \\ \hat{v} = v + (v - v_{0}) [k_{1} r^{2} + k_{2} r^{4}] \end{matrix}

where k₁ and k₂ are the coefficients of the radial distortion, (u,v) is the ideal image coordinates,

(\hat{u}, \hat{v})

is the the corresponding real observed image coordinates, r is the radius.

5.3 3D reconstructions

Once the scaling factor being defined, all the object points will have the same projection transformation, then the coordinates of 3D points in world coordinates could be calculated uniquely. According to Fig. 3, if the corresponding points are $p_{l} = {(u, v, 1)}^{T}$ and $p_{r} = {(u', v', 1)}^{T}$ , let

M_{L} = [\begin{matrix} m^{1 T} \\ m^{2 T} \\ m^{3 T} \end{matrix}], M_{R} = [\begin{matrix} m'^{1 T} \\ m'^{2 T} \\ m'^{3 T} \end{matrix}]

The equations of reflected ray in world coordinates are

{\begin{matrix} m^{2 T} P - v m^{3 T} P = 0 \\ m^{1 T} P - u m^{3 T} P = 0 \end{matrix}, {\begin{matrix} m'^{2 T} P - v' m'^{3 T} P = 0 \\ m'^{1 T} P - u' m'^{3 T} P = 0 \end{matrix}

where P is the homogeneous coordinates of object point, then we have

[\begin{matrix} \begin{matrix} m^{2 T} \\ m^{1 T} \\ m'^{2 T} \\ m'^{1 T} \end{matrix} & \begin{matrix} - v m^{3 T} \\ - u m^{3 T} \\ - v m'^{3 T} \\ - u m'^{3 T} \end{matrix} \end{matrix}] P = 0

Thus the coordinate of 3D points can be determined by solving Eq. (24).

6 Experimental results and analysis

The camera used in the experiments is a Prosilica GT1910 AVT CCD camera, with a Computar M2518-MPW low distortion lens. The image resolution is 1920*1080. Three bi-prism with different angles used in the experiments are made of K9 optical glass (refraction index 1.51630). Camera and prism are mounted on a mechanical stand which installs venire calipers in the X, Y, Z axes as well as on a rotational stage. The relative positions between the camera and prism are known and adjustable, as shown in Fig. 4 .

Fig. 4 Optical system set-up.

Download Full Size | PDF

In order to validate the proposed projection model, three experiments are conducted. In the first experiment, the reprojection errors of the points in the calibration board were calculated to evaluate the precision of proposed model. In the second experiment, the reconstruction errors of 3D discrete points are analyzed to estimate the performance of proposed model for 3D measurements. At last, the result of a 3D reconstruction of a complicated surface is measured to estimate the performance of the proposed model on 3D morphology reconstruction.

6.1 Reprojection error

Single camera calibration. In this experiment, a circular board was used for feature point extraction. The dimensions of the feature points in the calibration board were 20*20, and the horizontal and vertical interval of feature points were 15 mm. There was only one camera in our stereovision system. So it is easier for the step to calibrate the camera. A well-known calibration algorithm by Zhang [26] was implemented to calculate the intrinsic and external parameters of the camera.

Calculating the projection matrix for each view. The bi-prism was settled in front of the camera, and then the images captured by our system were divided into two segments. We still used calibration board as the feature points in 3D space, and three bi-prisms with angles 10°, 15° and 20° were used for comparing with the experimental results. From Eq. (16), we can acquire the projection matrix for each view of prism. An initial guess of the parameters can be obtained with the technique described in Section3.1. After that, the imaging matrix and the external matrix can be decomposed.

Reprojection error. Figure 5 shows the reprojection errors of a bi-prism based stereovision system with angles 10°, 15° and 20° respectively. The circular board was placed at some fix positions which were parallel to the camera image plane, and the sampling interval was 500 mm, so we can compare the average reprojection errors with different distances. As shown in these three figures, the accuracy of our proposed method is sufficient for a large range of prism-based stereovision systems. However, the accuracy was inversely proportional to the distance between cameras focus and object. In the meantime, the reprojection errors of the small angle prism are less than that of the large angle prism.

Fig. 5 Reprojection errors with the change of distance. (a)the angle 10°; (b)the angle 15°; (c)the angle 20°.

Download Full Size | PDF

6.2 3D reconstruction of discrete points

The 3D reconstruction of discrete points can be used in stereovision measurement. Thus, we designed a depth and a length measurement experiment to test and evaluate the feasibility of our proposed method. The calibration board was reused as its feature points could be extracted easily and accurately. The programs of reconstruction were implemented in VC2008 with OpenCV 2.3.1 and all the results were acquired on the same condition.

Fig. 6 Depth error for different angle.

Download Full Size | PDF

(1) Depth measurement. The calibration board was set at some known positions, and the distance from the board to the image plane of camera can be measured by laser with distance tolerance of 0.01 mm. During the experiment, the images of calibration board were captured in sequence. As shown in Fig. 6, we compared our depth measurement results with the really distance of feature points. There were 400 sampling points for each distance, and we use the average error to evaluate the performance of our system with the prism’s angles of 10°, 15° and 20° respectively.
(2) Length measurement. The interval of feature points in calibration board was known. So the same data which were acquired during the depth measurement were used in length measurement. In order to test and evaluate the feasibility of our method, we compared our results with the results of two camera methods. As shown in Tables 1-3
Table 1. Length error at distance 1000mm
View Table | View all tables in this article

Table 2. Length error at distance 1500mm.
View Table | View all tables in this article

Table 3. Length error at distance 2000mm.
View Table | View all tables in this article
.
(3) Error analysis. If we don’t consider the impact of the machining error and assembly error, one unavoidable problem with the biprism is the chromatic aberration. Chromatic aberration is generated from the non-constant refraction index of a material for lights with different colors. In this study, the effect of chromatic aberration was neglected and a compromised value of the refraction index was employed for the proposed projection model. However, the color aberrations become more severe when the angle θ increases further. According to Eq. (17), the angle mainly affects the scaring factor q, If $Δ_{e}$ denotes the error caused by chromatic aberration, when the angle of prism was change, we have $△ e = \frac{\partial q_{R}}{\partial θ} = \frac{\partial q_{R}}{\partial k} \frac{\partial k}{\partial θ} > 0$

Thus, with the increasing value of θ, the chromatic aberration will getting larger, and as a result, the measurement error will increase. However, when θ is close to 0, the prism will be reduced to be a thin glass plate, so much so that the effect of refraction would not exist. In this regard, there is virtually no disparity that can be measured and hence the depth. Thus, the value of θ cannot be too small.

In our work, another important parameter, the Field of View (FOV) was also found to be influenced by θ. As shown in Fig. 7 , the FOV of the camera is 2*β, the angle of the prism is θ, and the angle between the surface normal and the refracted ray which radiates from the boundary of the two half image plane are α₁ and α₂ respectively. The FOV of the prism based stereovision system can be regarded as the combination of α₁ and α₂. If the refractive index is n, we have

Fig. 7 The FOV of prism based stereovision system.

Download Full Size | PDF

{\begin{matrix} α_{1} = arc \sin {n \sin [a r c \sin [\frac{1}{n} \sin (θ + β)] - θ]} \\ α_{2} = a r c \sin {n \sin [θ - a r c \sin (\frac{1}{n} \sin θ)]} \end{matrix}

Thus, the FOV of the prism based stereovision system is mainly decided by the range of β and θ.

Figure 8 shows the FOV of the single-lens prism based stereovision system with the increasing value of θ. When θ is equal to 0, the FOV is the same as a single camera (Fig. 8 (a)); when θ is getting larger, the common view of the two half image planes is getting larger too. In practical application, we always need the object to be captured by both the image planes. However, in the situation depicted by Fig. 8(b), the common view of the two image plane was too small to be useful for stereovision. Thus, the FOV of the system should fall in the range between Fig. 8(c) and Fig. 8(f).

Fig. 8 The FOV of the prism based stereovision system with the increasing value of θ. (a) α₁ = β, α₂ = θ = 0; (b) α₁ > α₂; (c) α₁ = α₂; (d) α₁ < α₂; (e) α₁ = 0; (f) α₁ < 0.

Download Full Size | PDF

For the distance between the object and camera, according to the experimental results of depth measurement, the average depth recovered does increase with the increasing distance between the object and camera. This is because the chromatic aberration is proportional to the distance. The depth recovery of distant object seems to be more susceptible to the effect of chromatic aberration. However, do note from Fig. 8 the relationship between the prism angle θ and the intended measured distance. Configurations such as those shown in Fig. 8(b)-(c) are used to measure larger depth. They are with small θ, corresponding to a divergent FOV. Conversely, for measuring shorter depth, configurations shown in Fig. 8(d)-(f) with large values of θ are use, and they correspond to convergent FOV.

Considering these factors comprehensively, we can draw the following conclusions:

For commonly used single lens prism based stereovision system, in order to reduce the reconstruction error which is caused by chromatic aberration, the value of θ should as small as better; but in order to make sure that both image plane could acquire the object integrally, θ should be limited to a minimum value, which could be obtained by the equation: α₁ = α₂.

For smaller depth measurement purpose, system with larger values of θ should be used, but at the same time, the values should be limited to a maximal value, which could be determined by the equation: α₁ = 0. This is due to the fact that when θ is larger than this maximal value, the effect of chromatic aberration will become more pronounced to make the system of measurement ineffective.

6.3 Morphology reconstruction

Figure 9 shows an input biprism image and recovered shape for a textured model of Willis Tower. The distance from the front of the object to camera was 1100mm and the angle of prism be used was 10 degree. To find the corresponding points, a simple cross-correlation technique is used.

Fig. 9 Morphology reconstruction of a textured model. (a) captured image; (b) morphology reconstruction result.

Download Full Size | PDF

The height of the model is 27.4 mm. The errors in height between the recovered shape and the true shape is 1.5 mm.

Another morphology reconstruction result could be find in Fig. 10 . In this experiment, the distance from the front of the object to the camera was 850mm and the angle of prism used was 10 degree. The height of the model is 16.5 mm. The error in height between the recovered shape and the true shape is 1.16 mm. It can be seen that the experimental results agrees well with the original model, which demonstrates that the proposed method can be applied not only in three-dimensional measurement, but also in morphology retrieval.

Fig. 10 Morphology reconstruction of a toy. (a) captured image; (b) morphology reconstruction result.

Download Full Size | PDF

7. Conclusions

Prism based single–lens stereovision systems have many advantages compared with traditional two or more cameras’ system. On the basis of former researches, we introduced a new projection model for stereovision system using prism in this paper. Our method is based on projective geometry, which can translate the refraction of prism to an affine transformation matrix. Moreover, this method can be easily expanded to multi-ocular prism. The experiment results show that the method is efficient, robust, and has good property of convergence with small reprojection errors. It has also been demonstrated that the 3D map using the 3D structure can be reconstructed from a sequence biprism-stereo images.

Appendix A

Snell’s law can be expressed in a number of ways, one of which is in vectorial form .If the initial ray is specified by the unit vector V, after refraction the direction is V’, at the point where the ray intersects the surface between two homogeneous and isotropic media, described by indices n and n’, its normal has direction N, then V’ can be specified by V and N by the following equation

V' = \frac{1}{n} V + N ({[1 - {(\frac{1}{n})}^{2} (1 - {(V \cdot N)}^{2})]}^{1 / 2} - \frac{1}{n} (V \cdot N))

When the media is identified, the n is constant, then we can write the equation simply as

V' = f (V, N)

According to Eq. (12), if $V_{2} = {(v_{2 x}, v_{2 y}, v_{2 z})}^{T}$ ,then we have

{\begin{matrix} v_{2 x} = \frac{1}{n} v_{1 x} + f (V_{1}, N_{r}) n_{r x} \\ v_{2 y} = \frac{1}{n} v_{1 y} + f (V_{1}, N_{r}) n_{r y} \\ v_{2 z} = \frac{1}{n} v_{1 z} + f (V_{1}, N_{r}) n_{r z} \end{matrix}

V_{2} = f (V_{1}, N_{r}) = [1 - {(\frac{1}{n})}^{2} {(1 - {(V_{1} \cdot N_{r})}^{2}]}^{\frac{1}{2}} - \frac{1}{n} (V_{1} \cdot N_{r})

So,

V_{2} \cdot N_{r} = v_{2 x} n_{r x} + v_{2 y} n_{r y} + v_{2 z} n_{r z} = [1 - {(\frac{1}{n})}^{2} {(1 - (V_{1} \cdot N_{r})]}^{\frac{1}{2}}

Therefore,

q_{R} = \frac{n V_{2} \cdot N_{r}}{V_{1} \cdot N_{r}} - 1 = {(\frac{n^{2} - 1}{{(V_{1} \cdot N_{r})}^{2}} - 1)}^{\frac{1}{2}} - 1 = {(\frac{n^{2} - 1}{k^{2}} + 1)}^{\frac{1}{2}} - 1

Finally, q_L and q_B can be derived in the same way.

Appendix B

Firstly, we consider the position of the prism was placed at an ideal place: (1)the prism has consistent optical properties and its backplane is parallel to the real image plane, (2) the apex of the prism is perpendicular to the camera optical axis, and (3) the X and Y axes of the camera coordinates are parallel with the image plane coordinates and the Z axis is coincident with the camera’s optical axis. Then, the surface normal of the prism can be written as

{\begin{matrix} N_{r 0} = {(- \sin θ, 0, \cos θ)}^{T} \\ N_{l 0} = {(\sin θ, 0, \cos θ)}^{T} \\ N_{b 0} = {(0, 0, 1)}^{T} \end{matrix}

From Eq. (1), we have

{\begin{matrix} A_{R 0} = q_{R} N_{R 0} N_{R 0}^{T} + E \\ A_{L 0} = q_{L} N_{L 0} N_{L 0}^{T} + E \\ A_{B 0} = q_{B} N_{B 0} N_{B 0}^{T} + E \end{matrix}

In most cases, these conditions are hardly achieved in practical applications. There would be some alignment errors and the results would not be accurate if the above assumptions were used. If we use a Rotation matrix R to denote the skewing of prism, then the real surface normal of each plane is

{\begin{matrix} N_{l} = R N_{l 0} \\ N_{r} = R N_{r 0} \\ N_{b} = R N_{b 0} \end{matrix}

So the affine transformation changes to

{\begin{matrix} A_{R} = q_{R} N_{R} N_{R}^{T} + E = q_{R} R N_{R 0} {(R N_{R 0})}^{T} + E = q_{R} R N_{R 0} N_{R 0}^{T} R^{T} + E = R A_{R 0} R^{T} \\ A_{L} = q_{L} N_{L} N_{L}^{T} + E = q_{L} R N_{L 0} {(R N_{L 0})}^{T} + E = q_{L} R N_{L 0} N_{L 0}^{T} R^{T} + E = R A_{L 0} R^{T} \\ A_{B} = q_{B} N_{B} N_{B}^{T} + E = q_{B} R N_{B 0} {(R N_{B 0})}^{T} + E = q_{B} R N_{B 0} N_{B 0}^{T} R^{T} + E = R A_{B 0} R^{T} \end{matrix}

According to Eq.(16), let E_R express the first three rows of H_R, then

E_{R} = A_{R} A_{B} R_{e}

It could also be acquired that

E_{R} E_{R}^{T} = (A_{R} A_{B} R_{e}) {(A_{R} A_{B} R_{e})}^{T} = A_{R} A_{B} A_{B}^{T} A_{R}^{T} = A_{R} A_{B}^{2} A_{R} = R (A_{R 0} A_{B 0}^{2} A_{R 0}) R^{T} = R A_{R B R} R^{T}

Because R is an orthogonal matrix, so $H_{R} H_{R}^{T}$ and A_RBR are similar matrices, then they have the same eigenvalue. Let

{\begin{matrix} E_{R} E_{R}^{T} = U V U^{T} \\ A_{R B R} = D V D^{T} \end{matrix}

Then the rotation matrix R can be expressed as

R = U D^{- 1}

Given n 3D object points and m inclined planes of the prism, there should be n* m points in the image plane. Assume that the image points are corrupted by independent and identically distributed noise, the maximum likelihood estimation of the prism position can be obtained by minimizing the following function:

\sum_{i = 1}^{n} \sum_{j = 1}^{m} ‖ p_{i j} - \hat{m} (M_{L B}, M_{R B}, M_{e}) ‖

where

\hat{m} (M_{L B}, M_{R B}, M_{e})

is the projection of point P_i in view j, according to Eq. (9). By making use the derivation above, this function can be written as

\sum_{i = 1}^{n} \sum_{j = 1}^{m} ‖ p_{i j} - \hat{m} (R, d_{B}, θ) ‖

where R is an orthogonal matrix which only has 3 degrees of freedom, so the parameters need to be calculated are reduced to 5. This is a nonlinear minimization problem, which could be solved with the Levenberg–Marquardt algorithm.

Acknowledgments

The authors are grateful for the financial support from the National Natural Science Foundation of China (Grant Nos. 61501101, 61472069, 61402089), the Fundamental Research Funds for the Central Universities (Grant No. N130319002), and the general program of the education department of Liaoning province(Grant No. L2014086).

References and links

1. J. Y. Rau, J. P. Jhan, and Y. C. Hsu, “Analysis of oblique aerial images for land cover and point cloud classification in an urban environment,” IEEE Trans. Geosci. Rem. Sens. 53(3), 1304–1319 (2015). [CrossRef]

2. F. Huang, “Sensitivity analysis of pose recovery from multi-center panoramas,” Multimedia Tools Appl. 72(2), 1193–1213 (2014). [CrossRef]

3. P. Zhang, H. Zhao, X. Zhou, and J. Li, “Sub-aperture stitching interferometry using stereovision positioning technique,” Opt. Express 18(14), 15216–15222 (2010). [CrossRef] [PubMed]

4. X. Cao and H. Foroosh, “Camera calibration and light source orientation from solar shadows,” Comput. Vis. Image Underst. 105(1), 60–72 (2007). [CrossRef]

5. D. Li, C. Liu, and J. Tian, “Telecentric 3D profilometry based on phase-shifting fringe projection,” Opt. Express 22(26), 31826–31835 (2014). [CrossRef] [PubMed]

6. C. Li, Z. Liu, H. Xie, and D. Wu, “Novel 3D SEM Moiré method for micro height measurement,” Opt. Express 21(13), 15734–15746 (2013). [CrossRef] [PubMed]

7. A. Goshtasby and W. A. Gruver, “Design of a single-lens stereo camera system,” Pattern Recognit. 26(6), 923–937 (1993). [CrossRef]

8. D. H. Lee and I. S. Kweon, “A novel stereo camera system by a biprism,” IEEE Trans. Robot. 16(5), 528–541 (2000). [CrossRef]

9. K. B. Lim and Y. Xiao, “Virtual stereovision system: new understanding on single-lens stereovision using a biprism,” J. Electron. Imaging 14(4), 41–52 (2005). [CrossRef]

10. Y. Xiao and K. B. Lim, “A prism-based single-lens stereovision system: from trinocular to multi-ocular,” Image Vis. Comput. 25(11), 1725–1736 (2007). [CrossRef]

11. D. L. Wang, K. B. Lim, and W. L. Kee, “Geometrical approach for rectification of single-lens stereovision system with a triprism,” Mach. Vis. Appl. 24(4), 821–833 (2013). [CrossRef]

12. K. B. Lim, W. L. Kee, and D. L. Wang, “Virtual camera calibration and stereo correspondence of single-lens bi-prism stereovision system using geometrical approach,” Signal Process. Image Commun. 28(9), 1059–1071 (2013). [CrossRef]

13. W. L. Kee, K. B. Lim, Z. L. Tun, and B. Yading, “New understanding on the effects of angle and position of biprism on single-lens biprism stereovision system,” J. Electron. Imaging 23(3), 033005 (2014). [CrossRef]

14. C. Y. Chen, T. T. Yang, and W. S. Sun, “Optics system design applying a micro-prism array of a single lens stereo image pair,” Opt. Express 16(20), 15495–15505 (2008). [CrossRef] [PubMed]

15. W. S. Sun, C. L. Tien, C. Y. Chen, and D.-C. Chen, “Single-lens camera based on a pyramid prism array to capture four images,” Opt. Rev. 20(2), 145–152 (2013). [CrossRef]

16. K. Genovese, L. Casaletto, J. A. Rayas, V. Flores, and A. Martinez, “Stereo-digital image correlation (DIC) measurements with a single camera using a biprism,” Opt. Lasers Eng. 51(3), 278–285 (2013). [CrossRef]

17. Y. Maeda, D. Miyazaki, and T. Mukai, “Volumetric display using a rotating prism sheet as an optical image scanner,” Appl. Opt. 52(1), A182–A187 (2013). [CrossRef] [PubMed]

18. X. Y. Li and R. Wang, “Analysis and optimization of the stereo system with a biprism adapter,” In Proceedings of International Conference on Optical Instruments and Technology: Optical Systems and Modern Optoelectronic Instruments, Y. T. Wang, ed. (Academic, 2009), pp. 75061V. [CrossRef]

19. X. Cui, K. B. Lim, Q. Guo, and D. Wang, “Accurate geometrical optics model for single-lens stereovision system using a prism,” J. Opt. Soc. Am. A 29(9), 1828–1837 (2012). [CrossRef] [PubMed]

20. X. Cui, K. B. Lim, Y. Zhao, and W. L. Kee, “Single-lens stereovision system using a prism: position estimation of a multi-ocular prism,” J. Opt. Soc. Am. A 31(5), 1074–1082 (2014). [CrossRef] [PubMed]

21. L. F. Wu, J. G. Zhu, and H. M. Xie, “A modified virtual point model of the 3D DIC technique using a single camera and a bi-prism,” Meas. Sci. Technol. 25(11), 115008 (2014). [CrossRef]

22. L. Li and A. Y. Yi, “Design and fabrication of a freeform prism array for 3D microscopy,” J. Opt. Soc. Am. A 27(12), 2613–2620 (2010). [CrossRef] [PubMed]

23. Q. Gao, H. P. Wang, and J. J. Wang, “A single camera volumetric particle image velocimetry and its application,” Sci. China-Technol, Sci. 55(9), 2501–2510 (2012).

24. M. Zhang, Y. R. Piao, J. J. Lee, D. H. Shin, and B. G. Lee, “Visualization of partially occluded 3D object using wedge prism-based axially distributed sensing,” Opt. Commun. 313, 204–209 (2014). [CrossRef]

25. J. B. M. Numhauser and Z. Zalevsky, “Stereovision imaging in smart mobile phone using add on prisms,” 3D Research. 5(1), 1–10 (2014).

26. Z. Y. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000). [CrossRef]

Length(mm)	Absolute Length error(mm)
Length(mm)	10°prism	15° prism	20° prism	Two camera
30	0.259	0.261	0.270	0.263
60	0.536	0.536	0.566	0.544
90	0.856	0.861	0.863	0.876
120	1.177	1.184	1.284	1.193
150	1.497	1.540	1.636	1.518
180	1.902	2.017	2.259	1.966
210	2.303	2.405	2.410	2.303
240	2.804	2.960	3.216	2.828
270	3.807	4.581	5.769	3.978
Ave	1.682	1.816	2.030	1.718

Length(mm)	Absolute Length error(mm)
Length(mm)	10°prism	15° prism	20° prism	Two camera
30	0.308	0.408	0.386	0.298
60	0.634	0.644	0.781	0.633
90	1.004	1.194	1.243	0.987
120	1.227	1.313	1.799	1.219
150	1.621	2.353	2.376	1.554
180	2.188	2.908	3.255	2.122
210	2.877	4.223	4.697	2.753
240	3.388	3.764	6.36	3.352
270	6.093	9.432	11.808	5.787
Ave	2.149	2.915	3.634	2.078

Length(mm)	Absolute Length error(mm)
Length(mm)	10°prism	15° prism	20° prism	Two camera
30	0.369	0.446	0.529	0.359
60	0.742	0.914	1.076	0.735
90	1.182	1.455	1.686	1.148
120	1.440	1.841	2.162	1.428
150	1.890	2.421	2.858	1.893
180	2.502	3.172	3.687	2.509
210	3.343	4.137	4.907	3.355
240	4.144	5.392	6.092	3.824
270	7.587	9.252	9.576	6.291
Ave	2.577	3.225	3.619	2.393

Length(mm)	Absolute Length error(mm)
Length(mm)	10°prism	15° prism	20° prism	Two camera
30	0.259	0.261	0.270	0.263
60	0.536	0.536	0.566	0.544
90	0.856	0.861	0.863	0.876
120	1.177	1.184	1.284	1.193
150	1.497	1.540	1.636	1.518
180	1.902	2.017	2.259	1.966
210	2.303	2.405	2.410	2.303
240	2.804	2.960	3.216	2.828
270	3.807	4.581	5.769	3.978
Ave	1.682	1.816	2.030	1.718

Length(mm)	Absolute Length error(mm)
Length(mm)	10°prism	15° prism	20° prism	Two camera
30	0.308	0.408	0.386	0.298
60	0.634	0.644	0.781	0.633
90	1.004	1.194	1.243	0.987
120	1.227	1.313	1.799	1.219
150	1.621	2.353	2.376	1.554
180	2.188	2.908	3.255	2.122
210	2.877	4.223	4.697	2.753
240	3.388	3.764	6.36	3.352
270	6.093	9.432	11.808	5.787
Ave	2.149	2.915	3.634	2.078

Perspective projection model for prism-based stereovision

Abstract

1. Introduction

2. Lecture review

3. Virtual image formed by prism

3.1 Refraction of an arbitrary plane

3.2 Imaging model of multi-ocular prism

4. Perspective projection model for prism based stereovision

4.1 Perspective projection model

4.2 Scaling factor

5. Parameter calibration and 3D reconstruction of prism based stereovision

5.1 Model parameter calibration

5.2 Distortion correction

5.3 3D reconstructions

6 Experimental results and analysis

6.1 Reprojection error

6.2 3D reconstruction of discrete points

6.3 Morphology reconstruction

7. Conclusions

Appendix A

Appendix B

Acknowledgments

References and links

Cited By

Figures (10)

Tables (3)

Equations (43)

Optics Express