1. Introduction

Computer vision involves several disciplines, and based on studies conducted on the underlying mechanisms of human and animal vision, various models of the process can be built. The data obtained can be analyzed and processed using statistical methods [1]. An important step in computer vision technology is the calibration of the camera used. Cameras are essential for visual monitoring, mobile robot navigation, three-dimensional measurements, target recognition, video tracking, visual workpiece detection, virtual reality, and digital visualization [2–5]. Owing to the limited field of view of perspective cameras, a catadioptric camera was proposed by Hecht and Zajac [6]. Based on whether the single viewpoint of the catadioptric camera is fixed or not, Baker and Nayar [7] categorized catadioptric cameras into two types: central and noncentral. Furthermore, the mirrors of central catadioptric cameras can be divided into four types: parabolic, hyperbolic, elliptical, and planar. Central catadioptric cameras have a single effective viewpoint and a large imaging field of view and thus are especially useful in the field of computer vision.

In 2001, Geyer and Daniilidis [8] proposed that the imaging process of central catadioptric cameras is equivalent to a two-step projection on the unit viewing sphere. This provided a mathematical basis for the study of central catadioptric cameras. Based on this fact, central catadioptric cameras can be calibrated using the following five methods:

Based on an analysis of various calibration algorithms [9–22], in this study, novel algorithms are proposed to calibrate paracatadioptric cameras, using the imaging features of a space sphere in the camera and the pole-polar relationship. A sphere in space is projected onto a pair of antipodal circles on the unit viewing sphere model. According to the properties of antipodal small circles, an infinity point and symmetric axis can be obtained; in addition, we can obtain another infinity point according to the properties of antipodal points. On the image plane, the image of the symmetric axis is the polar of the vanishing point with respect to the IAC, and a vanishing line can be obtained by connecting two vanishing points. The vanishing point and line satisfy the pole-polar relationship. The camera internal parameters can be obtained using the pole-polar relationship and the orthogonal vanishing point.

The remainder of the paper is structured as follows: Section 2 introduces the projection model of a paracatadioptric camera and the relevant concepts. Section 3 describes in detail the proposed calibration algorithms based on sphere images and the pole-polar relationship. The results of evaluation tests are presented and discussed in Section 4. Finally, Section 5 presents the conclusions of the study.

2. Imaging model

In this section, we introduce the projection model of a paracatadioptric camera and the relevant concepts.

Geyer and Daniilidis [8] proposed that the imaging process of central catadioptric cameras is equivalent to a two-step projection on the unit viewing sphere (see Fig. 1). The world coordinate system ${\boldsymbol O}\textrm{ - }{x_w}{y_w}{z_w}$ is established, such that the origin is at the center, ${\boldsymbol O}$, of the unit viewing sphere, and the camera coordinate system, ${{\boldsymbol O}_c}\textrm{ - }{x_c}{y_c}{z_c}$, which takes any point in space as the origin, ${{\boldsymbol O}_c}$, is set up, such that the ${x_c}$ and ${y_c}$ axes are parallel to the ${x_w}$ and ${y_w}$ axes, respectively, and the ${z_c}$ and ${z_w}$ axes coincide with the optic axis, ${\boldsymbol O}{{\boldsymbol O}_c}$. That is to say, the image plane, $\pi$, is perpendicular to axis ${\boldsymbol O}{{\boldsymbol O}_c}$ and intersects at the principal point, ${\boldsymbol p}$.

 

Fig. 1. Imaging model of space point ${\boldsymbol A}$ and sphere ${\boldsymbol Q}$ in paracatadioptric camera.

Download Full Size | PDF

Step 1: A 3D point, ${\boldsymbol A} = {\left[ {\begin{array}{cc} {\begin{array}{ccc} {{X_w}}&{{Y_w}}&{{Z_w}} \end{array}}&1 \end{array}} \right]^\textrm{T}}$, is projected onto a pair of antipodal points ${{\boldsymbol A}_ \pm } = {\left[ {\begin{array}{cccc} {{{ \pm {X_w}} \mathord{\left/ {\vphantom {{ \pm {X_w}} {||{\boldsymbol A} ||}}} \right.} {||{\boldsymbol A} ||}}}&{{{ \pm {Y_w}} \mathord{\left/ {\vphantom {{ \pm {Y_w}} {||{\boldsymbol A} ||}}} \right.} {||{\boldsymbol A} ||}}}&{{{ \pm {Z_w}} \mathord{\left/ {\vphantom {{ \pm {Z_w}} {||{\boldsymbol A} ||}}} \right.} {||{\boldsymbol A} ||}}}&1 \end{array}} \right]^\textrm{T}}$ with $||{\boldsymbol A} ||= \sqrt {{X_w}^2 + {Y_w}^2 + {Z_w}^2}$ on the surface of the unit viewing sphere, using ${\boldsymbol O}$ as the projection center.

Step 2: Points ${{\boldsymbol A}_ \pm }$ are projected onto $\pi$ to form a pair of antipodal image points, ${{\boldsymbol a}_ \pm }$, with the camera optical center, ${{\boldsymbol O}_c}$, as the virtual projection center; here, +, - denote visible and invisible, respectively. The homogeneous coordinates of points ${{\boldsymbol a}_ \pm }$ can be expressed as

Proposition 1 [9]. In Fig. 1, if points ${{\boldsymbol a}_ + }$ and ${{\boldsymbol a}_ - }$ are a group of antipodal image points under a central catadioptric camera, then these points satisfy the following relationship:

Table 1. Relationship between length of $\xi$ (the mirror parameter) and mirror type.

View Table | View all tables in this article

3. Calibration of paracatadioptric camera

Duan and Wu [23] proposed that the imaging process for a sphere in the case of a paracatadioptric camera is as follows (see Fig. 1): a space sphere, ${\boldsymbol Q}$, is projected onto two parallel circles, ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$, on the unit viewing sphere; these are a pair of antipodal circles, and their centers are ${{\boldsymbol O}_ + }$ and ${{\boldsymbol O}_ - }$, respectively. Next, ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$ are projected onto conics ${{\boldsymbol C}_ + }$ and ${{\boldsymbol C}_ - }$, respectively, on $\pi$. ${{\boldsymbol C}_ + }$ is the sphere image, and ${{\boldsymbol C}_ - }$ is the antipodal sphere image of ${{\boldsymbol C}_ + }$.

Definition 1 [24]. In Fig. 1, the line through ${{\boldsymbol O}_ + }$ and ${{\boldsymbol O}_ - }$, is called the symmetry axis of ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$.

The symmetry axis through ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$ is ${{\boldsymbol L}_s}$. According to the projection property of antipodal circles, axis ${{\boldsymbol L}_s}$ is perpendicular to line ${{\boldsymbol L}_\infty }$ at infinity on the plane containing ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$, that is, ${{\boldsymbol L}_s} \bot {{\boldsymbol L}_\infty }$. On the image plane, $\pi$, ${{\boldsymbol l}_s}$, and ${{\boldsymbol l}_\infty }$ denote the images of ${{\boldsymbol L}_s}$ and ${{\boldsymbol L}_\infty }$, respectively. Hence, we have the following proposition:

Proposition 2. In Fig. 2, on plane $\pi$ of the paracatadioptric camera, ${{\boldsymbol l}_s}$ and the vanishing line, ${{\boldsymbol l}_\infty }$, on the plane containing ${{\boldsymbol C}_ + }$ and ${{\boldsymbol C}_ - }$, are orthogonal and meet the following condition:

 

Fig. 2. Relationship between pole ${{\boldsymbol v}_1}$ and polar ${{\boldsymbol l}_s}$ with regard to IAC.

Download Full Size | PDF

As per the property of affine invariance under projective transformation and the definition of the transformation of lines in Ref. [24], Proposition 2 is clearly true. Hence, its proof is omitted.

3.1 Calibration based on property of antipodal circles

Let ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$ intersect at two pairs of conjugate complex points, ${{\boldsymbol A}_i} (i = 1,2,3,4)$, one of which consists of circular points [25]. Let points ${{\boldsymbol A}_1}$ and ${{\boldsymbol A}_2}$ be the circular points, so that the line connecting these points is the line, ${{\boldsymbol L}_\infty }$, at infinity on the plane containing circles ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$. It intersects the line, ${{\boldsymbol L}_{34}}$, passing through points ${{\boldsymbol A}_3}$ and ${{\boldsymbol A}_4}$ at point ${{\boldsymbol V}_{1\infty }}$ at infinity. The line, ${{\boldsymbol L}_{13}}$, connecting points ${{\boldsymbol A}_1}$ and ${{\boldsymbol A}_3}$ intersects the line, ${{\boldsymbol L}_{24}}$, connecting points ${{\boldsymbol A}_2}$ and ${{\boldsymbol A}_4}$ at point ${{\boldsymbol V}_b}$; the line, ${{\boldsymbol L}_{14}}$, connecting points ${{\boldsymbol A}_1}$ and ${{\boldsymbol A}_4}$ intersects the line, ${{\boldsymbol L}_{23}}$, connecting points ${{\boldsymbol A}_2}$ and ${{\boldsymbol A}_3}$ at point ${{\boldsymbol V}_c}$; ${{\boldsymbol L}_s}$ can be obtained by connecting points ${{\boldsymbol V}_b}$ and ${{\boldsymbol V}_c}$ [26]. On plane $\pi$, the images of ${{\boldsymbol V}_{1\infty }}$ and ${{\boldsymbol L}_s}$ are denoted as ${{\boldsymbol v}_1}$ and ${{\boldsymbol l}_s}$, respectively.

Proposition 3. On plane $\pi$ of the paracatadioptric camera, given sphere images ${{\boldsymbol C}_ + }$ and ${{\boldsymbol C}_ - }$ of circles ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$, the image, ${{\boldsymbol l}_s}$, of the symmetry axis, ${{\boldsymbol L}_s}$, is the polar line of the vanishing point, ${{\boldsymbol v}_1}$, with regards to the IAC. Further,

3.2 B. Calibration based on property of antipodal points

In Fig. 1, let us take a point, ${{\boldsymbol A}_ + }$, on circle ${{\boldsymbol S}_ + }$. Then, there exists a point, ${{\boldsymbol A}_ - }$, on the antipodal circle, ${{\boldsymbol S}_ - }$. The tangent lines at points ${{\boldsymbol A}_ + }$ and ${{\boldsymbol A}_ - }$ to ${{\boldsymbol S}_ + }$ and ${{\boldsymbol S}_ - }$ are ${{\boldsymbol L}_ + }$ and ${{\boldsymbol L}_ - }$, respectively. According to the antipodal property, we have ${{\boldsymbol S}_ + }\parallel {{\boldsymbol S}_ - }$, ${\boldsymbol O}{{\boldsymbol O}_ + } = {\boldsymbol O}{{\boldsymbol O}_ - }$ and ${\boldsymbol O}{{\boldsymbol A}_ + } = {\boldsymbol O}{{\boldsymbol A}_ - }$. Thus, ${{\boldsymbol A}_ + }{{\boldsymbol O}_ + }{{\boldsymbol A}_ - }{{\boldsymbol O}_ - }$ is a parallelogram, that is, ${{\boldsymbol A}_ + }{{\boldsymbol O}_ + }\parallel {{\boldsymbol A}_ - }{{\boldsymbol O}_ - }$. Based on the tangents to the circle and given that ${{\boldsymbol L}_ + } \bot {{\boldsymbol A}_ + }{{\boldsymbol O}_ + }$, ${{\boldsymbol L}_ - } \bot {{\boldsymbol A}_ - }{{\boldsymbol O}_ - }$. In other words, ${{\boldsymbol L}_ + }\parallel {{\boldsymbol L}_ - }$. Lines ${{\boldsymbol L}_ + }$ and ${{\boldsymbol L}_ - }$ intersect at point ${{\boldsymbol V}_{2\infty }}$ at infinity. Let the polar lines of points ${{\boldsymbol V}_{1\infty }}$ and ${{\boldsymbol V}_{2\infty }}$ at infinity with regards to ${{\boldsymbol S}_ + }$ be ${{\boldsymbol H}_1}$ and ${{\boldsymbol H}_2}$, respectively. According to the definition of the diameter of a conic in Ref. [24], polar lines ${{\boldsymbol H}_1}$ and ${{\boldsymbol H}_2}$ are along two directions of the diameters of ${{\boldsymbol S}_ + }$, respectively. Hence, ${{\boldsymbol H}_1}$ and ${{\boldsymbol H}_2}$ intersect at the center, ${{\boldsymbol O}_ + }$, of ${{\boldsymbol S}_ + }$. Similarly, the center, ${{\boldsymbol O}_ - }$, of ${{\boldsymbol S}_ - }$ can be determined. On the basis of the harmonic conjugate relationship with respect to the projective transformation, point ${{\boldsymbol V}_{3\infty }}$ at infinity in the direction of ${{\boldsymbol L}_s}$ can be determined based on points ${\boldsymbol O}\textrm{,}\;{{\boldsymbol O}_ + }\textrm{,}{{\boldsymbol O}_ - }$. On the image plane, $\pi$, ${{\boldsymbol v}_3}$ is the image of ${{\boldsymbol V}_{3\infty }}$.

Proposition 4. In Fig. 3, on plane $\pi$ of the paracatadioptric camera, let ${{\boldsymbol l}_\infty }$ be a vanishing line and ${{\boldsymbol v}_3}$ be a vanishing point along the direction of ${{\boldsymbol l}_s}$. Then, ${{\boldsymbol v}_3}$ and ${{\boldsymbol l}_\infty }$ satisfy the pole-polar relationship with regard to the IAC, which is:

3.3 Calibration based on three vanishing points orthogonal to each other

Theorem 1 [24]. (Polarity Principle) If the polar line of point ${\boldsymbol a}$ with respect to conic ${\boldsymbol C}$ passes through point ${\boldsymbol b}$, then the polar line of point ${\boldsymbol b}$ with respect to conic also passes through point ${\boldsymbol a}$.

In Fig. 4, the lines at points ${{\boldsymbol A}_ + }$ and ${{\boldsymbol A^{\prime}}_ + }$ at a tangent to ${{\boldsymbol S}_ + }$ are ${{\boldsymbol L}_\textrm{ + }}$ and ${\boldsymbol L^{\prime}}$, respectively, where ${{\boldsymbol A}_ + }$ and ${{\boldsymbol A^{\prime}}_ + }$ are the two endpoints of the diameter of ${{\boldsymbol S}_ + }$. Let the point at infinity along the direction of the diameter, ${{\boldsymbol A}_ + }{{\boldsymbol A^{\prime}}_ + }$, be ${{\boldsymbol V}_{4\infty }}$. Then, ${{\boldsymbol L}_\textrm{ + }} \bot {{\boldsymbol A}_ + }{{\boldsymbol A^{\prime}}_ + }$, ${{\boldsymbol L^{\prime}}_\textrm{ + }} \bot {{\boldsymbol A}_ + }{{\boldsymbol A^{\prime}}_ + }$, and ${{\boldsymbol L}_\textrm{ + }}\parallel {{\boldsymbol L^{\prime}}_ + }$, and lines ${{\boldsymbol L}_\textrm{ + }}$ and ${{\boldsymbol L^{\prime}}_ + }$ intersect at point ${{\boldsymbol V}_{2\infty }}$ at infinity. Then, points ${{\boldsymbol V}_{2\infty }}$ and ${{\boldsymbol V}_{4\infty }}$ are a pair of points at infinity in the orthogonal directions. Further, from Theorem 1, diameter ${{\boldsymbol A}_ + }{{\boldsymbol A^{\prime}}_ + }$ is the polar ${{\boldsymbol H}_2}$ of point ${{\boldsymbol V}_{2\infty }}$ with respect to ${{\boldsymbol S}_ + }$. Point ${{\boldsymbol V}_{3\infty }}$ is the point at infinity in the direction of ${{\boldsymbol L}_s}$, and points ${{\boldsymbol V}_{2\infty }}$ and ${{\boldsymbol V}_{4\infty }}$ at infinity lie on ${{\boldsymbol L}_\infty }$ at infinity, with ${{\boldsymbol L}_s} \bot {{\boldsymbol L}_\infty }$. So, ${{\boldsymbol V}_{2\infty }}$ and ${{\boldsymbol V}_{3\infty }}$, and ${{\boldsymbol V}_{3\infty }}$ and ${{\boldsymbol V}_{4\infty }}$, are two pairs of points at infinity in the orthogonal directions. Points ${{\boldsymbol V}_{2\infty }},{{\boldsymbol V}_{3\infty }},{{\boldsymbol V}_{4\infty }}$ are three points at infinity in three directions orthogonal to each other. Three vanishing points, ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$, orthogonal to each other can be obtained on plane $\pi$ such that ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$ are the images of ${{\boldsymbol V}_{2\infty }},{{\boldsymbol V}_{3\infty }},{{\boldsymbol V}_{4\infty }}$, respectively.

Proposition 5. In Fig. 3, on plane $\pi$ of the paracatadioptric camera, given vanishing point ${{\boldsymbol v}_2}$ along the direction of the line, ${{\boldsymbol l}_ + }$, tangent to sphere image ${{\boldsymbol C}_ + }$, vanishing point ${{\boldsymbol v}_3}$ along the direction of image ${{\boldsymbol l}_s}$ of the symmetrical axis, vanishing point ${{\boldsymbol v}_4}$ along the direction of the polar line, ${{\boldsymbol h}_2}$, and three orthogonal vanishing points ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$ can be determined such that they meet the following conditions:

 

Fig. 3. Pole, ${{\boldsymbol v}_3}$, and polar, ${{\boldsymbol l}_\infty }$, with regard to the IAC and three vanishing points, ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$, orthogonal to each other.

Download Full Size | PDF

 

Fig. 4. Pair of points ${{\boldsymbol V}_{2\infty }}$ and ${{\boldsymbol V}_{4\infty }}$ at infinity in orthogonal directions.

Download Full Size | PDF

3.4 Algorithm

Based on the above analysis, ${{\boldsymbol l}_s}$ and vanishing point ${{\boldsymbol v}_1}$ are the pole and polar, respectively, with regard to the IAC as per Proposition 3 (Method 1); vanishing point ${{\boldsymbol v}_3}$ and vanishing line ${{\boldsymbol l}_\infty }$ satisfy the pole-polar relationship with respect to the IAC as per Proposition 4 (Method 2); and three orthogonal vanishing points, ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$, can be obtained using Proposition 5 (Method 3). The procedure for these algorithms is as follows:

Input: Sphere images $n(n \ge 3)$.

Output: Matrix of intrinsic parameters of camera, ${\boldsymbol K}$.

Step 1: Extract the pixel coordinates of the sphere images and the projective contour of the mirror, and solve the equations for the sphere images and the antipodal sphere images using Duan and Wu’s method [23].

Step 2: Take any point ${{\boldsymbol a}_ + }$ on sphere image ${{\boldsymbol C}_ + }$, and determine its antipodal point, ${{\boldsymbol a}_ - }$, using Eq. (3).

Step 3: Determine ${{\boldsymbol v}_1}$ using Eq. (10) and ${{\boldsymbol l}_s}$ using Eq. (5).

Step 4: Determine ${{\boldsymbol v}_2}$ using Eq. (19), and using Eq. (20).

Step 5: Determine ${{\boldsymbol v}_3}$ using Eq. (25) and ${{\boldsymbol v}_4}$ using Eq. (27).

Step 6: Once ${{\boldsymbol l}_s},\;{{\boldsymbol v}_1}$ have been determined, solve ${\boldsymbol \omega }$ based on Proposition 3 (Method 1); else if ${{\boldsymbol l}_\infty },{{\boldsymbol v}_3}$ have been determined, determine ${\boldsymbol \omega }$ using Proposition 4 (Method 2); else if ${{\boldsymbol v}_2},{{\boldsymbol v}_3},{{\boldsymbol v}_4}$ have been determined, determine ${\boldsymbol \omega }$ using Proposition 5 (Method 3);

Step 7: Using the expression ${\boldsymbol \omega } = {{\boldsymbol K}^{ - \textrm{T}}}{{\boldsymbol K}^{ - 1}}$, determine ${{\boldsymbol K}^{ - 1}}$ via the Cholesky factorization of ${\boldsymbol \omega }$, and obtain the matrix of the intrinsic parameters of the camera, ${\boldsymbol K}$, from the inverse of ${{\boldsymbol K}^{ - 1}}$.

4. Experiments

To confirm the feasibility and effectiveness of the proposed algorithms, we performed simulations and actual experiments and compared the obtained results with Refs. [12,17,20,23].

4.1 Simulations

During the simulations, we set up a virtual camera with $\xi = 1$ and intrinsic parameters matrix

 

Fig. 5. Sphere images generated using simulated camera.

Download Full Size | PDF

Canny edge detection [27] was performed to extract 200 data points for each sphere image and the mirror projective contour. In order to verify the effectiveness of the proposed algorithms, Gaussian noise with zero mean and standard deviation $\sigma ({0 \le \sigma \le 3} )$ was added to each data point.

Due to the manufacturing accuracy and the deviation due to the assembly process, the camera mirror will produce distortion, resulting in distortion of the original image. Lens distortion mainly includes radial distortion and tangential distortion. The initial values of intrinsic parameters of the camera can be obtained by the above algorithms, and the rotation matrix ${\boldsymbol R}$ and the translation vector ${\boldsymbol t}$ can be obtained using the method described in Ref. [20]. It is not much easier to solve the distortion coefficient with the sphere as the calibration than with the plane calibration, because it’s difficult to get the points correspondence between the sphere contour and its projection. Here, we use the method in Ref. [28] to match the measurement point with the ideal point. Intrinsic parameters are optimized by minimizing the following objective function:

We assume the distortion coefficient ${\boldsymbol k = }[{0.1203,0.1354,0.0106, - 0.0312} ]$. Starting from 10 pairs of point correspondence, we compared the variations of the absolute error of distortion coefficient from our proposed three algorithms with the results from Refs. [12,17,20,23]. As shown in Figs. 6(a-d), we find that when the point correspondence reaches 50 pairs, the change of distortion coefficient tends to be stable.

 

Fig. 6. Comparisons of results for proposed algorithms and references [12,17,20,23], (a-d) show absolute errors of four distortion coefficient, namely, ${k_1},{k_2},{k_3},{k_4}$, respectively.

Download Full Size | PDF

For each noise level, 100 independent experiments were performed to determine the absolute average errors of the five intrinsic parameters, namely, ${f_u},{f_v},s,{u_0},{v_0}$, and analyze the variations in these parameters. After adding noise, we compared the variation from our proposed three algorithms with the results from Refs. [12,17,20,23]. Under different noise levels, the variations in the absolute errors of are shown in Figs. 7(a-e), wherein we can see that where $\sigma$ was zero, the absolute errors of the intrinsic parameters of the camera were zero. With an increase in noise level, the absolute errors of the intrinsic parameters increased linearly. At the same noise level, the absolute errors of our proposed algorithms are lower than that of Deng [12], Ying [17], Li [20] and Duan [23], and the absolute error of Ying [17] was slightly larger than that of other algorithms, because Ying’s algorithm [17] was nonlinear. The absolute error variation trend of the three calibration algorithms presented in this study was mostly the same, showing that our algorithms are effective in a certain noise range.

 

Fig. 7. Comparisons of results for proposed algorithms and references [12,17,20,23], (a-e) show absolute errors of five intrinsic parameters, namely, ${f_u},{f_v},s,{u_0},{v_0}$, respectively.

Download Full Size | PDF

4.2 Real data

Next, we used a real central catadioptric camera consisting of a conventional camera and a paraboloid mirror; the effective focal length of the camera was 16 mm; its imaging range was 300 − 400 mm; and the image resolution was 1300 × 1170 pixels. The calibration object was a yellow table-tennis ball with a diameter of 40 mm, which was placed on a checkerboard with a $10 \times 10$ grid. The distance between two adjacent characteristic points on the checkerboard was 11 mm, and the accuracy was 0.1 mm. The ball was placed in different positions on the checkerboard and photographed with the camera, as shown in Figs. 8(a)–8(f). Figures 8(a)–8(c) were selected as the calibration images.

 

Fig. 8. (a-f) Six images of a table tennis ball taken at different positions on checkerboard.

Download Full Size | PDF

First, Canny edge detection [27] was performed for each image in Figs. 8(a)–8(c); the results are shown in Figs. 9(a)–9(c). Next the projection of the sphere image and mirror contour in each image was obtained using the least-squares method [29]. Finally, the initial parameters of the paracatadioptric camera were calculated [17], and the equations for the sphere images and antipodal sphere images were optimized [23].

 

Fig. 9. (a-c) Canny edge detection for images in Figs. 8(a-c), respectively.

Download Full Size | PDF

After the above-mentioned operations, considering the lens distortion, the initial intrinsic parameters of the camera were obtained using the proposed methods and Refs. [12,17,20,23]; the rotation matrix ${\boldsymbol R}$ and the translation vector ${\boldsymbol t}$ were obtained by using the method described in Ref. [20]. Assume we know the space position of the mirror contour, The projection of the mirror contour can be generated by camera parameters, and we can match the measurement point and the ideal point [28]. In the simulation experiment, we find that when the matching point reaches 50 pairs, the variation of the distortion coefficient tends to be stable, so we use 50 pairs of points to calculate the distortion coefficient. The optimized distortion coefficient and the internal parameters can be obtained using Eq. (8). The average values were taken after 100 independent experiments. The distortion coefficients and calibration results are shown in Tables 2 and 3, respectively. The original resolution of Fig. 8 was 3197 × 3078 pixels in this study, which was demodulated to 1300 × 1170 pixels; while the original resolution in Ref. [20] was 1304 × 1172 pixels, and the results exhibit clear differences. The error is small and within the acceptable range. It can be seen that the calibration results obtained using the three methods used in this study were almost identical and were very similar to those of Refs. [12,17,20,23]; this confirms that the three methods used in this study were suitable within a certain margin of error.

Table 2. Lens distortion coefficient.

View Table | View all tables in this article

Table 3. Calibration results for Methods 1, 2, and 3 and references [12,17,20,23] (unit: pixels).

View Table | View all tables in this article

The camera extrinsic parameters were obtained based on the results listed in Table 3 and the calibration algorithm in Ref. [30]. A reprojection error analysis [24] and 3D reconstruction of the feature points on the checkerboard were performed, and we verified the parallelity and orthogonality of the reconstruction information. Initially, Harris corner detection [31] was performed on each image in Figs. 9(a)–9(c) to obtain the feature points on the checkerboard. However, only 95 feature points could be detected owing to the occlusion of the ball on the checkerboard. The red points indicate those detected feature points, as shown in Figs. 10(a)–10(c). Then, we calculated the reprojection errors of the feature points of the three images in Figs. 10(a)–10(c) based on the intrinsic parameters in Table 3. Starting with 9 points in the upper left corner $3 \times 3$ of 95 feature points, according to the rule of adding one row and one column each time, the change of reprojection error with the increase of the number of rows of feature points is shown in Fig. 11(a). Wherein we can see that the reprojection errors of our methods were almost close to each other and distributed around 0-0.24 pixels in the end, while the results of Refs. [12,17,20,23] were approximately 0 − 0.32 pixels, and the reprojection error of the three calibration algorithms fluctuates less with the increase of the number of rows of feature points than the results of Refs. [12,17,20,23]. Similarly, on the basis of the three images in Figs. 8(a)–8(c), according to the rule of adding one image each time in Figs. 8(d)–8(f), we also performed the same operation, as shown in Fig. 11(b).

 

Fig. 10. (a-c) Harris corner detection for images in Figs. 9(a-c), respectively.

Download Full Size | PDF

 

Fig. 11. Comparisons of results for proposed algorithms and references [12,17,20,23], (a), (b)show the reprojection error with the increase of the number of rows of feature points and the number of images, respectively

Download Full Size | PDF

We performed 3D reconstruction on feature points on the checkerboard grid in Figs. 10(a)–10(c), and calculated the parallelity and orthogonality of the feature points on the checkerboard; the angles between any two lines in the parallel and orthogonal directions were calculated to obtain the average value, respectively. During the tests, the angles between any two lines in the parallel and orthogonal directions of the checkerboard were 0° and 90°, respectively. We calculated parallelity and orthogonality, starting with 9 points in the upper left corner $3 \times 3$ of 95 feature points, according to the rule of adding one row and one column each time. The change of absolute error of parallelity and orthogonality with the increase of the number of rows of feature points is shown in Figs. 12(a) and 12(b). We can see that the included angles between any two lines in the parallel and orthogonal directions of the checkerboard after 3D reconstruction become smaller and smaller as the number of rows of feature points increases. The three algorithms have small absolute errors and stable fluctuations, and the 3D reconstruction effect is more stable. Similarly, on the basis of the three images in Figs. 8(a)–8(c), according to the rule of adding one image each time in Figs. 8(d)–8(f), we also performed the same operation, as shown in Figs. 12(c) and 12(d), respectively.

 

Fig. 12. Comparisons of results for proposed algorithms and references [12,17,20,23], show the absolute error of (a), (c) parallelity and (b), (d) orthogonality with the increase of the number of rows of feature points and with the increase of the number of images, respectively.

Download Full Size | PDF

5. Conclusions

In this study, we proposed novel algorithms to calibrate paracatadioptric cameras based on a sphere image and the pole-polar relationship. Under the unit viewing sphere model, a sphere in space is projected onto a pair of antipodal circles, which have two pairs of conjugate complex intersection points, with one point being at infinity. This allows the symmetry axis to be determined. Another point at infinity can be determined using the property of the antipodal points, so the line at infinity can be determined by connecting the two points at infinity. One point at infinity and another point at infinity in the direction of the polar line with respect to a circle is a pair of points at infinity in orthogonal directions. Based on the property of affine invariance under projective transformation, the vanishing points, image of the symmetry axis, and vanishing line can be determined. Finally, the calibration of a paracatadioptric camera was performed based on the pole-polar relationship, the three vanishing points orthogonal to each other, and the IAC. We evaluated the feasibility and effectiveness of the proposed algorithms through several simulations and real-world tests.

Reference [16] used a line for the calibration, and it is unnecessary to consider the length, position, and other information about the line. Method 1, Method 2, Method 3, and Refs. [19,20] all used a sphere as the calibrating object, without any measurement information, and only used the sphere image contour for calibration, with high accuracy. Method 1 and Method 2 carried out a calibration using the pole-polar relationship of IAC. Method 3, on the basis of Method 2, requires only two images to perform calibration using the tri-orthogonal vanishing points; thus, the noise had a lower impact on the calibration. References [16] and [20] completed calibration using the image of the imaged circle points, and in Ref. [19], the calibration was completed using the orthogonal vanishing point. However, two sphere images cannot be completely extracted owing to mutual occlusion impacting the calibration accuracy. In these three methods, three images were generated to complete the calibration.