Open Access
Add to BibSonomy
Abstract
In order to expand the field of view and measuremenst range, the camera is often mounted on a two-axis turntable to perform various visual tasks. And the calibration of the position and attitude relationship between the mounted camera and the two-axis turntable is a prerequisite for visual measurement. The turntable is considered an ideal orthogonal two-axis turntable in conventional methods. However, the rotation axes of the actual two-axis turntable may be neither vertical nor intersecting, and the optical center of the mounted camera is not always located in the rotation center of the turntable even for orthogonal two-axis turntables. The quite difference between the actual physical model of the two-axis turntable and the ideal model can cause large errors. Therefore, what we believe to be a novel position and attitude calibration method between a non-orthogonal two-axis turntable and the mounted camera is proposed. This method describes the spatial hetero-planar lines relationship between the azimuth axis and pitch axis of the turntable accurately. By the geometric invariant characteristics of the mounted camera in motion, the axes of turntable are recovered and the base coordinate system is established, and the position and attitude of the camera are calibrated. Simulation and experiments verify the correctness and effectiveness of our proposed method.
© 2023 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
Visual measurement [1–13] and 3D reconstruction [14–23] through images are one of core problems in computer vision and photogrammetry. For large-scale visual measurement tasks, which are widely used in aerospace [24–28] and industry [29–34]. Because of the limited field of view, a fixed camera is often difficult to meet the requirements of large-scale observation. The measurement range of the camera can be extended by arranging multiple cameras to build a camera network [35–39]. However, multiple cameras need to be calibrated, and the economic cost is also large. Another way is to use a two-axis turntable and a rigid mounted camera to form an active vision system. And the turntable can rotate around the azimuth axis and pitch axis so that the camera can obtain a larger observation range. Then, together with feature detection and tracking technology, it can be flexibly applied to a variety of vision tasks. The calibration of the spatial relationship between the two-axis turntable’s rotation axes and the calibration of the position and attitude relationship between the turntable and the camera are the prerequisites for visual tasks.
The kinematics model of the two-axis turntable is divided into the orthogonal two-axis turntable and the non-orthogonal two-axis turntable. The azimuth axis and pitch axis in the orthogonal two-axis turntable intersect and are vertical to each other. And the motion of the turntable is pure rotation. The camera is mounted to ensure that the optical center of the camera is consistent with the rotation center of the turntable. There are two cases of the spatial relationship between the turntable and the camera. One case is that the turntable coordinate axes are collinear with the camera coordinate axes, that is, the optical axis of the camera is orthogonal to two rotation axes of the turntable, and the other two axes of the camera coordinate system are collinear with the two rotation axes of the turntable. As shown in Fig. 1(a), the azimuth axis ${a_{azi}}$ of the turntable is collinear with the axis ${y_w}$ of the turntable base coordinate system, and the ${y_c}$ axis of the camera. The pitch axis ${a_{pit}}$ of the turntable is collinear with the axis ${x_w}$ of the turntable base coordinate system and the axis ${x_c}$ of the camera. For the ideal model, there is no need for calibration and measurement can be directly carried out.
Fig. 1. Concentric model of the rotation center of orthogonal two-axis turntable and camera optical center.
Another case is that the rotation center of the turntable coincides with the optical center of the camera, but the coordinate axes of the turntable are not collinear with the coordinate axes of the camera as shown in Fig. 1(b). In this case, the base coordinate system of the turntable and the camera coordinate system only have a rotation relationship. To achieve above two situations, precise installation and adjustment are required to achieve, which puts forward extremely high requirements for the accuracy of the turntable itself and the installation accuracy of the camera. Yuan Yun et al. proposed a calibration method for the rotation relationship between the camera and the two-axis turntable for this approximate model which ignores translation between camera and turntable [40].
In other cases, the camera cannot meet the above installation requirements due to the limitations of installation and adjustment conditions, as shown in Fig. 2. In this case, there is a fixed rigid body transformation relationship between the camera and the turntable. Gu Guohua et al. transformed the rotation and translation transformation relationship of the camera into a pure rotation relationship of a two-axis turntable through a pair of coordinate system transformations and inverse transformations; Then, with the help of the reading from the turntable and the fixed transformation relationship from the camera to the turntable, the calibration of the spatial transformation relationship between the turntable and the camera is achieved [41].
Fig. 2. Non-concentric model of the rotation center of orthogonal two-axis turntable and camera optical center.
The orthogonal two-axis turntable is difficult to assemble and expensive. Repeated inspection and adjustment are often required in the production and use process, which inevitably increases the production and maintenance costs of equipment. In addition, the orthogonal state of the orthogonal 2-axis turntable when it leaves the factory may change in the process of transportation and long-term use. In order to ensure turntable accuracy, it must be periodically adjusted, resulting in increased maintenance costs. For the non-orthogonal two-axis turntable, the azimuth axis and pitch axis of the turntable are not required to intersect or be vertical. The design requirements, manufacturing difficulties, and manufacturing and maintenance costs of the turntable are all greatly reduced. However, the kinematic parameters of the turntable are not completely determined, the conventional hand eye calibration method [42–48] cannot be directly used to solve the position and attitude of the camera in the base system of the turntable. Wu Bin et al. proposed a non-orthogonal two-axis turntable calibration method that requires third-party auxiliary equipment for calibration, which increases conversion errors [49–51].
Based on the geometric invariants of the camera in the single axis motion, this paper proposes a method to calibrate the spatial relationship between the rotating axes of the non-orthogonal two-axis turntable and the position and attitude relationship of the camera. The orthogonal two-axis turntable is regarded as a special case of the non-orthogonal two-axis turntable, so this method is also suitable for the orthogonal two-axis turntable.
2. Model of the non-orthogonal two-axis turntable with a mounted camera
The non-orthogonal two-axis turntable is that the azimuth axis and pitch axis of the turntable are spatial hetero-plane lines in 3D space, which don’t need to be perpendicular or intersect at one point. This type of turntable has no requirements for orthogonality, which is convenient for processing and adjustment. By calibrating its non-orthogonal axes 3D relative position and attitude relationship, its kinematic model is established. Then the camera is fixedly mounted to the non-orthogonal axis turntable. The vision task can be carried out after the position and attitude between the non-orthogonal axis turntable and camera is calibrated. As shown in Fig. 3. the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ and the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ are spatial hetero-plane lines. There is a unique common vertical line ${\mathbf{l}_{\mathbf{o - }{\mathbf{T}_{\mathbf{off}}}}}$ for ${\mathbf{a}_{\mathbf{azi}}}$ and ${\mathbf{a}_{\mathbf{pit}}}$ in 3D space. The intersection point between ${\mathbf{l}_{\mathbf{o - }{\mathbf{T}_{\mathbf{off}}}}}$ and ${\mathbf{a}_{\mathbf{azi}}}$ is ${\mathbf{o}_\mathbf{w}}$. And the intersection point between ${\mathbf{l}_{\mathbf{o - }{\mathbf{T}_{\mathbf{off}}}}}$ and ${\mathbf{a}_{\mathbf{pit}}}$ is ${\mathbf{T}_{\mathbf{off}}}$.
Rotating around the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ causes the azimuth $\alpha $ change, and drives the camera and the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ to rotate around the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$. And rotating around the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ only causes the pitch $\beta $ change, and drives the camera to rotate around the azimuth axis ${\mathbf{a}_{\mathbf{pit}}}$.
The initial turntable coordinate system ${s_{w0}}\textrm{ - }{x_{w0}}{y_{w0}}{z_{w0}}$ is established in the initial attitude of the turntable (azimuth angle is 0, and pitch angle is 0). The origin is the point ${o_{w0}}$, the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ upward is the axis ${y_{w0}}$, the common vertical line is collinear with the axis ${z_{w0}}$, and the direction of axis ${z_{w0}}$ is from the point o to the point ${\mathbf{T}_{\mathbf{off}}}$. The axis ${x_{w0}}$ is determined by the right-hand rule. In the established initial turntable coordinate system ${s_{w0}}$, the 3D space relationship between the azimuth axis and the pitch axis can be accurately described. The translation between the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ and the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ is ${\mathbf{T}_{\mathbf{off}}}$, and the coordinate of ${\mathbf{T}_{\mathbf{off}}}$ in the initial turntable coordinate system is ${\left( {\begin{array}{ccc} 0&0&d \end{array}} \right)^T}$. Define the line ${\mathbf{a^{\prime}}_{\mathbf{pit}}}$ parallel to the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ through the origin ${o_{w0}}$, and the angle between line ${\mathbf{a^{\prime}}_{\mathbf{pit}}}$ and line ${\mathbf{a}_{\mathbf{pit}}}$ is $\theta $, that is, the angle between the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ and the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ is $\theta $.
When the pitch $\beta $ is 0 and the azimuth $\alpha $ changes, the initial coordinate system ${s_{w0}}$ of the turntable is actively rotated to obtain the actual coordinate system ${s_w}$ of the turntable, and the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ coincides with the axis ${y_w}$. When pitch $\beta $ is not 0, the actual coordinate system ${s_w}$ is obtained by active rotation around the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ of the initial coordinate system ${s_{w0}}$. At this time, the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ and the axis ${y_w}$ no longer coincide.
There is a fixed position and attitude relationship between the camera coordinate system ${s_{cam}}$ and the initial coordinate system ${s_{w0}}$ of the turntable
According to the principle of pinhole imaging, the corresponding image points are obtained by
When the azimuth is $\alpha $ and the pitch is $\beta $, the point ${\mathbf{P}_{{\mathbf{w}_\mathbf{0}}}} = {\left( {\begin{array}{cccc} {{x_{{w_0}}}}&{{y_{{w_0}}}}&{{z_{{w_0}}}}&1 \end{array}} \right)^T}$ of the initial coordinate system ${s_{w0}}$ of the turntable is transformed into the point ${\mathbf{P}_\mathbf{w}} = {\left( {\begin{array}{cccc} {{x_w}}&{{y_w}}&{{z_w}}&1 \end{array}} \right)^T}$ of the actual coordinate system of the turntable. The azimuth $\alpha $ indicates rotation around the coordinate axis ${a_y}$, which can be directly expressed by the 3d rotation matrix ${\mathbf{R}_{\mathbf{azi}}}$ which indicated rotating azimuth $\alpha $ around axis ${y_{w0}}$. The pitch $\beta $ indicates rotation around a non-axis line ${\mathbf{a}_{\mathbf{pit}}}$ in the initial coordinate system ${s_{w0}}$ of the turntable, and the kinematics of indication by the pitch $\beta $ is expressed as
The rigid body transformation of feature points in the turntable caused by azimuth $\alpha $ and pitch $\beta $ is
Among them, the intrinsic parameters $\mathbf{K}$ have been calibrated in advance and are known. ${\mathbf{R}_{\mathbf{azi}}}$ changes with the azimuth $\alpha $, and ${\mathbf{R}_{\mathbf{pit}}}$ changes with the pitch $\beta $. ${\mathbf{T}_{\mathbf{off}}},\theta ,{\mathbf{R}_{\mathbf{c}{\mathbf{w}_\mathbf{0}}}},{\mathbf{T}_{\mathbf{c}{\mathbf{w}_\mathbf{0}}}}$ are parameters to be evaluated.
3. Calibration algorithm
The calibration algorithm mainly recover the azimuth axis and pitch axis under the inherent geometric constraints of single axis rotation, respectively, when the turntable is at zero position (azimuth $\alpha $ is 0, pitch $\beta $ angle is 0); Then establish the base coordinate system of the non-orthogonal turntable; Thirdly, calculate the structure parameters of the turntable and the spatial transformation relationship between the turntable and the camera, and finally perform nonlinear optimization for refinement. The algorithm diagram is shown in Fig. 4.
3.1 Single axis motion decomposition
Any 3D Euclidean transformation matrix is equivalent to rotation around a screw axis plus translation along that screw axis, as shown in Fig. 5.
Figure 5(a) represents rotation around the rotation axis $\mathbf{a}$ and Fig. 5(b) shows translation ${\mathbf{t}_\parallel }$ along the direction of rotation axis $\mathbf{a}$. where $\mathbf{a}$ is the unit 3D direction vector of the rotation axis, i.e., $\mathbf{Ra} = \mathbf{a}$. $\mathbf{t}$ can be decomposed into the sum of vectors that are parallel and perpendicular to the direction of the rotation axis, i.e., $\mathbf{t} = {\mathbf{t}_\parallel } + {\mathbf{t}_ \bot }$. The direction of the line which connected o and s is perpendicular to the direction $\mathbf{a}$. Therefore, s is the closest point o on the rotation axis. In the same way, $s^{\prime}$ is the closest point $o^{\prime}$ on the rotation axis.
Especially, the following relationship is satisfied for any Euclidean transformation caused by single axis motion
At this point, $\mathbf{t} = {\mathbf{t}_ \bot },{\mathbf{t}_\parallel } = \mathbf{0}$, therefore $\mathbf{a} \bot \mathbf{t}$. The eigenvalues of eigenvalue decomposition of the $4 \times 4$ Euclidean transformation matrix are composed of two complex conjugate eigenvalues and two equal real eigenvalues. Eigenvectors that correspond to the two real eigenvalues differ. And two real eigenvectors corresponding to the real eigenvalues define a line with two fixed points, which is the rotation axis.
3.2 Recover axes of the turntable in the camera coordinate system
In the case of the initial position of the turntable, that is, the azimuth angle is 0 and the pitch angle is 0, the first image of the planar target is taken as a reference image ${I_{azi - 0}}$, The point on the target is ${\mathbf{p}_{\mathbf{tar}}}$, and the corresponding image point is ${\mathbf{P}_{\mathbf{azi\_0}}}$; When the azimuth angle of the turntable is $\alpha $ and the pitch angle is 0, the second image ${I_{azi - 1}}$ of the plane target is taken, and the corresponding image point is ${\mathbf{P}_{\mathbf{azi\_1}}}$. Then the following relationship is satisfied:
The rigid body transformation relationship between the position of the camera that captured the first image ${I_{azi - 0}}$ and the position of the camera where the second image ${I_{azi - 1}}$ taken is
The rigid body transformation matrix ${\mathbf{H}_{\mathbf{azi\_1\_0}}}$ describes the single axis motion around the azimuth axis. Then the eigenvalue decomposition to matrix ${\mathbf{H}_{\mathbf{azi\_1\_0}}}$ is conducted. There are two eigenvectors ${\mathbf{v}_{\mathbf{azi\_1}}},{\mathbf{v}_{\mathbf{azi\_2}}}$ corresponding to equal real eigenvalues, which are two points on the azimuth axis. one is a finite point ${\mathbf{v}_{\mathbf{azi\_non\_inf}}}$ and the other one is an infinity point ${\mathbf{v}_{\mathbf{azi\_inf}}}$. And in the eigenvalue decomposition of the rotation matrix ${\mathbf{R}_{\mathbf{azi\_1\_0}}}$, the eigenvector corresponding to the eigenvalue 1 is the direction of the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$.
Analogously, in the case of the initial position of the turntable, that is, the azimuth angle is 0 and the pitch angle is 0, the first image of the planar target is taken as a reference image ${I_{pit - 0}}$, The point on the target is ${\mathbf{p}_{\mathbf{tar}}}$, and the corresponding image point is ${\mathbf{P}_{\mathbf{pit\_0}}}$; When the azimuth angle of the turntable is 0 and the pitch angle is $\beta $, the second image ${I_{pit - 1}}$ of the plane target is taken, and the corresponding image point is ${\mathbf{P}_{\mathbf{pit\_1}}}$.
The following relationships are satisfied:
The rigid body transformation relationship between the position of the camera that captured the first image ${I_{azi - 0}}$ and the position of the camera where the second image ${I_{azi - 1}}$ taken is
The rigid body transformation matrix ${\mathbf{H}_{\mathbf{pit\_1\_0}}}$ describes the single axis motion around the pitch axis. Then the eigenvalue decomposition of matrix ${\mathbf{H}_{\mathbf{pit\_1\_0}}}$ is conducted. There are two eigenvectors ${\mathbf{v}_{\mathbf{pit\_1}}},{\mathbf{v}_{\mathbf{pit\_2}}}$ corresponding to equal real eigenvalues, which are two points on the pitch axis. one is a finite point ${\mathbf{v}_{\mathbf{pit\_non\_inf}}}$ and the other one is an infinity point ${\mathbf{v}_{\mathbf{pit\_inf}}}$. And in the eigenvalue decomposition of the rotation matrix ${\mathbf{R}_{\mathbf{pit\_1\_0}}}$, the eigenvector corresponding to the eigenvalue 1 is the direction of the azimuth axis ${\mathbf{a}_{\mathbf{pit}}}$.
3.3 Solve the attitude parameters of the turntable and camera
When rotating around the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$, the orientation of the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ does not change in the initial turntable coordinate system ${s_{w0}}$, and its orientation in the camera coordinate system is
Analogously, when a single axis rotation is performed around the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$, the pitch axis does not change in the initial coordinate system of the turntable ${s_{w0}}$, the pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ is parallel to the plane ${o_w} - {x_w}{y_w}$, and the angle between pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ and the coordinate axis ${x_{{w_0}}}$ is $\theta $. Its direction is expressed as ${\left( {\begin{array}{cccc} {\cos \theta }&{\sin \theta }&0&0 \end{array}} \right)^T}$. The direction in the camera coordinate system ${s_{cam}}$ is
The angle between the azimuth axis ${\mathbf{a}_{\mathbf{azi}}}$ and pitch axis ${\mathbf{a}_{\mathbf{pit}}}$ is $\frac{\pi }{2}\textrm{ - }\theta $. Thus, it obtained by
Therefore, it is available
Substitution Eq. (14), and then
Due to the orthogonal nature of the rotation matrix, ${\mathbf{r}_3} = {\mathbf{r}_1} \times {\mathbf{r}_\mathbf{2}}$. Then ${\mathbf{R}_{\mathbf{c}{\mathbf{w}_\mathbf{0}}}}\textrm{ = }\left[ {\begin{array}{ccc} {{\mathbf{r}_\mathbf{1}}}&{{\mathbf{r}_\mathbf{2}}}&{{\mathbf{r}_\mathbf{3}}} \end{array}} \right]$.
3.4 Solve the translation parameters of the turntable and camera
The line on which the azimuth axis and pitch axis are located in the camera coordinate system can be expressed as parametric equations
Building simultaneous equation group by Eqs. (18), (19) find the common vertical foot ${\mathbf{p}_{\mathbf{azi\_ver}}}$ on the azimuth axis and the public vertical foot ${\mathbf{p}_{\mathbf{pit\_ver}}}$ on the pitch axis of two spatial hetero-plane lines.
The azimuth axis and pitch axis are spatial hetero-plane lines, and according to the definition of the turntable initial coordinate system ${s_{w0}}$, the distance between the common vertical feet of the azimuth axis and pitch axis is ${\mathbf{T}_{\mathbf{off}}}$, that is
According to the definition of the initial coordinate system of the turntable and the camera coordinate system, the common vertical foot ${\mathbf{p}_{\mathbf{azi\_ver}}}$ on the azimuth axis is the translation parameter of the initial coordinate system of the turntable in the camera coordinate system, that is
3.5 Nonlinear optimization
After the spatial relationship between rotating axes of turntable and the position and attitude between the base coordinate system of the turntable and the coordinate system of the camera is calibrated, the nonlinear optimization is adopted to refine. From Eq. (6) it can be obtained that the rigid body transformation relationship between the camera located in the initial turntable’s position and the camera located in the position of turntable’s azimuth $\alpha $ and pitch $\beta $ is
The back projection error between the initial position of the turntable and the position of azimuth $\alpha $ and pitch $\beta $ is defined as
4. Simulation and experiment
4.1 Simulation experiments
4.1.1 Orthogonal two-axis turntable simulation
The emulation of the position and attitude calibration method between the orthogonal two-axis turntable and camera is conducted. The camera parameters are set as ${f_x} = {f_y} = 2000$, ${u_0} = {v_0} = 1000$, and the skew factor is 0. The camera's rotation vector relative to the turntable is set as ${[ - 0.0167,0.0290, - 2.6177]^T}$, and the translation vector is set as ${[30,20,10]^T}mm$. The two-axis turntable is set as an orthogonal model, that is, the model parameters of the two-axis turntable are set as: the angle between the azimuth axis and the pitch axis is $\theta = 0^\circ $, and the distance between the azimuth axis and the pitch axis is $d = 0cm$. Add Gaussian noise with a mean of 0 and different variances to the image. At different noise levels, the mean of 100 results is used as an estimated result. Compared with the ground truth, the simulation results of the estimation of the structure parameters of the turntable are shown in Fig. 6 and Fig. 7.
Fig. 6. The angle error between the azimuth axis and the pitch axis of the orthogonal two-axis turntable with noise changes.
Fig. 7. The distance error between the azimuth axis and the pitch axis of the orthogonal two-axis turntable with noise changes.
From Fig. 6 and Fig. 7, it can be seen that the structure parameters of the turntable such as the angle and distance between the azimuth axis and pitch axis basically increase linearly with the noise level. The method has good estimation results for the structure parameters of the orthogonal two-axis turntable.
The estimation results of the camera's position and attitude parameters in the base coordinate system of the turntable are shown in Figs. 8 and 9. The attitude estimation accuracy is evaluated in two ways, one of which evaluates the mean error of the Euler angle decomposed by the rotation matrix according to the order of $x,y,z$ axis rotation. The second is to convert the rotation matrix into an axis-angle representation. The angle error between the estimated direction of the rotation axis and the true direction of the rotation axis is calculated, and the error between the estimated angle of rotation around the rotation axis and the true angle of rotation around the rotation axis is calculated. The camera displacement parameter error is evaluated by the Euclidean distance between the estimated displacement parameter and the true value of the displacement parameter.
Fig. 9. Camera displacement parameter error of the orthogonal two-axis turntable with noise changes.
It can be seen from Fig. 8, as the noise increases, the evaluation indicators of the camera attitude parameters increase. Similarly, it can be seen from Fig. 9, as noise increases, the camera's displacement parameter error increases. This method has good estimation results for the camera position and attitude of the orthogonal two-axis turntable.
4.1.2 Non-orthogonal two-axis turntable simulation
The emulation of the position and attitude calibration method of the non-orthogonal turntable and camera is conducted. The camera parameters are set to ${f_x} = {f_y} = 2000$, ${u_0} = {v_0} = 1000$, and the skew factor is 0. The camera's rotation vector relative to the turntable is set as ${[ - 0.0167,0.0290, - 2.6177]^T}$ and the translation vector is set as ${[30,20,10]^T}mm$. The non-orthogonal model parameters of the two-axis turntable are set as: the angle between the azimuth axis and the pitch axis is $\theta = 5^\circ $ and the distance between the azimuth axis and the pitch axis is $d = 1cm$. Add Gaussian noise with a mean of 0 and different variances to the image. At different noise levels, the mean of the results of 100 results is used as an estimated result. Comparing them with the ground truth, the simulation results of the turntable structure parameters estimation are shown in Figs. 10 and 11.
Fig. 10. The angle error between the azimuth axis and the pitch axis of the non-orthogonal two-axis turntable with noise changes.
Fig. 11. The distance error between the azimuth axis and the pitch axis of the non-orthogonal two-axis turntable with noise changes.
From Fig. 10 and Fig. 11, it can be seen that the structure parameters of the turntable such as the angle and distance between the rotation axes basically increase linearly with the noise level. This method has good estimation results for the structure parameters of the non-orthogonal two-axis turntable.
The estimation results of the camera's attitude parameters in the base coordinate system of the turntable are shown in Fig. 12 and Fig. 13. The attitude estimation accuracy is evaluated in two ways, one of which evaluates the average error of the Euler angle decomposed by the rotation matrix according to the order of $x,y,z$ axis rotation; The second is to convert the rotation matrix into an axis-angle representation, the angle error between the estimated direction of the rotation axis and the true direction of the rotation axis is calculated, and the error between the estimated angle of rotation around the rotation axis and the true angle of rotation around the rotation axis is calculated. The camera displacement parameter error is evaluated by the Euclidean distance between the estimated displacement parameter and the true value of the displacement parameter.
Fig. 12. Camera attitude parameter error of the non-orthogonal two-axis turntable with noise changes.
Fig. 13. Camera displacement parameter error of the non-orthogonal two-axis turntable with noise changes.
It can be seen from Fig. 12, as the noise increases, the error evaluation indicators of the camera attitude parameters increase. Similarly, it can be seen from Fig. 13, as noise increases, the camera's displacement parameter error increases. This method has good estimation results for the camera attitude parameters of the non-orthogonal two-axis turntable.
Compared to section 4.1.1 and section 4.1.2, except for the different structure parameters of the turntable ($\theta \textrm{ = }0^\circ, d = 0cm$,vs. $\theta \textrm{ = }5^\circ, d = 1mm$), the intrinsic parameters of the camera and the spatial transformation relationship between the camera and the turntable are completely same. Comparing Fig. 6 and Fig. 10, Fig. 7 and Fig. 11, Fig. 8 and Fig. 12, Fig. 9 and Fig. 13 can lead to significant errors if the non-orthogonal turntable model is considered as an orthogonal turntable model for calculation.
4.2 Real experiments
4.2.1 Experiment 1
The two-axis turntable model is FLIR D48E in the experiment, and the resolution of azimuth is 0.003° and the resolution of pitch is 0.006°. The camera’s model is Daheng mer-1070-4u3 with the resolution 3840 × 2748. Experimental settings and planar target is shown in Fig. 14. Thirty images were taken for planar targets, and the camera intrinsic parameters were calibrated. The results are shown as Table 1.
Table 1. Camera intrinsic parameters
(1) Solving the azimuth axis
The image is taken at the initial position of the turntable, then the azimuth changes, and the pitch remains unchanged, then another image is taken. Adjust the position of the planar target and repeat the process to take five pairs of images. Images of the target are shown in Fig. 15. Images in the first row are taken located in the turntable’s initial position, and images in the second row are taken located where the azimuth is -16.0725° and the pitch angle is 0°.
Solve the azimuth axis in the camera coordinate system, and the parameter equation is
The image is taken at the initial position of the turntable. then the azimuth does not change, the pitch changes, and another image is taken. Adjust the position of the planar target and repeat the process to take five pairs of images. Images of the target are shown in Fig. 16. Images in the first row are taken located in the turntable’s initial position, and images in the second row are taken located where the azimuth is 0° and the pitch is -19.2840°.
The parameter equation for the pitch axis in the camera coordinate system is
Images are taken in the turntable’s initial position, and then other images are taken where azimuth is -12.8580° and pitch is -16.0700°. Adjust the planar target’s position and repeat the process to take five pairs of images. Images of the target are shown in Fig. 17. Images in the first row are taken located in the turntable’s initial position, and images in the second row are taken where the azimuth is -12.8580° and the pitch is -19.2840°.
Fig. 17. Partial target images of the initial position and the position with changed azimuth and pitch
(4) Calibration results
After solving the structure parameters of the turntable and the camera’s position and attitude in the base coordinate system of the turntable, the optimization is carried out. The structure parameters of the turntable are solved as the angle between azimuth axis and pitch axis $\theta = 0.045^\circ $, and the distance between the azimuth axis and the pitch axis $0.84mm$. The rotation vector corresponds to the rotation matrix of the camera coordinate system in the base coordinate system of the turntable is ${[ - 1.5696, - 0.0159,0.0214]^T}$, and the translation vector of the camera coordinate system in the base coordinate system of the turntable is ${[ - 0.8732,76.5020,70.3228]^T}(mm)$. A histogram of the back projection error is shown in Fig. 18. The horizontal axis is the range of the back projection error, and the vertical axis is the number of pixels that fall in a certain back projection error range. The mean of the back projection error is 0.22 pixels.
Using calibrated parameters, perform 3D reconstruction of the target, as shown in Fig. 19. The distance between adjacent corner points of the target is 7 mm, and the RMS error between the distance of the reconstructed target points and the true value is 0.068 mm. Verified the accuracy of the calibration results.
4.2.2 Experiment 2
The experiment used a two-axis turntable in the China Institute of Metrology, with a resolution of 0.0006° for azimuth and pitch, and the angle between the azimuth and pitch axes less than 0.001°. The camera adopts AVT GT1920 with a resolution of $1936 \times 1456$. The two-axis turntable and camera are shown in Fig. 20, and the planar target is shown in Fig. 21. Thirty images were taken for planar targets, and the camera intrinsic parameters were calibrated.The results are shown in Table 2.
Table 2. Camera intrinsic parameters
(1) Solve azimuth axis
Take an image at the initial position of the turntable, then change the azimuth and keep the pitch unchanged, and take another image. Adjust the position of the planar target and repeat the above process to capture five pairs of images. The images of some targets are shown in Fig. 22. Imaegs in the first row are the target images taken during the initial position of the turntable, while images in the second row are other images taken at azimuth 18 ° and pitch 0 °.
The parameter equation of the azimuth axis in the camera coordinate system is solved as
Take an image at the initial position of the turntable, then keep the azimuth unchanged and the pitch changes, and take another image. Adjust the position of the planar target and repeat the above process to capture five pairs of images. The images of some targets are shown in Fig. 23. Images in the first row are the target images taken during the initial position of the turntable, while images in the second row are other images taken at azimuth 0 ° and pitch -13 °.
The parameter equation of the pitch axis in the camera coordinate system is solved as
Take an image at the initial position of the turntable, and then take another image at an azimuth 16° and pitch 1 °. Adjust the position of the planar target and repeat the above process to capture five pairs of images. The images of some targets are shown in Fig. 24. Images in the first row are target images taken during the initial position of the turntable, while Images in the second row are other images taken when the turntable azimuth is 16 ° and the pitch is 11 °.
Fig. 24. Partial target images of the initial position and the position with changed azimuth and pitch.
(4) Calibration results
After solving the structure parameters of the turntable and the initial pose of the camera in the turntable base coordinate system, optimization and refinement are carried out. The structure parameters of the turntable are: the angle between the azimuth axis and the pitch axis is $\theta = 0.033^\circ $; and the distance between the azimuth axis and the pitch axis is $d = 0.024mm$. The rotation vector corresponding to the rotation matrix of the camera coordinate system in the turntable base coordinate system is ${[\textrm{ - }36.7670, - 375.5378,\textrm{ - }452.5768]^T}$, and the translation vector of the camera coordinate system in the turntable base coordinate system is ${[ - 0.873,76.502,70.323]^T}(mm)$. The histogram of the back projection error is shown in Fig. 25. The horizontal axis represents the range of backprojection errors, while the vertical axis represents the number of pixels that fall within a certain range of backprojection errors. The average backprojection error is 0.04 pixel.
Using calibrated parameters, perform 3D reconstruction of the target, as shown in Fig. 26. The distance between adjacent corner points of the target is 20 mm, and the RMS error between the reconstructed distance of the target points and the true value is 0.19 mm, verifying the accuracy of the calibration results.
5. Conclusion
For visual tasks with a large field of view, fixed monocular cameras often cannot meet the requirements, on the other hand, there is also a contradiction between the field of view and spatial resolution of a fixed camera. By mounting the camera on a two-axis turntable, this problem can be solved well. For an ideal two-axis turntable and camera model, the two rotation axes of the two-axis turntable are orthogonal to each other. More strictly, the optical center of the camera needs to coincide with the rotation center of the turntable. It proposes high demands on the assembling and adjusting of the turntable itself, as well as on the camera mounting and adjusting. In this paper, a position and attitude calibration method for the non-orthogonal two-axis turntable and camera is proposed. This method does not need to know the structure parameters of the turntable in advance, and the robot calibration and hand-eye calibration of the turntable can be completed at one time. Simulation and experiments verify the correctness and effectiveness of the proposed method.
Funding
National Natural Science Foundation of China (52127809, 52005028, 51625501).
Acknowledgments
This article is supported by the Key Laboratory of Precision Opto-mechatronics Technology, Ministry of Education, Beihang University, China
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. G. Cioffi, T. Cieslewski, and D. Scaramuzza, “Continuous-time vs. discrete-time vision-based SLAM: A comparative study,” IEEE Robot. Autom. Lett. 7(2), 2399–2406 (2022). [CrossRef]
2. Y. Cui, F. Zhou, Y. Wang, L. Liu, and H. Gao, “Precise calibration of binocular vision system used for vision measurement,” Opt. Express 22(8), 9134–9149 (2014). [CrossRef]
3. G. Du and P. Zhang, “Online robot calibration based on vision measurement,” Robotics and Computer-Integrated Manufacturing 29(6), 484–492 (2013). [CrossRef]
4. R. L. Greene, M.-L. Lu, M. S. Barim, X. Wang, M. Hayden, Y. H. Hu, and R. G. Radwin, “Estimating trunk angle kinematics during lifting using a computationally efficient computer vision method,” Human Factors 64(3), 482–498 (2022). [CrossRef]
5. J. Huang, W. Xu, W. Zhao, and H. Yuan, “An improved method for swing measurement based on monocular vision to the payload of overhead crane,” Transactions of the Institute of Measurement and Control 44(1), 50–59 (2022). [CrossRef]
6. Z. Jia, J. Yang, W. Liu, F. Wang, Y. Liu, L. Wang, C. Fan, and K. Zhao, “Improved camera calibration method based on perpendicularity compensation for binocular stereo vision measurement system,” Opt. Express 23(12), 15205–15223 (2015). [CrossRef]
7. T. Jin, H. Jia, J. Li, H. Xiang, D. Zhang, J. Chen, and S. Zhuang, “Full-aperture diopter distribution measurement method for single vision and progressive addition lenses,” Opt. Lasers Eng. 151, 106883 (2022). [CrossRef]
8. G. Lee, S. Kim, S. Ahn, H.-K. Kim, and H. Yoon, “Vision-based cable displacement measurement using side view video,” Sensors 22(3), 962 (2022). [CrossRef]
9. L. M. Song, M. P. Wang, L. Lu, and H. J. Huan, “High precision camera calibration in vision measurement,” Opt. Laser Technol. 39(7), 1413–1420 (2007). [CrossRef]
10. B. F. Spencer Jr, V. Hoskere, and Y. Narazaki, “Advances in computer vision-based civil infrastructure inspection and monitoring,” Engineering 5(2), 199–222 (2019). [CrossRef]
11. L. Wu, Y. Su, Z. Chen, S. Chen, S. Cheng, and P. Lin, “Six-degree-of-freedom generalized displacements measurement based on binocular vision,” Struct Control Health Monit 27(2), e2458 (2020). [CrossRef]
12. L. Xing, W. Dai, and Y. Zhang, “Improving displacement measurement accuracy by compensating for camera motion and thermal effect on camera sensor,” Mechanical Systems and Signal Processing 167, 108525 (2022). [CrossRef]
13. Y. Xu and J. M. Brownjohn, “Review of machine-vision based methodologies for displacement measurement in civil structures,” J Civil Struct Health Monit 8(1), 91–110 (2018). [CrossRef]
14. L. Barazzetti, “Parametric as-built model generation of complex shapes from point clouds,” Advanced Engineering Informatics 30(3), 298–311 (2016). [CrossRef]
15. T. Czerniawski and F. Leite, “Automated digital modeling of existing buildings: A review of visual object recognition methods,” Automation in Construction 113, 103131 (2020). [CrossRef]
16. H. Ham, J. Wesley, and H. Hendra, “Computer vision based 3D reconstruction: A review,” Int. J. Elect. Comput. Eng. 9, 2394 (2019). [CrossRef]
17. M. Huang, J. Ninić, and Q. Zhang, “BIM, machine learning and computer vision techniques in underground construction: Current status and future perspectives,” Tunnelling and Underground Space Technology 108, 103677 (2021). [CrossRef]
18. L. Li, J. Chen, X. Su, and A. Nawaz, “Advanced-Technological UAVs-Based Enhanced Reconstruction of Edges for Building Models,” Buildings 12(8), 1248 (2022). [CrossRef]
19. Q. Lu, L. Chen, S. Li, and M. Pitt, “Semi-automatic geometric digital twinning for existing buildings based on images and CAD drawings,” Automation in Construction 115, 103183 (2020). [CrossRef]
20. G. Pintore, C. Mura, F. Ganovelli, L. Fuentes-Perez, R. Pajarola, and E. Gobbetti, “State-of-the-art in Automatic 3D Reconstruction of Structured Indoor Environments,” in Computer Graphics Forum (Wiley Online Library, 2020), pp. 667–699.
21. Z. Pučko, N. Šuman, and D. Rebolj, “Automated continuous construction progress monitoring using multiple workplace real time 3D scans,” Advanced Engineering Informatics 38, 27–40 (2018). [CrossRef]
22. X. Tian, R. Liu, Z. Wang, and J. Ma, “High quality 3D reconstruction based on fusion of polarization imaging and binocular stereo vision,” Information Fusion 77, 19–28 (2022). [CrossRef]
23. W. Yifan, C. Doersch, R. Arandjelović, J. Carreira, and A. Zisserman, “Input-level Inductive Biases for 3D Reconstruction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2022), pp. 6176–6186.
24. T. Jiang, H. Cui, and X. Cheng, “A calibration strategy for vision-guided robot assembly system of large cabin,” Measurement 163, 107991 (2020). [CrossRef]
25. T. Phisannupawong, P. Kamsing, P. Torteeka, S. Channumsin, U. Sawangwit, W. Hematulin, T. Jarawan, T. Somjit, S. Yooyen, and D. Delahaye, “Vision-based spacecraft pose estimation via a deep convolutional neural network for noncooperative docking operations,” Aerospace 7(9), 126 (2020). [CrossRef]
26. A. R. Vetrella, G. Fasano, and D. Accardo, “Attitude estimation for cooperating UAVs based on tight integration of GNSS and vision measurements,” Aerosp. Sci. Technol. 84, 966–979 (2019). [CrossRef]
27. Y. Wang, X. Wang, Y. Shan, and N. Cui, “Quantized genetic resampling particle filtering for vision-based ground moving target tracking,” Aerosp. Sci. Technol. 103, 105925 (2020). [CrossRef]
28. Z. Zhang, Y. Cao, M. Ding, L. Zhuang, and J. Tao, “Monocular vision based obstacle avoidance trajectory planning for Unmanned Aerial Vehicle,” Aerosp. Sci. Technol. 106, 106199 (2020). [CrossRef]
29. A. Ahmed and H. Abdalla, “The role of innovation process in crafting the vision of the future,” Comput. Industrial Eng. 37(1-2), 421–424 (1999). [CrossRef]
30. H. Alzarok, S. Fletcher, and A. P. Longstaff, “Survey of the current practices and challenges for vision systems in industrial robotic grasping and assembly applications,” Adv. Industrial Eng. Management 9, 19–30 (2020). [CrossRef]
31. E. Asoudegi and Z. Pan, “Computer vision for quality control in automated manufacturing systems,” Comput. Industrial Eng. 21(1-4), 141–145 (1991). [CrossRef]
32. M. MassirisFernández, JÁ Fernández, J. M. Bajo, and C. A. Delrieux, “Ergonomic risk assessment based on computer vision and machine learning,” Comput. Industrial Eng. 149, 106816 (2020). [CrossRef]
33. S. Mehta, R. A. Singh, Y. Mohata, and M. Kiran, “Measurement and analysis of tool wear using vision system,” in 2019 IEEE 6th International Conference on Industrial Engineering and Applications (ICIEA) (IEEE, 2019), pp. 45–49.
34. L. Xinmin, L. Zhongqin, and Z. Ziping, “A study of a reverse engineering system based on vision sensor for free-form surfaces,” Comput. Industrial Eng. 40(3), 215–227 (2001). [CrossRef]
35. Z.-C. Chao, Q.-F. Yu, S.-H. Fu, and G.-W. Jiang, “Adjustment-based data fusion method in the pose relay videometric using camera network,” in International Symposium on Photoelectronic Detection and Imaging 2011: Advances in Imaging Detectors and Applications (SPIE, 2011), pp. 91–98.
36. G. Jiang, S. Fu, Z. Chao, and Q. Yu, “Pose-relay videometrics based ship deformation measurement system and sea trials,” Chin. Sci. Bull. 56(1), 113–118 (2011). [CrossRef]
37. Q. Yu, G. Jiang, Z. Chao, S. Fu, Y. Shang, and X. Yang, “Deformation monitoring system of tunnel rocks with innovative broken-ray videometrics,” in ICEM 2008: International Conference on Experimental Mechanics 2008 (SPIE, 2009), pp. 572–577.
38. Q. Yu, G. Jiang, S. Fu, Z. Chao, Y. Shang, and X. Sun, “Fold-ray videometrics method for the deformation measurement of nonintervisible large structures,” Appl. Opt. 48(24), 4683–4687 (2009). [CrossRef]
39. Q. Yu, G. Jiang, S. Fu, Y. Shang, Y. Liang, and L. Lib, “Broken-ray videometric method and system for measuring the three-dimensional position and pose of the non-intervisible object,” The Int. Arch. Photogrammetry, Remote Sensing and Spatial Inform. Sci. 37, 145–148 (2008).
40. Y. Yuan, Z. Zhu, and X. Zhang, “Imaging model and high-precision calibration of quasi-concentric general theodolite-camera,” Acta Opt. Sin. 32(7), 0715003 (2012). [CrossRef]
41. W. J. Gugh, Q. Chen, and W. Qian, “Camera parameter calibration based on two-dimensional rotating platform,” J. Opt. Precision Eng. 25, 1890–1899 (2017). [CrossRef]
42. K. Daniilidis, “Hand-eye calibration using dual quaternions,” The International Journal of Robotics Research 18(3), 286–298 (1999). [CrossRef]
43. R. Horaud and F. Dornaika, “Hand-eye calibration,” The international journal of robotics research 14(3), 195–210 (1995). [CrossRef]
44. J. Jiang, X. Luo, Q. Luo, L. Qiao, and M. Li, “An overview of hand-eye calibration,” Int. J. Adv. Manuf. Technol. 119(1-2), 77–97 (2022). [CrossRef]
45. W. Lin, P. Liang, G. Luo, Z. Zhao, and C. Zhang, “Research of Online Hand–Eye Calibration Method Based on ChArUco Board,” Sensors 22(10), 3805 (2022). [CrossRef]
46. K. H. Strobl and G. Hirzinger, “Optimal hand-eye calibration,” in 2006 IEEE/RSJ international conference on intelligent robots and systems (IEEE, 2006), pp. 4647–4653.
47. R. Y. Tsai and R. K. Lenz, “A new technique for fully autonomous and efficient 3 d robotics hand/eye calibration,” IEEE Trans. Robot. Automat. 5(3), 345–358 (1989). [CrossRef]
48. W. Wang, Y. Liu, H. Song, and Z. Du, “A novel singular-free solution based on principle of transfer theory for the hand-eye calibration problem,” Mechanism and Machine Theory 170, 104723 (2022). [CrossRef]
49. W. Bin, Z. Yu, W. Zhansheng, P. Congcong, and H. Rongfang, “Coordinate Measurement Technology of Laser Total Station with Non-Orthogonal Shafting,”.
50. B. Wu, F. Yang, W. Ding, and T. Xue, “A novel calibration method for non-orthogonal shaft laser theodolite measurement system,” Rev. Sci. Instrum. 87(3), 035102 (2016). [CrossRef]
51. F. Yang, B. Wu, T. Xue, M. F. Ahmed, and J. Huang, “A cost-effective non-orthogonal 3D measurement system,” Measurement 128, 264–270 (2018). [CrossRef]
