Open Access
Add to BibSonomy
Abstract
Panoramic stereo imaging system can not only provide panoramic images to obtain more information, but also obtain the depth information of the target object, and it has broad application prospects in robot navigation, road planning, and obstacle avoidance. A new panoramic stereo imaging system combined with a panoramic annular lens (PAL) and a convex mirror is proposed. It provides a large panoramic vision up to 360°× (30°∼150°) and, to the best of our knowledge, the entire FOV is the largest in the existing literature. Meanwhile, the stereo field of view is 35°.
© 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The panoramic stereo imaging system can capture a large amount of image information with the advantages of a large field of view (FOV), and meanwhile, obtain the depth information of the object under the panoramic view. These imaging characteristics make the panoramic stereo system have extensive application prospects in machine vision, security monitoring, and autonomous driving [1].
Existing single-sensor panoramic imaging systems include fisheye lens system [2], catadioptric panoramic system [3], and panoramic annular lens system [4]. On this basis, in order to obtain the three-dimensional information under the panoramic view of the target, many researchers have proposed different panoramic stereo imaging system schemes. It was common to use two or more fisheye systems or other omnidirectional cameras to achieve stereo imaging [5,6]. These systems calculate the depth information of an object from the different imaging positions in the camera by arranging the cameras in a fixed horizontal or vertical orientation. To reduce the size and cost of the entire system, researchers have adopted single-sensor systems to achieve stereo imaging. A panoramic stereo imaging system was presented which used a single camera coaxially combined with a fisheye lens and a convex mirror [7]. The proposed system captured a panorama with a FOV of 360° (horizontal) by 125° (vertical). A novel panoramic stereo imaging system with double PAL was introduced [8], which contained two coaxial PAL units. However, it was bulky, and the FOV was relatively small (360°× (60°∼105°)). In the existing wide-angle lens, the maximum FOV can reach 280° [9]. In order to further simplify the system structure, a panoramic stereo system with single-viewpoint optics was designed [10]. The system used a single camera and multiple mirrors, and the vertical FOV was more than 16° for both upward and downward views.
To sum up, the panoramic stereo imaging system can be designed mainly by combining one or two single-sensor panoramic systems. Fish-eye lens is the most common optical system to achieve panoramic imaging. However, as the FOV increases, aberrations such as distortion and chromatic aberration are more difficult to correct, which increases the difficulty of design and the complexity of structure. The panoramic system with a single convex mirror could increase the rear FOV as the light from a large FOV is deflected into the subsequent lens, which also brings some disadvantages such as a complex mirror surface, severe distortion, and it has a central blind area. The PAL system can deflect rays by the two reflective surfaces from a large FOV so that the rays could enter the rear lens at a small angle, which reduces the difficulty of aberration correction. But it also has a central blind area, and it is difficult to expand the FOV. Combining the advantages and disadvantages of the above three panoramic systems, it is an important consideration in this paper to choose which two types of combinations are suitable to design a panoramic stereo imaging system.
In this work, a novel real-time panoramic stereo imaging system using a single sensor coaxially combined with a PAL and a convex mirror was proposed. Compared with other panoramic stereo systems, the proposed system has four obvious advantages. First, the two light paths can image clearly at the same time without adjusting the back focus. Second, the two optical paths are assembled as the same panoramic stereo imaging system. It is not required to fix the system position additionally. Third, because the imaging area of the convex mirror optical path is located in the blind area of the PAL system, it improves the utilization rate of the sensor. Forth, the convex aspheric surface is light in weight, simple in surface shape, greatly increases the rear field of view of the panoramic system, and improves the information capture ability of the entire system for the object space. The entire FOV is larger than existing wide-angle lenses [9]. Both light path parts have a large field of view detection capabilities, so the overlapping part of the two light paths’ field of view is used as the stereo imaging part. These advantages make the system play a vital role in guiding unmanned vehicles or robots to avoid roadblocks and road planning.
This paper is organized as follows. Section 2 presents the principle of the panoramic stereo imaging system with a PAL and a convex mirror. Section 3 introduces the lens design process. Section 4 shows the design results. Section 5 summarizes the panoramic stereo imaging system designed in this paper and looks forward to the future work.
2. Design principle of the stereo imaging system with a PAL and a convex mirror
2.1 Imaging principle
The proposed design consists of a PAL and a convex mirror as shown in Fig. 1. The PAL and the convex mirror are placed along the optical axis. In addition, the system contains two lens groups coaxially placed with the former. The lens group 1 functions to expand the front FOV of the PAL, thereby expanding the edge FOV of the convex mirror. The lens group 2 mainly corrects various complex aberrations caused by the front lens group, such as distortion, field curvature, and chromatic aberration. The two light paths emitted from the object point are displayed, and they are respectively imaged to different image point positions of the same sensor through a convex mirror and PAL. The red line indicates the optical path of the convex mirror, and the blue line represents the optical path of the PAL. The FOV of the system consists of two parts, i.e. the convex mirror part (with the vertical angle from ${\alpha _1}$ to ${\alpha _2}$), and the PAL part (with the vertical angle from ${\beta _1}$ to ${\beta _2}$). The two overlapping parts constitute the FOV of stereo imaging, which is from ${\gamma _1}$ to ${\gamma _2}$. The angle ${\gamma _1}$ is equal to the angle ${\alpha _1}$, and the angle ${\gamma _2}$ is equal to the angle ${\beta _2}$. Moreover, the whole FOV of the system is from ${\beta _1}$ to ${\alpha _2}$.
The PAL system follows the imaging principle of Flat Cylinder Perspective (FCP) [11], which can project a cylindrical FOV around the optical axis 360° into a two-dimensional annular image plane. As shown in Fig. 1, the PAL structure is a lens with two reflective surfaces and multiple refractive surfaces, which can reflect the optical path to achieve a 360° surround view imaging. The PAL system is composed of a PAL and relay lens group. PAL brings in the most distortion, field curvature, and chromatic aberration in the whole PAL system. To improve imaging quality, PAL usually takes the form of thick lenses, low dispersion, and small curvature. The aperture stop is usually placed on the last transmission surface of the PAL to suppress stray lights [12] and effectively correct chromatic aberration [13].
The catadioptric panoramic system consists of a mirror and refractive lens group. The refractive lens group has a small FOV and a simple structure. The field of view of the entire system is enlarged by the convex mirror. Therefore, the surface shape of the mirror is usually complicated and needs to be calculated.
As shown in Fig. 2, the image surface is rectangular, of which the long side is L and the short side is H. In order to image all the FOV of the two optical paths on the image surface, the short side of the sensor is used as the complete image height for lens design in general. The image surface is divided into three parts, i.e. the imaging area of the PAL unit (blue region in Fig. 2), the imaging area of the convex mirror unit (red region), and the non-imaging area (grey region), respectively. The gray area is divided into two parts, one is the central circular blind area, and the other is the image surface part outside the blue region. ${\textrm{r}_{{\alpha _1}}}$ and ${\textrm{r}_{{\alpha _2}}}$ are the outer and inner radius of the catadioptric optical path imaging annular region, while ${\textrm{r}_{{\beta _2}}}$ and ${\textrm{r}_{{\beta _1}}}$ are the outer and inner radius of the PAL optical path imaging annular region, respectively. The value of ${\textrm{r}_{{\alpha _2}}}$ is limited by the FOV ${\alpha _2}$ of the convex mirror, while the FOV ${\alpha _2}$ is limited by the PAL aperture. ${\textrm{r}_{{\alpha _1}}}$ and ${\textrm{r}_{{\beta _1}}}$ are the imaging circles of the FOV ${\alpha _1}$ of the convex mirror and the front FOV ${\beta _1}$ of the PAL, respectively. ${\beta _1}$ is affected by the outer diameter of the front reflecting surface of the PAL and the diameter of the convex mirror surface. Generally, ${\beta _1}$ has a smaller value than ${\alpha _1}$. The small FOV of the PAL is easier to design, which could increase the upper FOV of the entire system. The value of ${\alpha _1}$ is related to the aperture size of the convex mirror, the distance between the convex mirror and the PAL, and the front FOV of the convex mirror. ${\textrm{r}_{{\beta _2}}}$ is the imaging circle corresponding to the FOV ${\beta _2}$, which is related to the maximum aperture of the PAL. As large FOV optical systems, both of the two optical path parts follow the F-Theta distortion, although they have different FOV, different focal lengths, and different image heights. In order to avoid interference between the imaging areas of the two optical paths, the imaging area needs to satisfy the following conditions as Eq. (1).
2.2 Calculation principle of the reflective surface of the catadioptric panoramic system
The convex mirror of the catadioptric system can be parabolic, hyperboloid, or other free-form surfaces. And Baker S discussed several imaging characteristics of curved mirrors and concludes: The hyperboloid is configured to meet single-view imaging requirements and to obtain a large FOV [14,15]. In order to simplify the calculation of surface parameters, we use a hyperbolic mirror to design a catadioptric panoramic system.
As shown in Fig. 3, the midpoint of the two focal points ${F_1}$, ${F_2}$ of the hyperboloid is taken as the coordinate axis origin. The focal point ${F_2}$ of the hyperboloid coincides with the perspective center point C of the lens. Because the optical system is rotationally symmetric, the YOZ plane is selected for analysis here. Suppose the hyperboloid equation of the mirror is
where a, b represents the half of the long axis and the short axis of the hyperboloid, respectively, ${c^2} = {a^2} + {b^2}$, and 2c is the distance between the two focal points of the hyperboloid.The length of the short side of the sensor is d, L is the axial distance from the point C to the edge of the mirror, f is the focal length of the lens, and D is the aperture of the mirror. From the principle of similar triangles, we can get
The light of the maximum FOV emitted by the object point ${P_1}$ is reflected at the point ${Q_1}$ of the mirror. The coordinate of ${Q_1}$ is $\left( {\frac{D}{2},L - c} \right)$, which can be obtained by the hyperboloid equation:
The maximum FOV is $\textrm{ + }{\theta _{\max }}$, which can be expressed as
Combined with design indicators and actual needs, the appropriate mirror aperture D and baseline length L are given. As the focal length of the lens and the maximum FOV of this system are known, Eq. (2) ∼ (5) can be used to calculate parameters a and b of the catadioptric panoramic system. To characterize the hyperboloid in the design software ZEMAX, it is necessary to write the following formula,
3. Optical design
A 4K high-resolution CMOS image sensor was chosen, of which the resolution is 6252(H) 4176(V) with the pixel size of 3.76 μm. To avoid the time-consuming and error-prone problems caused by the multi-configuration in the design process, the PAL optical system is designed first, followed by the catadioptric system, and finally, the two are spliced into one sensor.
3.1 PAL unit
Using the short side of the sensor as the image height, the maximum vertical FOV of the PAL unit is 100°. The focal length can be calculated as −4.5mm from the F-Theta distortion. The F number is set to 3 to ensure the image surface light intensity. It is much harder to design by the method of aberration calculation because of the complex PAL structure [13]. Therefore, a 4K high-resolution PAL system previously designed in our laboratory was selected as the initial structure [16].
The structure of the PAL system is shown in Fig. 4. The PAL unit consists of 6 lenses. The PAL unit is a block lens with a circular reflective coating on the front center surface and a ring reflective coating on the rear surface. The FOV is 360°${\times} $ (30°∼100°), the maximum diameter is 70mm, the total length is about 86.3mm, and the focal length is −4.48mm. The system contains 5 simple even aspheric surfaces, and the order of aspheric coefficients are all less than 6.
Fig. 4. The structure of the PAL unit. Eight FOV are shown, i.e. 30°, 40°,50°, 60°, 70°, 80°, 89°, and 100° respectively.
According to the pixel size of the sensor, the Nyquist cutoff frequency can be calculated as 133lp/mm. As shown in Fig. 5, the MTF of all fields is high than 0.45 at 133lp/mm.
As shown in Fig. 6, the field curvature is less than ±0.05mm, and the F-Theta distortion is less than ±2%. The imaging quality of PAL is good and can be prepared for the convex mirror unit.
3.2 Convex mirror unit
The optical path of the convex mirror unit was designed on the basis of the PAL unit. The incident height of the light with the minimum FOV of the PAL system on the image surface is 2.33mm. It limited the size of the image height of the catadioptric system. The PAL system with the reflective surface removed is reversed, and the object height is set to 2.33mm. As shown in Fig. 7, the intermediate image surface is curved, the focal length of the reverse PAL unit is 14.23mm, and the exit angle of the system is 4.68°. The angle of the light emitted from the reverse PAL unit is very small, and the magnification of this system structure is small.
Fig. 7. The reverse PAL unit. Different colors of light represent the FOV with object heights of 0 mm, 1.5 mm, and 2.33 mm.
According to the above-mentioned refracting lens group, the mirror’s surface formula can be solved. In order to coordinate with the PAL optical path, the diameter D = 70mm of the mirror is given, and the baseline length L = 213.75mm is calculated. The focal length of the refractive lens group directly affects the baseline length of the convex mirror to the refractive lens. At this point, the baseline is too long, causing the overall system to be too large. In order to solve this problem, the focal length of the refractive lens is reduced in the case of ensuring the size of the image plane.
The reverse PAL unit has a larger field curvature and a small exit angle. Therefore, in order to design the convex mirror optical path and rationalize the overall structure, negative lenses need to be added at the intermediate image plane to expand the FOV and correct aberrations. Three negative lenses are added between the convex mirror and the PAL to expand the FOV before adding the convex mirror surface, as shown in Fig. 8. The focal length of the refractive system is 5.27mm. These three lenses all use low-dispersion materials to reduce the chromatic aberration of the system.
Fig. 8. The refractive lenses group of the convex mirror unit. Four FOVs are shown, i.e. 20°, 30°, 40°, and 50°, respectively.
There are three key points in the design of the optical path of the convex mirror. One is to match the half image height with the radius of the PAL circular blind area. The second is that the light from the edge FOV of the reflective surface should not be blocked by the PAL. The third is to ensure a good imaging quality to achieve simultaneous focus with the PAL optical path. The vertical FOV of the refracting lens group we designed is 20°∼50°.
The mirror parameters could be calculated according to the principle of the section 2. The high-order equation of the aspheric mirror surface is obtained by curve fitting. The specific parameters of the mirror surface are shown in Table 1.
Table 1. The specific parameters of the mirror surface.
In order to combine with the PAL optical path, the parameters of the three lenses between the convex mirror and the PAL unit are set as variables for further optimization. The structure of the convex mirror unit is shown in Fig. 9. The front surfaces of the first and third lenses are even-ordered aspherical surfaces to increase the freedom of optimization and simplify the system structure. The structure of the rear lens group is consistent with that of the PAL, and the PAL remains unchanged in the process of optimizing the convex mirror unit. The FOV is 360°${\times} $ (30°∼115°), the focal length is −1.27mm, and the total length is about 188mm. The diameter of the mirror is less than 102mm. The vertical FOV here is based on the horizontal plane perpendicular to the optical axis, which is 85° (forward 25°, backward 60°).
Fig. 9. The structure of the convex mirror unit. Five FOV are shown, i.e. 30°, 50°, 70°, 90°, and 115°, respectively.
As shown in Fig. 10, the MTF of all fields is high than 0.40 at 133lp/mm. As shown in Fig. 11, the field curvature is less than ±0.05mm, and the F-Theta distortion is less than ±10%. The existence of appropriate distortion can compensate for the relative illuminance [17], which is conducive to maintaining uniform illuminance between the edge FOV and the center FOV of the image and improving the recognition accuracy.
3.3 Tolerance analysis
Tolerance analysis plays an important role in the lens design process, which is related to the difficulty and cost of fabrication and alignment. The tolerance range is shown in Table 2. Diffraction MTF at the 133lp/mm of this system was selected as the evaluation criterion. The tolerance results of the two configurations are shown in Fig. 12. Within the tolerances given above, the final system will maintain an MTF higher than 0.1 in all FOV. The system achieves relatively low tolerance sensitivity. Therefore, the implementation of the system is feasible based on current fabrication and alignment.
Table 2. Tolerance values of the whole system
4. Discussion and results
The last step is to combine the PAL unit and the catadioptric system. Taking the horizontal line as the reference, the FOV of the catadioptric panoramic optical path can be expressed as ${\omega _1}$ and ${\omega _2}$, and the FOV of the PAL optical path can be expressed as ${\theta _1}$ and ${\theta _2}$, as shown in Fig. 13(a). Therefore, the entire FOV is ${\theta _1}$ +${\omega _2}$, the stereo imaging FOV is ${\omega _1}$ +${\theta _2}$. The image area is shown in Fig. 13(b). A and B are the imaging areas of the convex mirror optical path, and C and D are the imaging areas of the PAL optical path. Among them, region B and region D correspond to the same detection range, which can be regarded as the panoramic stereo imaging part. The grey region is the non-imaging area between the imaging areas of the two optical paths, which can avoid the problem of overlapping of region B and region C. The size of regions A, B, C, and D of the image plane is shown in Table 3.
Fig. 13. Schematic diagram of the whole system. (a) Schematic diagram of the FOV distribution, (b) Schematic diagram of the image distribution.
Table 3. Structural Parameters
The entire structural model is shown in Fig. 14. The green lines in Fig. 14(a) represent the convex mirror optical path, the blue lines represent the PAL optical path. In order to have a lightweight structure, and reduce the occlusion of the FOV, a truss rod mechanical structure [3,17–19] is adopted to support the convex reflective surface and the PAL mirror. The structural parameters of the entire system and the two optical paths are shown in Table 4.
Fig. 14. The entire structure model of the panoramic stereo imaging system combined with the PAL and the convex mirror. (a) Two-dimensional structure, (b) sectional view, (c) three-dimensional structure.
Table 4. Structural Parameters
5. Conclusion
A novel panoramic stereo imaging system combined with a panoramic annular lens (PAL) and a convex mirror is proposed. The same object point is imaged at different image plane positions through two optical paths to obtain the three-dimensional information of the system. The system has a large vertical FOV (30°∼150°) which is larger than existing wide-angle lenses, and a stereo imaging FOV (65°∼100°). This system can provide a solution for a variety of applications where both a large panoramic view and three-dimensional position information are required, such as the field of intelligent robots. In the near future, we will build the system and analyze the actual panoramic stereo imaging effect.
Funding
National Natural Science Foundation of China (61875173); Science Challenge Project (TZ2016006-0502-02).
Acknowledgments
We acknowledge Dr. Yiming Huang for the revision of the manuscript.
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. S. Thibault, “Panoramic lens applications revisited,” in Conference on Optical and Digital Image Processing, 2008).
2. L. R. Fan and L. J. Lu, “Design of a simple fisheye lens,” Appl. Opt. 58(19), 5311–5319 (2019). [CrossRef]
3. J. S. Chahl and M. V. Srinivasan, “Reflective surfaces for panoramic imaging,” Appl. Opt. 36(31), 8275–8285 (1997). [CrossRef]
4. T. Ma, J. C. Yu, P. Liang, and C. H. Wang, “Design of a freeform varifocal panoramic optical system with specified annular center of field of view,” Opt. Express 19(5), 3843–3853 (2011). [CrossRef]
5. S. Peleg, M. Ben-Ezra, and Y. Pritch, “Omnistereo: Panoramic stereo imaging,” IEEE Trans. Pattern Anal. Machine Intell. 23(3), 279–290 (2001). [CrossRef]
6. H. C. Huang and Y. P. Hung, “Panoramic stereo imaging system with automatic disparity warping and seaming,” Graphical Models Image Processing 60(3), 196–208 (1998). [CrossRef]
7. W. M. Li and Y. F. Li, “Single-camera panoramic stereo imaging system with a fisheye lens and a convex mirror,” Opt. Express 19(7), 5855–5867 (2011). [CrossRef]
8. Z. Huang, J. Bai, and X. Y. Hou, “Design of panoramic stereo imaging with single optical system,” Opt. Express 20(6), 6085–6096 (2012). [CrossRef]
9. “Datasheet of m12-fisheye lens product specification brief”, retrieved http://www.entaniya.cn.
10. G. Jang, S. Kim, and I. Kweon, “Single-camera panoramic stereo system with single-viewpoint optics,” Opt. Lett. 31(1), 41–43 (2006). [CrossRef]
11. R. Matthys and Donald, “Endoscopic measurement using radial metrology with digital correlation,” Opt. Eng. 30(10), 1455 (1991). [CrossRef]
12. Z. Huang, J. Bai, T. X. Lu, and X. Y. Hou, “Stray light analysis and suppression of panoramic annular lens,” Opt. Express 21(9), 10810–10820 (2013). [CrossRef]
13. K. Iizuka, Engineering optics (Springer, 2008), Vol. 35.
14. B. Simon and S. Nayar, “A theory of single-viewpoint catadioptric image formation,” Int J Comput Vis 35(2), 175–196 (1999). [CrossRef]
15. J. Zeng, X. Su, and G. Jin, “Panoramic stereo imaging using hyperboloid catadioptric cameras,” J. Optoelectronics . laser (2006).
16. J. Wang, X. Huang, J. Bai, K. W. Wang, and Y. X. Hua, “Design of high resolution panoramic annular lens system,” in AOPC 2019: Optical Sensing and Imaging Technology, (International Society for Optics and Photonics, 2019), 113382I.
17. K. Zhang, X. Zhong, L. Zhang, and T. Zhang, “Design of a panoramic annular lens with ultrawide angle and small blind area,” Appl. Opt. 59(19), 5737–5744 (2020). [CrossRef]
18. G.-I. Kweon, K. T. Kim, G.-H. Kim, and H.-S. Kim, “Folded catadioptric panoramic lens with an equidistance projection scheme,” Appl. Opt. 44(14), 2759–2767 (2005). [CrossRef]
19. X. Wang, X. Zhong, R. Zhu, F. Gao, and Z. Li, “Extremely wide-angle lens with transmissive and catadioptric integration,” Appl. Opt. 58(16), 4381–4389 (2019). [CrossRef]
