1. Introduction

The human eye is the main medium for acquiring external information, and its imaging is characterized by the non-uniformity of the density of photoreceptor cell arrangement in the retina [1], with only a high density of cone cells in the central concave region and a relatively low density of cone cells in other regions. In addition, the human eye also has a relatively large imaging field of view (FOV), with FOV angles up to 200° and 130° in the horizontal and vertical directions, respectively [2]. Based on the above characteristics, bionic human eye imaging is suitable for applications that require high FOV and local resolution at the same time, and therefore has a wide application prospect in the fields of target recognition and tracking, intelligent surveillance, and biomedicine [3].

There are a variety of devices to realize bionic human eye imaging, such as humanoid eye sensors with discrete distribution of cells based on sensor processing technology [4,5] and humanoid eye sensors based on curved lens arrays [6]. However, the structure of the above devices is complicated, and the existing manufacturing process level can hardly meet the requirements of practical applications. As a typical beam pointing adjustment device, Risley-prism has been widely used in the field of beam control, such as infrared countermeasures, laser communication, optical obstacle avoidance [7]. With the development of research on Risley-prism, researchers are no longer satisfied by using Risley-prism to control a single beam, but turn to consider how to use Risley-prism to control the camera FOV, and develop a bionic human eye imaging method based on Risley-prism imaging system (RPIS) to verify its ability in FOV extension, local high resolution, and flexible adjustment in high-resolution region [8].

Due to the deflection of the beam by the prism, the RPIS produces serious imaging distortion. In order to correct the imaging distortion of the RPIS, a lot of related research has been conducted and good results have been achieved. In 2007, Lavigne proposed a distortion correction method based on the first-order near-axis approximation for ray tracing, but the correction accuracy is not high [9]. On this basis, Zhou proposed the reverse ray tracing distortion correction method based on vector form of Snell’s law in 2015. The method has a higher computational accuracy [10].

At the same time, due to the different refractive ability of prisms for different wavelengths, the beam propagation direction of different wavelengths differs after passing through the prisms, resulting in the difference of spatial position of different wavelengths of light on the imaging focal plane of the camera, which generates imaging dispersion. In order to solve the dispersion problem, in the current mainstream RPIS, a filter is placed in front of the camera to only capture images of a single band channel, by sacrificing the amount of information to ensure the quality of imaging. However, the multiple channels imaging is vital for various applications, which is in line with the way that the human eye detects the world, and better provides more information in spectrum about detection scene to meet the requirements for detection accuracy and target recognition. Currently, the existing methods to eliminate imaging dispersion in RPIS are mainly based on optical system design. For example, in 2011, Florea proposed using chalcogenide glass in combination with other materials to design Risley-prism. Though these combinations reduced spectral dispersion, the design was not applied in the imaging field [11]. In 2022, Lee used the doublet structure of electrically wetted prisms to obtain refractive index conditions for conducting and non-conducting liquids to minimize dispersion, and meet the dispersion conditions by mixing two conducting liquids and two non-conducting liquids, and their designed dispersion prisms were found to be effective in eliminating dispersion [12]. However, this design is applied to prism arrays and has not been used in Risley-prism, and the optical system-based design for dispersion elimination has the drawbacks of bulky structure and high design difficulty. Besides, there are many digital image processing methods used to eliminate the dispersion. For example, in 2010, Chung used color lines on the chromatic aberration edge to correct both horizontal and vertical dispersion in the image [13]. In 2017, Sun utilized pixel correlation between color channels to achieve cross channel information recovery, thereby achieving dispersion elimination [14]. In 2022, Eboli estimated local Gaussian blur kernels for overlapping patches and sharpen them with a non-blind deblurring technique and remove the remaining lateral chromatic aberrations with a convolutional neural network [15]. But the above algorithms are mainly aimed at the dispersion caused by the camera itself, not designed for RPIS, and cannot solve the distortion problem that coexists with dispersion.

In addition, based on the simultaneous existence of image distortion and dispersion, which present a non-uniform distribution in space, achieving objective and accurate evaluation of the image dispersion is also an important aspect. The commonly used evaluation indexes for reference images are peak signal-to-noise ratio (PSNR), mean square error (MSE), structural similarity index (SSIM) [16], and feature similarity index (FSIM) [17]. These evaluation indexes are widely used in tasks such as image denoising, super-resolution reconstruction, etc. However, due to the deflection of light by the prism, the spatial position of the original imaging FOV of the camera is changed and the rectangular FOV boundary is deflected to a curved boundary with nonconvex characteristics, so it is unable to obtain a reference image with the same spatial position and FOV boundary characteristics as the deflected FOV of the RPIS. Therefore, reference image evaluation indexes are not applicable to the RPIS. In order to get rid of the dependence on the reference image, the evaluation indexes of reference-free images have been developed and established, and the following reference-free image evaluation indexes such as Blind Image Quality Index (BIQI) [18], Spatial-spectral Entropy Quality (SSEQ) [19] and Natural Image Quality Evaluator (NIQE) [20] are commonly used. However, the above evaluation indexes mainly reflect the image clarity and cannot reflect the spatial position difference between the three channels of the degraded image of the RPIS, and is difficult to ensure the objectivity of the evaluation of the degraded image. Furtherly, the above evaluation indexes fail to define the boundary of the upper and lower limits of the image dispersion, which cannot judge the enhancement effect of the dispersion correction method. Therefore, it is necessary to design an evaluation method for the degraded images of RPIS.

To address the above problems, this paper systematically analyzes the causes of dispersion formation and its characteristics based on the imaging model and beam propagation law of the RPIS. The change law of spatial position between different channels of the degraded image is demonstrated by using the ray tracing analysis method. A cooperative correction method for image distortion and dispersion is proposed, and multicolor and good-quality imaging is achieved by correcting the aberrations of each channel and superposition of multi channels. In addition, by analyzing the variation law of gray values between different channels of the image, an image dispersion evaluation index based on Pearson's correlation coefficient (PCC) is established, and the superiority of the proposed evaluation index is proved compared with other reference-free image evaluation indexes for RPIS. Besides, the selection of refractive index of each RGB channel and color stripes at the boundary of deflected FOV were also discussed. Furtherly, a RPIS based on RGB camera is built to verify the effectiveness of proposed method for various targets in indoor and outdoor scenes, and the comparative evaluation analysis was performed on the experimentally acquired images. The results show that the method achieves multi-channel detection imaging without increasing the complexity of the Risley-prism imaging optical path, which breaks the limitation on the waveband for existing RPIS and shows great application potential in the fields such as imaging reconnaissance, target tracking, and intelligent identification.

2. Imaging model of Risley-prism and dispersion analysis

This chapter will demonstrate the imaging model of a RPIS, and introduce basic ray tracing algorithms for Risley-prism and discuss the formation and properties of dispersion.

2.1 Imaging model of Risley-prism

As shown in Fig. 1, the system consists of a fixed RGB camera and a pair of rotating prisms. The prisms Π₁ and Π₂ can be independently rotated around the Z-axis and are in zero position when their main cross-section lies in the XOZ plane with the thick end pointing in the positive direction of the X-axis [21]. The rotation angle of both prisms is defined as the counterclockwise angle between their main cross sections and the positive direction of the X-axis, denoted by θ₁ and θ₂, respectively. The wedge angles and refractive indices of both prisms are equal. By changing the rotation angle, the imaging view axis (red dashed line) can be changed, which in turn changes the imaging FOV. The black dashed box represents the original FOV of the camera, and the area enclosed by the blue curve is calculated by the reverse ray tracing method and represents the FOV that changes in size and orientation after refraction by the prism, and this FOV is the deflected FOV. The orientation of the FOV after deflection by the prism is often expressed in terms of the altitude angle Φ and azimuth angle Θ of the visual axis.

 

Fig. 1. Framework of the Risley-prism imaging system (RPIS).

Download Full Size | PDF

2.2 Image dispersion and its properties

The nonlinear deflection of the prism to different wavelengths of light will lead to both distortion and dispersion. The current correction method for imaging distortion is mainly the reverse ray tracing correction method based on the law of vector form of Snell’s law, this method is based on the beam deflection model of a Risley-prism. Similarly, the causes of dispersion can also be directly analyzed through this model, so it is necessary to first discuss the propagation law of different beam of light in the RPIS. Figure 2 shows the red, green, blue light beam propagation process in the RPIS with the camera as the generation source, which is opposite to the actual imaging optical path.

 

Fig. 2. Schematic diagram of the different beam propagation process in the Risley-prism imaging system.

Download Full Size | PDF

The camera optical axis is considered as the positive direction of the Z-axis, and the Cartesian coordinate system is established with the midpoint of the camera lens surface as the origin. The four surfaces of the Risley-prism are labeled as 1∼4. The normal vectors of each surface of the prism are established and represented by the symbols N₁∼ N₄ according to the prism surface number in turn [22]. According to the pinhole imaging model, for a pixel point on the focal plane of the camera, the vector of light emitted from it as the light source can be expressed as ${(u_{int}^c,v_{int}^c,f)^T }$, $c \in \{ r,g,b\} $. After normalizing it, the direction vector of the camera's emitted vector is obtained as $V_0^c = {(x_0^c,y_0^c,z_0^c)^T }$. According to the refraction formula in vector form, the direction vector of the different light beam after refraction by the i-th prism surface is $V_i^c$ [23]:

The final outgoing vector $V_4^c = {(x_4^c,y_4^c,z_4^c)^T }$ of the RPIS can be calculated from the four refractions, and if the distance from the camera to the object plane is D, then the object plane coordinates P(x_p, y_p) corresponding to the imaging of the pixel point $(u_{int}^c,v_{int}^c)$ are:

Based on the above formula, we are able to analyze the properties of image dispersion and conduct a comparative study with pertinent experiments. To achieve this, a series of system parameters have been established. For example, the Risley-prism wedge angle is α₁=α₂= 11.35°, the prism material is H-K9 L optical glass, the camera resolution is 3072 × 2048, the equivalent focal length f is 5012pixel, and the object distance is taken L = f (The object distance is set to f here in order to project the FOV before and after deflection onto the same scale to compare their differences more intuitively).

To perform a detailed analysis of the properties of image dispersion, we initially acquired an unprocessed dispersion image with both prism angles set at 45°. The Fig. 3(a) displays the experimental results of spatial position differences of different channels. The region in the red box is chosen to analysis the imaging position in the same coordinate system. Both G and B channels have a certain offset relative to the R channel, it can also be found that the offset direction is basically collinear with the direction of the azimuth angle. In order to further prove the observed phenomenon, Fig. 3(b) presents the boundaries of the deflected FOV obtained by employing the inverse ray tracing method based on the original FOV. Origin FOV is the FOV of camera without prisms, while R-channel, G-channel and B-channel are simulated imaging results of three channels. It can be found that both the G and B channels exhibit noticeable deviations relative to the R channel, showcasing a consistent offset pattern that differs from the actual image. This phenomenon highlights the intriguing relationship between the offsets observed across different channels.

 

Fig. 3. (a) The experiment results of position offset of different channels in a dispersive image when θ₁=θ₂= 45°. (b) The simulation results of FOV position offset for different channels when θ₁=θ₂= 45°. (c) and (d), (e) and (f), (g) and (h) show the pseudo-color images that the offset of pixel points between channels of the degraded image when θ₁=θ₂= 45°, θ₁=θ₂= 135°, θ₁=θ₂= 180°. (c), (e), (g) is a pseudo-color image of the G channel relative to the R channel. (d), (f), (h) is a pseudo-color image of the B channel relative to the R channel.

Download Full Size | PDF

Based on the aforementioned methodology, we further computed the Euclidean distance between each pixel's spatial position in the G and B channels relative to the R channel. This distance served as a measure to quantify the displacement of pixels at different locations between degraded image channels. We generated the pseudo-color images to visualize these displacement levels. Figure 3(c) and (d) depict the pseudo-color images representing the displacement magnitudes of the G and B channels relative to the R channel, respectively, when both prisms were rotated at 45°. From the computed results, it can be observed that the maximum displacement of the G channel relative to the R channel was 9.09 pixels, while the minimum displacement was 7.21 pixels. On the other hand, the maximum displacement of the B channel relative to the R channel reached 21.12 pixels, with a minimum displacement of 16.77 pixels. Additionally, it is noteworthy that the displacement magnitude in the B channel was greater than that in the G channel.

In addition to Fig. 3(c) and Fig. 3(d), as shown in Fig. 3(e)–(h), we also plotted pseudo-color images of the G and B channels relative to the R channel when the both prisms rotation angles are 135° and 180°. It is not difficult to find from Fig. 3(c)–(h) that the displacement magnitude of pixels varied across different positions within the image, the displacement magnitude was higher in the edge regions compared to the central area, displaying an increasing trend from the center towards the periphery. Moreover, the direction of increasing displacement was found to be roughly aligned with the azimuthal angle. To illustrate this issue, we use the simplified first-order paraxial formula to calculate the azimuth angle [24]:

3. Cooperative correction method for distortion and dispersion based on RGB channels

The preceding analysis establishes a compelling rationale for the inherent law of channels within dispersion images, indicating a discernible displacement pattern. Guided by this pattern, a single channel from the original degraded image can be chosen as a reference, then offset the pixel points of other channels and align them with the reference channel, thereby achieving a vital dispersion correction. However, the original degraded image has distortion and dispersion at the same time [25], and the degree of dispersion is not the same at different positions in the original degraded image due to the nonlinear deflection of light by the prism. Therefore, it is not reasonable to directly offset the whole original degraded image to achieve dispersion correction. If the three channels of the degraded image are corrected separately for distortions, and then two of them are aligned to the third channel using the approximate translation method, although the dispersion can be eliminated, this method splits the two parts of distortion correction and dispersion correction [26].

To solve the above problems, a cooperative correction method for distortion and dispersion based on RGB channels is proposed. According to the distortion and dispersion characteristics of RPIS, taking one of the channels of the deflected FOV as the reference FOV, the reverse mapping algorithm is used to calculate the spatial position of the integer points $(x_p^r,y_p^r)$ inside the deflected FOV in different channels of the original degraded image and their grayscale information, and the grayscale information of different channels are reassigned to the integer points $(x_p^r,y_p^r)$ inside the deflected FOV, and then the obtained grayscale information of different channels is superimposed pixel by pixel, which achieves the simultaneous correction of aberration and dispersion (Fig. 4).

 

Fig. 4. Technical process for distortion and dispersion correction of RPIS.

Download Full Size | PDF

Due to the prism has different refraction ability for different wavelengths of light, the imaging quality of different channels of the original degraded image varies. Since the wavelength of red light is the largest compared to green and blue light and the refractive index of the prism for the red light is the smallest, the image of R channel is less deflected and have the highest image quality. The image shown in Fig. 3(a) can powerful prove the above results that the overall imaging quality of the R channel is better than that of the G and B channels in terms of both blurring and artifacts. Thus, it is reasonable to choose image of R channel as reference for the correction. In addition, the FOV of the G and B channels after reverse mapping does not completely cover the original camera FOV, and there is information loss in the two channels. Different from the G and B channels, the FOV of the R channel can completely cover the original camera FOV, as shown in Fig. 3(b). Therefore, using the deflected FOV of the R channel as the reference, the imaging area of the R channel can be preserved as much as possible to achieve the best image restoration. As shown in Fig. 5, the specific processes of distortion and dispersion correction based on the deflection FOV of the R channel in the object plane as follows: (1) Construct the normal vectors of each face of the Risley-prism, and calculate the position of deflected FOV boundary by the reverse ray tracing method according to the Risley-prism rotation angles and refractive index n_r based on the rectangular boundary of the original camera FOV; (2) Get the coordinates of all the integer points $(x_p^r,y_p^r)$ within the calculated closed deflected FOV boundary; (3) Construct the incident light vector again based on the integer points; (4) Calculate the non-integer point coordinates positions $(u_{dou}^r,v_{dou}^r)$ and grayscale values $f(u_{dou}^r,v_{dou}^r)$ in the origin FOV using the reverse mapping bilinear interpolation technique based on the reverse ray tracing method according to the refractive index n_r; (5) Fill the grayscale values into the corresponding deflected FOV pixel by pixel to achieve the distortion correction of R channel; (6) Replace refractive index n_r with n_g and n_b, and the perform distortion correction for the B and G channels respectively by repeating steps 3-5 using the integer point coordinates of the R channel derived in step 2; (7) Superimpose the distortion corrected images of R, G and B channels into one RGB image, and complete the dispersion correction.

 

Fig. 5. Flow chart of the proposed method for the simultaneous correction for distortion and dispersion for RPIS.

Download Full Size | PDF

Traditionally, degraded images with n channels would require establishing n deflected FOV. However, our proposed algorithm necessitates only the determination of a deflected FOV for degraded images with three channels. Leveraging this deflected FOV, distortion and dispersion correction can be effectively executed by separately processing the three channels of the degraded image, this obviates the need for additional post-processing procedures.

4. Evaluation methodology

In order to accurately characterize the effectiveness of the proposed dispersion correction method, it is necessary to evaluate objectively the dispersion characteristics of the images which are captured by RPIS and corrected by the above algorithm. As mentioned in the introduction section, reference image evaluation indexes are not applicable to the RPIS due to the failure to obtain a reference image with the same spatial position and FOV boundary characteristics as the deflected FOV of the RPIS. On the other hand, reference-free image evaluation indexes such as BIQI, SSEQ and NIQE are difficult to ensure the objectivity of the evaluation of the degraded image due to the uniqueness in imaging of RPIS.

In order to establish a dispersion evaluation method applicable to the RPIS, the spatial position relationship of the three channels of the image needs to be explored first. The gray value changes of the RGB channels in the horizontal direction for the non-degraded image and the degraded image are plotted as shown in Fig. 6. For the non-degraded image, the gray value change patterns of the RGB channels are basically the same, showing a high correlation. However, in the degraded image, the changing behavior of gray value for the RGB channels are less correlated, and the decreasing trend of the gray value of the B and G channel are noticeably ahead of the R channel in the junction region of different colors. A large number of experiments have been carried out and the similar phenomenon occurs. Therefore, we can conclude that the image dispersion phenomenon has a strong correlation with the gray value change law of the RGB channels, and so PCC is chosen to quantify the similarity of the gray value change trend between different channels [27], which is used as the evaluation criterion for the dispersion of RPIS.

 

Fig. 6. The variation pattern of RGB channels grayscale values in the horizontal direction of the image: (a) Reference image. (b) Degraded image.

Download Full Size | PDF

In terms of practical application of PCC in the dispersion evaluation, the R channel with the highest imaging quality is first considered as the reference channel. Secondly, the degree of deflection of blue light due to light refraction is greater than that of green light from both theoretically and experimentally. Therefore, the pcc for a row or column for the R-B channel in the image should be calculated, which is more representative in quantifying the dispersion of the degraded image. The calculation equation is:

In order to characterize the overall image dispersion, all the pcc in the horizontal and vertical directions of the whole image need to be calculated. The following problems will arise: (1) due to the prism characteristics, there is a part of black area in the edge of correction image, and pure black area would lead to an abnormal increase in the value of pcc. (2) The non-convex boundary of the deflected FOV would bring complexity for the calculation.

In our previous work, we have proven that the simplified FOV [28] can effectively replace the deflected FOV without obvious loss of valuable information of image. In terms of calculation of the index of pcc, the advantages of using the simplified FOV are as follows: (1) the simplified FOV has the similar rectangular boundary characteristics with the degraded image, and also similar data volumes in the horizontal and vertical directions, so the pcc of the degraded images and the corrected images is more comparable; (2) the determination of the simplified FOV range is simple and efficient, and can avoid the problem of increasing computational time costs caused by the black edge areas or non-convex boundary.

Based on the above discussion, the PCC of an image to be evaluated are defined as the average of the PCC_h and PCC_v that represent the average values of the pcc of all rows and columns of the image. The specific calculation equation is follows:

To further prove the capability of the established PCC index on the evaluation of dispersive image, degraded images with dispersion are obtained by RPIS. As shown in the Fig. 7, a standard checkerboard calibration board is chosen as the target, and rotation angles of the prism is set at different values to capture images with different dispersion. For the RPIS, the relative angle of the two prisms is negatively related to the distortion and dispersion of the target image. Therefore, the rotation angle of prism Π₁ is set to be 0° and prism Π₂ is set to be 0°, 30°, 60°, 90°, 120°, 150°, and 180°, respectively and Fig. 7(a)–(g) are local enlargements of the degraded images taken under the above different combinations of prism rotation angles. It is obvious to be found that, as the relative rotation angle of prism Π₁ and prism Π₂ increase, the color stripes gradually become narrower and the resolution of image in the figure become better, indicating that the dispersion and distortion of the images decreases with the increase of the relative rotation angle. Besides the index PCC, the several mainstream evaluation indexes such as SSEQ, NIQE, BIQI are also calculated to compare with the proposed PCC in the objectivity and accuracy of evaluation, and the results are shown in the Fig. 7(h)–(k). Due to the monotonically decrease of dispersion of the captured image with the increase of relative angle, the specific values of the corresponding evaluation indexes should also be monotonically increasing or decreasing. However, SSEQ, NIQE and BIQI do not show monotonicity for the degraded images with different degrees of dispersion, and only our established evaluation index is consistent with the principle of monotonicity, and increases with the decreasing dispersion and approaches 1, which proves the superiority of our index.

 

Fig. 7. Local amplification of dispersion images acquired with standard black-and-white checkerboard grids and different relative rotation angle: (a)-(g). Test results of different evaluation indexes for images with different degrees of dispersion: (h)PCC. (i)SSEQ. (j)BIQI. (k)NIQE.

Download Full Size | PDF

Therefore, our evaluation index can corroborate with the subjective visual perception, and can objectively and effectively reflect the dispersion degree of degraded and corrected images, and has good adaptability to different scenes, and has stronger ability to evaluate dispersion than other evaluation indexes.

5. Parameter and imaging quality optimization of cooperative correction method

For the proposed cooperative correction method for addressing distortion and dispersion in the Section 3, the choice of refractive index significantly impacts the quality of the correction, so that the schemes including the central wavelength scheme and the highest peak scheme that both aimed at determining the optimal refractive index are studied.

In addition, although the cooperative correction method effectively mitigates distortion and dispersion within the deflected FOV, it would introduce an adverse colored stripes at the boundary of the deflected FOV. So a method of mask matrices for different channels are proposed to eliminate the color stripes while minimizing information loss.

5.1 Optimization selection of refractive indices for RGB channels

Compared with the existing RPIS based on a single waveband, the RPIS based on an RGB camera needs to determine the refractive indices for different wavelengths of light. However, the wavelength range of visible light is wide, and it is impractical to determine the refractive index corresponding to each wavelength of light in that wavelength range, while it is more feasible to select the refractive index that can represent different channels.

The refractive index has a very important influence on the dispersion correction. To ensure that the optimal elimination of dispersion can be achieved, it is necessary to determine a suitable selection scheme for refractive index. In general, there are two common methods that can be used to determine the refractive index as follows: The first one is central wavelength scheme. The wavelength range of visible light is generally between 400–700 nm, and the R, G and B channels also have their own response range. It is an alternative to select the central wavelength of each response range as the refractive index of the channel. According to the Schott formula [29], the refractive index of the prism can be obtained at any wavelength. The other method to determine the proper refractive index for dispersion correction is the highest point scheme: The spectral response curve of RGB camera reflects the contribution of each wavelength to imaging. If the points with the highest spectral responsiveness in the R, G, and B channels is chosen to represent the band, the color dispersion caused by the most contributing wavelengths can be eliminated.

According to the above scheme, we calculated the wavelengths and refractive indices of different schemes, as shown in Table 1.

Table 1. The wavelengths and refractive indices of different schemes.

View Table

5.2 Removing color stripes at the boundary of deflected FOV in the corrected image

Although the dispersion inside the deflected FOV has been well corrected, due to the difference in spatial location between different channels of the degraded image, the grayscale information in the border area of different channels is not consistent, so it will cause the color stripes on the border of deflected FOV in corrected image. The existence of boundary color stripes, while not affecting the internal information of the corrected image, can have a detrimental effect on visual perception and subsequent image processing tasks based on the corrected image. Although cropping the corrected image to mitigate the impact of boundary color stripes is a feasible approach, it leads to an unacceptable loss of information. Hence, we propose employing masks of different channels within the corrected image to eliminate the boundary color stripes while minimizing the loss of information in the image.

Since the pixel positions of the corrected image for the different channels are known, we construct corresponding mask matrices R_mask, G_mask, B_mask for each channel. For a specific rotating angle, Figs. 8(a–c) illustrates the three matrices where the value of elements within the light blue region are set to 1, and the value of gray region is 0. The values of the corresponding positions of the three mask matrices are dot multiplied to obtain the final mask matrix C_mask representing the intersection of the non-zero regions of the three matrices. As shown in Fig. 8(d), the light blue regions represent the intersection of the deflected FOV of different channels, and all the elements within the light blue region are 1 while the elements for the gray region are 0. Next, C_mask is dot multiplied with the matrices R, G, B respectively, which generates new channels where the pixels within the intersection of the deflected FOV for all three channels are preserved, while pixels outside this intersection are eliminated. Finally, these new channels are combined to form an RGB image, effectively eliminating the boundary color stripes.

 

Fig. 8. Visualization of mask matrix: (a) R_mask. (b) G_mask. (c) B_mask. (d) C_mask.

Download Full Size | PDF

6. Fabrication of Risley-prism imaging system

To capture original degraded images, we constructed the RPIS, which is illustrated in Fig. 9. The system comprises several components such as Risley-prism, control panel, and computer. Two prisms share a coaxial optical axis with the camera's optical axis. Two Stepping motors driver them, respectively, enabling them to rotate independently while remaining coaxial. Under the action of the stepper motor, the rotation control accuracy of the system is less than 0.1 °. The computer controls the motor through the control panel and captures images through camera. Among them, the control board consists of a power module, communication module, instruction and data processing module, motor drive module, and data acquisition module, mainly completing functions such as power conversion, communication, motor motion control, and current position feedback. During the operation of the control panel, it can convert the target angle into the number of steps of the stepper motor according to the commands issued by the system, thereby controlling the operation of the motor. At the same time, the current positions of the two stepper motors are fed back to the system and displayed in real-time on the computer through the communication module. The prism's wedge angle is set at 11.35°, with a diameter of 25 mm and a thin edge thickness of 3 mm, and the prism material used is H-K9 L glass. The camera used for this system is a CMOS industrial camera with pixel dimensions of 2.4 × 2.4um, a sensor size of 1/1.8 in, a resolution of 3072 × 2048, and a focal length of 12 mm, which corresponds to an equivalent focal length of 5012pixel.

 

Fig. 9. The Risley-prism imaging system for colorful imaging with adjustable FOV.

Download Full Size | PDF

7. Experimental results

Using the established system, we conducted a series of relevant experiments to validate the correction and adaptability capabilities of the proposed algorithm and its subsequent optimization effects. The experimental results, in conjunction with the rating indicators, are depicted in Fig. 10. In Fig. 10(a), the degraded images were captured by the RPIS with all prism rotation angles set to 45°. Figures 10(b) and (c) exhibit the correction outcomes of Fig. 10(a) employing the parameters from Scheme 1 and Scheme 2 outlined in Section 5.1. Figure 10(d) showcases the subsequent refinement of Fig. 10(c) through the application of the stripe removal method detailed in Section 5.2. Figures 10 (e)–(h) respectively show the local enlarged images of Fig. 10 (a)–(d).

 

Fig. 10. (a) Degraded images were captured by the RPIS when θ₁=θ₂= 45°. (b) Corrected image based on scheme 1: Central wavelength scheme. (c) Corrected image based on Scheme 2: highest point scheme. (d) Stripe removal processing results for Fig. 10(c) based on mask method.

Download Full Size | PDF

As shown in Fig. 10(a), the distortion and dispersion can be clearly observed, but in terms of the visual effects of Fig. 10 (b) and (c), the distortion of the corrected image is much lower than that of the degraded image. In addition, the PCC of degraded images is only 0.5169, but the PCC of the corrected images obtained from two schemes are much greater than those of degraded images, which further proves the good performance of our correction algorithm. Furthermore, in order to explore the performance of two refractive index selection schemes, we conducted comparative research experiments as shown in Fig. 10 (b) and (c). The experimental results show that the correction of highest point scheme is significantly better than that of the central wavelength scheme in both visual perception and evaluation index, so the refractive index determined by this scheme is adopted as the basic parameter for the subsequent experiments. While we have successfully established the refractive index selection scheme and achieved effective correction of degraded images based on this scheme, it is worth noting that conspicuous color stripes persist along the boundary of the corrected image's deflection FOV, significantly impacting overall visual perception. Consequently, we undertook additional refinement of Fig. 10 (c) through the implementation of the stripe removal method. Figure10(d) presents the processed image, demonstrating the effective removal of the boundary color stripes. From the perspective of evaluation index, the PCC of Fig. 10 (c) and Fig. 10 (d) are almost consistent, indicating that the elimination of these stripes had no adverse impact on the internal information within the deflected FOV and the dispersion correction effect remained consistent. Notably, apart from the minimal area loss attributable to the boundary color stripes, there are no additional affected areas. This outcome validates the superior performance of our mask-based method in preserving crucial image details while successfully mitigating the presence of boundary color stripes.

Moreover, we also acquired degraded images with different combinations of prism rotation angles when the relative rotation angles of two prism remain 0° all the time, which can examine the correction effect of distortion and dispersion in the case of maximum dispersion degree in different spatial orientations. The chosen θ₁=θ₂= 40°, θ₁=θ₂= 120°, θ₁=θ₂= 210° and θ₁=θ₂= 330° corresponds to four different directions respectively. Figure 11 shows the selected locally enlarged area to highlight the details of these areas, and the first and second lines are the local regions of the degraded image and the corrected image, respectively. It can be clearly observed that the dispersions in the degraded image are well corrected and the text in the figure is more clearly visible. Based on the established evaluation index, the value of PCC of both degraded and corrected images are calculated. The PCC of the corrected images showed an obvious improvement compared with the degraded images, and the PCC improvement is the largest at the prism rotation angle of θ₁=θ₂= 120°, reaching 19.9%. Even though the improvement is the smallest at the prism rotation angle of θ₁=θ₂= 40°, it still exceeded 10%, reaching 11.6%.

 

Fig. 11. Comparison of degraded images and corrected images for indoor scenes, zoom in for details.

Download Full Size | PDF

 

Fig. 12. Comparison of degraded images and corrected images in outdoor scenes, zoom in for details.

Download Full Size | PDF

Furtherly, the experiments in the outdoor scene are also carried out. Similarly, the images based on outdoor scenes is captured when the prism rotation angles are set to be θ₁=θ₂= 0°, θ₁=θ₂= 90°, θ₁=θ₂= 180° and θ₁=θ₂= 270°, respectively. From the comparison between degraded image and correction results, the dispersion phenomenon of “buildings”, “clouds”, “lake” and other scenes is significantly weakened, and the image quality of the corrected image is significantly better than that of the degraded image. But the PCC of corrected images compared to degraded images has no significant improvement. The main reason is that the outdoor scenes are taken at a longer distance and the degraded images captured under these rotation angle combinations contain more the single-color scenes, such as “sky” or “grove”, so the dispersion phenomenon of the degraded image is not obvious, and the calculated PCC is closer to 1. Although the improvement of PCC is relatively small, it still reflects the degree of dispersion before and after image correction, which proves the effectiveness of our evaluation index (Fig. 12).

The aforementioned experimental findings clearly demonstrate the adaptability of proposed algorithm towards handling degraded images across various scenarios and combinations of prism rotation angles. Simultaneously, our evaluation index effectively gauges the extent of dispersion present in both degraded and corrected images, ensuring an objective assessment of the image quality improvements.

8. Conclusion

In conclusion, this paper proposes a new kind of correction algorithm for RPIS based on deflected FOV, which can correct the distortion and dispersion at the same time. The algorithm uses the deflected FOV of the R channel as the reference FOV, and combines the spectral response curve of the camera and Schott's formula to calculate the refractive indices, and then uses the reverse mapping algorithm based on the reverse ray tracing method to correct the distortion and dispersion of the degraded image. Compared with the degraded image, the experimental results of different scenes show that the dispersion of the corrected image is significantly lower, and the image quality is significantly better, which proves that our proposed algorithm has strong universality and robustness. Furtherly, in order to objectively evaluate the dispersion correction effect, this paper proposes PCC as the dispersion evaluation index, and the results proves that the proposed evaluation index has stronger evaluation ability by comparing with other evaluation indexes, and shows the better validity and objectivity on the evaluation of dispersion degree.