Abstract
The optical sparse aperture technique can improve the imaging resolution significantly under the ideal co-phase condition. However, the position deviation between different sub-apertures leads to notorious co-phase errors, seriously impacting the image quality. While the position deviation arises in practical applications, it is difficult to detect the errors in real-time for traditional iterative algorithms because of their narrow detection range and long-time iteration process. The deep neural network has shown its potential in optical information process, and it has some attempts in the detection of piston error. However, all existing deep learning-based methods just focus on the detection of piston error with the weak or corrected tilt error, which is not in line with reality. Here we implement the deep convolutional neural network to detect tilt error with large-scale piston error, and compare the detection performance of two kinds of network, one takes the point spread function as input while the other takes the phase diversity features as the input. The detection ability and generalization capability of network are verified under single wavelength, broadband light and turbulence aberration in simulation. The object-independent of tilt error detection ability is offered because the phase diversity features and point spread function are all unrelated to the object. In addition, the cyclic correction strategy is carried out to improve the generalization performance facing the larger errors. As a result, the deep learning-based method can detect the tilt error accurately with fast calculation, and the trained network is hopeful for the real-time correction with cyclic correction strategy.
© 2021 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement
1. Introduction
The optical synthetic aperture technique, which has developed rapidly for large aperture telescope manufacturing in recent years, can improve the resolution significantly under ideal co-phase condition. However, the position deviation between different sub-apertures leads to the exist of co-phase errors, impacting the quality of imaging seriously. Considerable effort has been taken to solve the misalignment of sub-apertures in optical method, such as Hartmann wavefront sensor [1], pyramid sensor [2], curvature sensor [3], Mach–Zehnder interferometer sensor [4] and Zernike phase contrast sensor [5]. The added special sensors can analyze the wavefront state in real time, but are expensive and inconvenient to detect the co-phase errors in practical applications. Several phase retrieval ways are also suitable for wavefront sensing, but the iterative process is really time consuming [6–8]. The phase diversity (PD) is an image-based wavefront sensing technique without additional sensors [9–10]. It utilizes the focused and defocused images with a known defocused phase to estimate the system aberrations. Combined with the PD theory, the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm is used for detecting the piston and tilt errors in the segmented mirror system [11–12]. However, even the modified BFGS algorithm needs 500 steps iteration, and can only detect the [0, 0.5λ] range co-phase errors [12].
The rise of deep learning technology has shown its potential in hologram reconstruction [13–14], scattering medium imaging [15–16], phase information processing [17–19] and image restoration [20–21], etc. The trained deep neural network is suitable for the fast detection of co-phase errors. Guerra-Ramos et al. take the junction of three mirror segments as the feature to train the network [22]. Although the performance is excellent with [0, 11λ] detection range, the method doesn’t consider the exists of tilt error, and is not suitable for the sparse aperture system. Li et al. and Hui et al. both combine the PD theory with the convolutional neural networks for segmented mirrors, but Li just corrects the piston error from 10λ to λ under [−0.5, 0.5λ] tilt error, and Hui use the same method from λ to 0.06 λ supposing that the tilt error is corrected previously [23–24]. Ma et al. propose two deep learning-based methods, which take the PD features or point spread function (PSF) as inputs, for the piston error detection of sparse aperture system [25–26], but she doesn’t compare these two methods and analyze the characteristics. The above-mentioned methods all only focus on the detection of piston error, assuming that the tilt error is relatively weak or even effectively corrected. Moreover, the characteristics of PD-based and PSF-based network, and the generalization capability of error range are not discussed. In fact, the tilt error is always simultaneous with the piston error, and has a more serious impact on the image quality. Although the modified BFGS algorithm can detect piston and tilt errors meanwhile, its narrow detection range and long-time iteration, make itself difficult to apply in dynamic detection and real-time correction. Compared with the piston error, the tilt error involves more variations and needs more parameters to characterize, which may bring challenges for detection. To implement the fast co-phase errors detection of neural network, its performance and characteristics to tilt error detection need to be investigated urgently.
In this paper, we propose the deep learning-based tilt error detection methods, and compare the effect of three deep convolutional neural networks (DCNN), which use PD features and PSF as the inputs, respectively. At first, the optical sparse aperture imaging (OSAI) system is built for acquiring data. The two feature matrices by PD theory are obtained as the inputs of DCNN, as well as the PSF of system. Through simulation, the PSF-based and PD-based DCNNs both show their detection ability and generalization capability under single wavelength, broadband light and turbulence aberration. At last, all DCNNs can detect the tilt error of OSAI system with fast calculation, and are expected to the real-time correction with cyclic correction strategy in practical applications.
2. Principle
2.1 Optical imaging model
According to the linear system theory, the imaging process of synthetic aperture system is expressed as
where f (x, y) is the ideal intensity distribution of objects, g (x, y) and PSF (x, y) are the observed image on the focal plane and the point spread function of system, * is the convolution operation and (x, y) is the coordinates of imaging plane. Considering the piston and tilt errors without the turbulence aberration, the pupil function of OSAI system is2.2 Feature matrices
As a data-driven method, the neural network can establish the underlying relationship between inputs and labels from a large number of samples through learning. It means that the dataset should satisfy both of the following characteristics: 1) There is a strict one-to-one correspondence between the inputs and labels, while the labels are different co-phase errors here; 2) The data samples are easy to obtain relatively. The PSF is an ideal feature as the input because it can reflect the imaging effect accurately. PD is an adaptive optics technology to infer the phase aberrations from the intensity images. The PD method requires two images, one in the focal plane and the other in the defocus plane. The defocused image can be captured in similar way
To calculate the relative error of the other two sub-apertures, the sub-aperture 2 is regarded as the standard aperture in Fig. 2(a). The piston and tilt errors are added into sub-aperture 1 for analyzing characteristics. From Fig. 2(b), the energy distribution of the PSF expands around and the shape of MTF also changes with the increase of piston error until π. That means the piston error causes the shift of interference position, lead to the frequency transfer response attenuation. The change reaches its maximum at π, and then decays to original shape until 2π. The focused and defocused images are almost the same in imaging process. It can be concluded that the imaging effect of piston error has the period of 2π, and it is relatively weak to imaging effect at a single wavelength. The sharpness matrix and the power matrix also keep the periodic change, which proves their consistency with the piston error. However, the period does not exist in tilt error from Fig. 3, and the energy center of PSF is not displaced. The influence to imaging effect becomes more serious with the increase of tilt angle, due to the continuous cluttered distribution of PSF and MTF. The sharpness matrix and the power matrix still keep their consistency with the tilt error. As for the combination of piston error and tilt error in Fig. 4, the influence of piston error is almost covered by that of tilt error, which indicates that the tilt error should be controlled in a small range before correcting the piston error. A common advantage of PD features and PSF is offered as the object independent, which means that two feature matrices and PSF only depend on the co-phase errors of system, rather than the content of extended objects. In conclusion, the feature matrices of PD are corresponding to the co-phase errors in OSAI system, which is the same as PSF and MTF.
We can use the PSF, the sharpness and power matrices as the features to extract the corresponding co-phase errors in “Golay-3” OSAI system. The neural network has shown its potential in extracting piston error [23–26]. However, the tilt error will introduce more changes compared with the piston error, leading to the more complex features. To achieve the deep learning-based co-phase errors sensing, the tilt error detection should be taken into consideration first.
3. Method
The deep convolutional neural network is a widely-used classic backbone [30]. Figure 5 denotes that the cascaded convolution layers and pooling layers extract and compress the features from inputs, and then the features are reshaped into one-dimensional vectors with the fully connected layers (FC). The convolution layers contain the convolution operation (Conv) with 3×3 kernel, the batch normalization (BN), and the leaky rectified linear unit (LRelu) function. The pooling layers are the max pooling operation with 2×2 kernel. The number in the lower left of features is channel × size. The mean absolute error (MAE) is calculated to back propagate and update parameters of DCNN. At last, the output is a one-dimensional vector with N values. In fact, N can vary while extracting the tilt error of one sub-aperture or all sub-apertures with DCNN. In this paper, N is set as 2 because extracting tilt error of any one sub-aperture is the basis for detecting that of all sub-apertures.
The imaging process is simulated through the optical imaging model. To verify the independence of imaging objects, 10000 and 2000 optical remote sensing images from DIOR dataset [31] are chosen for training and testing. In Fig. 6, the PSF, the focused and defocused images are obtained by the OSAI system. The power and sharpness matrices can be calculated by the two images. Then we respectively set the power matrices Mpower, the sharpness matrices Msharpness, and the PSF as inputs, and regard the tilt angles Δθx and Δθy of one sub-aperture as labels to construct the dataset. The networks trained by Mpower, Msharpness and PSF are named as DCNN-Mp and DCNN-Ms, DCNN-P, respectively. The relative piston and tilt errors are both added into sub-aperture 1 and 3 with the limit of [-X1, X1] µm and [-X2, X2] µrad in Fig. 6, and the specific Δz, Δθx and Δθy are all generated independently by random numbers. From Fig. 6, the piston and tilt error limited [−3, 3] µm and [−3, 3] µrad are equal to [−5λ, 5λ] and [−2λ, 2λ] under 600 nm wavelength in train set. The time consumption is ∼5.5 h using 10000 images, which is divided into 4:1 for train and validation. The test process takes ∼12 s for 2000 images with the Intel Core Processor CPU (2.5 GHz) and NVIDIA GeForce RTX 2080Ti GPU.
4. Simulation results
4.1 Tilt error detection at a single wavelength
The detection ability for tilt error is taken into consideration first. The dataset, containing one train set and four test sets, is obtained at a single wavelength. Three DCNNs are trained with the different train sets to extract the tilt error of sub-aperture 1. The loss curves during training are shown in Fig. 7. In comparison, The DCNN-P converges faster with less oscillation and lower values than the PD-based DCNNs. The detection performance of DCNNs in four test sets is shown as data distribution in Fig. 8. The original tilt error along X and Y axis increase from Test1 to Test4 sets in Fig. 8(a). All three DCNNs can perform well in Test1 and Test2 because of the similar data distribution in Figs. 8(b-d). The Test3 and Test4 sets with a larger error range than train set are used to verify the generalization capability. The DCNN-P performances better than DCNN-Mp and DCNN-Ms in Test3 and Test4, although all DCNNs have declining effects. Figure 9 shows the corresponding percentages of phase distribution in four test sets. From Figs. 9(a-b), the tilt phase φtx and φty are almost corrected into [0, λ/10] with three DCNNs in Test1 and Test2. However, the percentages of [0, λ/10] in Test3 and Test4 are below 60% and 30%, which is higher than original distribution but lower than expected level. It is noteworthy that the percentage of [0, λ] in Test3 and Test4 also increase, which indicates the overall tilt error reduces after correction in Figs. 9(c-d). It is feasible to use cyclic correction instead of one-step correction theoretically, thanks to the high speed of trained DCNN in test process.
4.2 Tilt error detection under broadband light
The single wavelength is unsuitable for the practical applications, so the broadband light is taken into research. The PSF of system under the broadband light can be obtained as
The similar detection performance and phase distribution are both shown in Fig. 11 and Fig. 12. Compared with those in Fig. 8, there are slight decreases in the detection effect of three DCNNs in Fig. 11, due to the broadband light. In general, all DCNNs still can correct the phase error into [0, λ/10] in Test1 and Test2, and reduce the overall errors in Test3 and Test4 in Fig. 12. However, the DCNN-P keeps the same while the DCNN-Mp and DCNN-Ms decrease in Test1 and Test2 from Figs. 12(a-b). And the percentages of [0, λ/10] in Test3 and Test4 from Figs. 12(c-d) are close to 50% and 20%, which are lower than those in Figs. 9(c-d). All in all, the detection ability of DCNNs receive impact due to the broadband light, while the decreases of DCNN-Mp and DCNN-Ms are larger. In following discussion, DCNN-P and DCNN-Mp are used for comparison without DCNN-Ms, because Mpower and Msharpness are obtained from the same two images and have similar properties and change trends.
4.3 Tilt error detection considering turbulence aberration
The turbulence aberration is an inevitable problem in remote target imaging. With the additional phase introduced by turbulence aberration and the position deviation of sub-apertures, the real-time correction of tilt error will become important and difficult in applications. The turbulence aberration limited to 5λ is added into the simulation process under broadband light to explore its impact. The turbulence phase is constructed by Zernike polynomial obtained from the Hartmann wavefront sensor in the turbulence generation pool. In simulation, we randomly select the turbulence phase loading from the collected data to increase the randomness, because the experimental acquisition speed is only tens of Hz. The influence of turbulence aberration is clear from the changes in Mpower and PSF in Fig. 13(a). Thanks to the fast calculation of trained DCNN, we can use cyclic correction here for the real-time correction of “Golay-3” OSAI system. Four DCNNs, including DCNN-P and DCNN-Mp for the tilt error detection of sub-aperture 1 and 3, are all retrained with the turbulence aberration dataset respectively.
Figure 13(b) shows the effect after one-time tilt error correction in Test2. The impact of tilt error in PSF, Mpower, focused and defocused images are controlled effectively, and Δθx and Δθy of sub-aperture 1 and 3 are reduced to a small value in Fig. 13(b). There is the comparison of phase distribution belong to [0, λ/10] after cyclic correction in Test2, Test3 and Test4 in Fig. 14. It is worth mentioning that the one-time correction effect in Test3 and Test4 is bad because of the larger tilt error range, which is proved in Section 4.1. The cyclic correction with N times (C-N) is taken into consideration and has good performance in Figs. 14(b-c). The correction with three and five times in Test3 and Test4 only take ∼20 ms and ∼33 ms in all. DCNN-P can both achieve over 90% accuracy in Test3 and Test4, while DCNN-Mp can only get close to 80% in Test3 and 60% accuracy in Test4. In fact, Mpower is more complex and changeable than PSF, resulting in the mistaken detection during the generalization process. Some large tilt errors are corrected to smaller gradually, while the mistakes may accumulate resulting in the correction failure of others. To solve this problem, the train set containing a larger range of tilt error may be used to let DCNN have better generalization capability, and the constraints may be added to limit the output value in cyclic correction.
5. Analysis and discussions
The research focuses on the fast correction of tilt error in “Golay-3” OSAI system in the last section, and detection ability and generalization capability of DCNNs is verified. To further prove the performance of DCNN-P and DCNN-Mp, different arrangements and number of sub-apertures are considered. The “Annular-4”, “Annular-5”, “Annular-6”, “Goaly-6” and “Y-6” OSAI system are simulated with the same main parameters. Several DCNNs with corresponding datasets of X1=X2=3 are taken to train and test.
From Table 2, DCNN-P can keep the stable performance in detecting tilt error of one sub-aperture in all OSAI systems, while DCNN-Mp declines seriously. To analysis the reasons behind this, more data about DCNN-Mp is shown in Fig. 15. The shape of Mpower is really similar to the MTF of system in Fig. 15(a). Different number of sub-apertures and arrangements corresponding to different shapes of MTF and Mpower. Figures 15(b) and (c) show the loss curves of DCNN-Mp in different number of sub-apertures with “Annular” arrangement and in different arrangements with 6 sub-apertures, respectively. Compared with the loss curves in Fig. 7, these loss curves are much higher and even not converge. The features in Mpower are combined more complex due to the increase of sub-apertures, but the training set and network parameters keep the same capacity. In Figs. 15(b-c), DCNN-Mp of “Annular-6” gets a relatively better convergence effect than others in valid loss, while all train loss curves have similar convergence level. It shows that all DCNNs have overfitting phenomenon and can’t get a good generalization effect in test sets. From their corresponding Mpower, the shape of “Annular-6” is the most uniform while others are scattered or crowded. Some features with tilt error information may be covered or hidden, resulting in the network can’t demodulate smoothly. Deepening the network and increasing the amount of data can help solving the overfitting problem effectively, but this will also increase the time costs. As a result, the number of sub-apertures and the arrangements will lead to the shape changes of Mpower, and then affect the detection of tilt error using DCNN-Mp. By contrast, DCNN-P has strong detection ability, generalization capability and robustness.
6. Conclusions
In conclusion, the deep learning-based method can extract the tilt error of OSAI system under single wavelength, broadband light and turbulence aberration, and once detection takes less than 10 ms, which is suitable for the real-time correction. A common advantage of PD-based and PSF-based networks is offered as the object independent, which greatly increase the generalization capability of the trained DCNN on the object diversity. As for the generalization problem with the large error, cyclic correction can be an effective solution to improve performance. The PD-based DCNN have advantages in applications because it only uses focused and defocused images to form the inputs of network, but it may be affected by the apertures. By comparison, the DCNN-P shows stronger detection ability, generalization capability and robustness than DCNN-Mp. In following research, the actual effect of DCNN needs closed-loop correction experiment to prove, and how to correct the piston and tilt errors meanwhile through network is still a problem for research.
Funding
National Natural Science Foundation of China (61905197, 61927810, 62075183); State Key Laboratory of Transient Optics and Photonics (SKLST202008).
Disclosures
The authors declare no conflicts of interest.
Data availability
Data underlying the results presented in this paper are not publicly available at this time but may be obtained from the authors upon reasonable request.
References
1. G. A. Chanan, Design of the Keck Observatory alignment camera, in Precision Instrument Design, vol. 1036 (International Society for Optics and Photonics, 1989), p. 59–71.
2. S. Esposito, E. Pinna, A. Puglisi, A. Tozzi, and P. Stefanini, “Pyramid sensor for segmented mirror alignment,” Opt. Lett. 30(19), 2572–2574 (2005). [CrossRef]
3. V. G. Orlov, S Cuevas, and F. Garfias, Co-phasing of segmented mirror telescopes with curvature sensing, in Telescope Structures, Enclosures, Controls, Assembly/Integration/Validation, and Commissioning, vol. 4004 (International Society for Optics and Photonics, 2000), p. 540–551.
4. N. Yaitskova, K. Dohlen, and P. Dierickx, “Mach–Zehnder interferometer for piston and tip–tilt sensing in segmented telescopes: theory and analytical treatment,” J. Opt. Soc. Am. A 22(6), 1093–1105 (2005). [CrossRef]
5. I. Surdej, N. Yaitskova, and F. Gonte, “On-sky performance of the Zernike phase contrast sensor for the phasing of segmented telescopes,” Appl. Opt. 49(21), 4052–4062 (2010). [CrossRef]
6. R. E. Carlisle and D. S. Acton, “Demonstration of extended capture range for James Webb Space Telescope phase retrieval,” Appl. Opt. 54(21), 6454–6460 (2015). [CrossRef]
7. P. Bao, F. Zhang, G. Pedrini, and W. Osten, “Phase retrieval using multiple illumination wavelengths,” Opt. Lett. 33(4), 309–311 (2008). [CrossRef]
8. M. Guizar-Sicairos and J. R. Fienup, “Phase retrieval with transverse translation diversity: a nonlinear optimization approach,” Opt. Express 16(10), 7264–7278 (2008). [CrossRef]
9. R. L. Kendrick, D. S. Acton, and A. L. Duncan, “Phase-diversity wave-front sensor for imaging systems,” Appl. Opt. 33(27), 6533–6546 (1994). [CrossRef]
10. M. G. Lofdahl, R. L. Kendrick, and A. Harwit, Phase diversity experiment to measure piston misalignment on the segmented primary mirror of the Keck II telescope, in Space Telescopes and Instruments V, vol. 3356 (International Society for Optics and Photonics, 1998), p. 1190–1201.
11. D. H. Li and M. Fukushima, “On the global convergence of the BFGS method for nonconvex unconstrained optimization problems,” SIAM J. Optim. 11(4), 1054–1064 (2001). [CrossRef]
12. D. Yue, S. Xu, and H. Nie, “Co-phasing of the segmented mirror and image retrieval based on phase diversity using a modified algorithm,” Appl. Opt. 54(26), 7917–7924 (2015). [CrossRef]
13. J. Wu, K. Liu, X. Sui, and L. Cao, “High-speed computer-generated holography using an autoencoder-based deep neural network,” Opt. Lett. 46(12), 2908–2911 (2021). [CrossRef]
14. Z. Ren, Z. Xu, and E. Y. Lam, “End-to-end deep learning framework for digital holographic reconstruction,” Adv. Photonics 1(1), 016004 (2019). [CrossRef]
15. E. Guo, S. Zhu, Y. Sun, L. Bai, C. Zuo, and J. Han, “Learning-based method to reconstruct complex targets through scattering medium beyond the memory effect,” Opt. Express 28(2), 2433–2446 (2020). [CrossRef]
16. M. Lyu, H. Wang, G. Li, S. Zheng, and G Situ, “Learning-based lensless imaging through optically thick scattering media,” Adv. Photonics 1(3), 036002 (2019). [CrossRef]
17. K. Wang, Y. Li, Q. Kemao, J. Di, and J. Zhao, “One-step robust deep learning phase unwrapping,” Opt. Express 27(10), 15100–15115 (2019). [CrossRef]
18. K. Wang, M. Zhang, J. Tang, L. Wang, L. Hu, X. Wu, W. Li, J. Di, G. Liu, and J. Zhao, “Deep learning wavefront sensing and aberration correction in atmospheric turbulence,” PhotoniX 2(1), 1–11 (2021). [CrossRef]
19. K. Yan, Y. Yu, T. Sun, A. Asundi, and Q. Kemao, “Wrapped phase denoising using convolutional neural networks,” Opt Lasers Eng 128, 105999 (2020). [CrossRef]
20. J. Tang, K. Wang, Z. Ren, W. Zhang, X. Wu, J. Di, G. Liu, and J. Zhao, “RestoreNet: a deep learning framework for image restoration in optical synthetic aperture imaging system,” Opt Lasers Eng 139, 106463 (2021). [CrossRef]
21. J. Tang, J. Wu, K. Wang, Z. Ren, X. Wu, L. Hu, J. Di, G. Liu, and J. Zhao, “RestoreNet-Plus: Image restoration via deep learning in optical synthetic aperture imaging system,” Opt Lasers Eng 146, 106707 (2021). [CrossRef]
22. D. Guerra-Ramos, L. Díaz-García, and J. Trujillo-Sevilla, “Piston alignment of segmented optical mirrors via convolutional neural networks,” Opt. Lett. 43(17), 4264–4267 (2018). [CrossRef]
23. D. Li, S. Xu, D. Wang, and D. Yan, “Large-scale piston error detection technology for segmented optical mirrors via convolutional neural networks,” Opt. Lett. 44(5), 1170–1173 (2019). [CrossRef]
24. M. Hui, W. Li, M. Liu, L. Dong, L. Kong, and Y. Zhao, “Object-independent piston diagnosing approach for segmented optical mirrors via deep convolutional neural network,” Appl. Opt. 59(3), 771–778 (2020). [CrossRef]
25. X. Ma, Z. Xie, H. Ma, Y. Xu, D. He, and G. Ren, “Piston sensing for sparse aperture systems with broadband extended objects via a single convolutional neural network,” Opt Lasers Eng 128, 106005 (2020). [CrossRef]
26. X. Ma, Z. Xie, H. Ma, Y. Xu, G. Ren, and Y. Liu, “Piston sensing of sparse aperture systems with a single broadband image via deep learning,” Opt. Express 27(11), 16058–16070 (2019). [CrossRef]
27. R. G. Paxman and J. R. Fienup, “Optical misalignment sensing and image reconstruction using phase diversity,” J. Opt. Soc. Am. A 5(6), 914–923 (1988). [CrossRef]
28. F. Cao, Y. Zhu, and Z. Wu, “Optical misalignment sensing for optical aperture synthesis telescope by using phase diversity,” in Adaptive Optics Systems vol. 7015 (International Society for Optics and Photonics, 2008), p. 70154Q.
29. J. R. Fienup, B. J. Thelen, R. G. Paxman, and D. A. Carrara, “Comparison of phase diversity and curvature wavefront sensing,” in Adaptive Optical System Technologies vol. 3353 (International Society for Optics and Photonics, 1998), p. 930–940.
30. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” vol. 25 (Advances in neural information processing systems, 2012), p. 1097–1105.
31. K. Li, G. Wan, G. Cheng, L. Meng, and J. Han, “Object detection in optical remote sensing images: A survey and a new benchmark,” ISPRS J. Photogramm 159, 296–307 (2020). [CrossRef]