Open Access
Add to BibSonomyAbstract
Structured illumination technique enhances the lateral resolution by projecting non-uniform intensity patterns on a sample. In a typical implementation, three lateral phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern, and 3 different orientations are needed to double the bandwidth isotopically in the Fourier domain. To this end, 9 incoherent images are needed in the acquisition process. In this paper, we discuss an imaging strategy for the structured illumination technique and demonstrate the use of a modified incoherent Fourier ptychographic procedure for reducing the number of acquisitions. In the first implementation, we used complementary sinusoidal patterns for sample illumination. We show that, the number of lateral phase shifts can be reduced from 3 to 2 for each orientation of the sinusoidal pattern and the total number of image acquisitions can be reduced to 6 with 3 orientations. In the second implementation, we further reduce the number of image acquisitions to 4. We also show that, the resolution-doubled image can be recovered even with unknown phases of the sinusoidal patterns. We validate the proposed imaging procedure with non-fluorescence samples. The reported approach may shorten the acquisition time of super-resolution imaging and reduce phototoxicity of biological samples.
© 2015 Optical Society of America
1. Introduction
Improving the lateral resolution beyond the diffraction limit remains an important challenge in optical imaging. To achieve this goal, it is possible to use non-uniform patterns for sample illumination and combine multiple acquisitions for super-resolution image recovery. Frequency mixing between the sample and the non-uniform illumination pattern modulates the high-frequency components to the passband of the collection optics. Therefore, the recorded images contain sample information that is beyond the limit of the employed optics.
Conventional structured illumination microscopy (SIM) employs sinusoidal patterns for sample illumination [1–3]. In a typical implementation, three different lateral phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern and 3 different orientations are needed to double the bandwidth isotopically in the Fourier domain. Other commonly used non-uniform patterns in structured illumination technique include known/unknown speckles [4–9] and point-array patterns [10–12]. Depending on the properties of the non-uniform patterns, different digital processing algorithms can be used to recover the super-resolution sample image [1, 4–6, 8, 10, 13], including the direct matrix inversion using 3 lateral phase shifts [1], minimization of a cost function using unknown speckles [4], Bayesian estimation [7, 13], phase retrieval [6, 8], pixel reassignment [11, 12], and etc. In particular, Bayesian estimation has been demonstrated for SIM with a reduced number of raw images [13]. The capability of bypassing the diffraction limit has made structured illumination technique a popular tool for sub-diffraction imaging, enabling microscopy with high spatiotemporal resolution and reduced phototoxicity [14].
Recently, Fourier ptychography (FP) has been demonstrated for resolution improvement in coherent microscopy settings [15]. We note that, the original FP is a coherent imaging approach and uses oblique-angle illumination for expanding the frequency support. The SIM, however, is an incoherent approach and is able to achieve true super-resolution that cannot be reach by any single oblique illumination [16]. In this paper, we discuss an imaging strategy for the structured illumination technique and demonstrate the use of a modified incoherent Fourier ptychographic (FP) procedure [5, 17] for reducing the number of image acquisitions. In the first implementation, we used complementary sinusoidal patterns for sample illumination. We show that, the number of lateral phase shifts can be reduced from 3 to 2 for each orientation of the sinusoidal pattern and the total number of image acquisitions can be reduced to 6 with 3 orientations. In the second implementation, we further reduce the number of image acquisitions to 4. We also show that, the resolution-doubled image can be recovered even with unknown phases of the sinusoidal patterns. This paper is structured as follows: in section 2, we will discuss the use of complementary patterns for sample illumination. In section 3, we will demonstrate SIM recovery with 4 frames. In section 4, we will report the experimental results. Finally, we will summarize the results and discuss the future directions.
2. Complementary illumination patterns with maximum modulation strength
We propose two design guidelines for the structured illumination technique: 1) the summation of illumination patterns should be a constant for uniform modulation in the spatial domain; 2) the energy of the illumination patterns should locate at the edge of the frequency cutoff in the Fourier domain for maximizing the modulation strength.
We first review the conventional SIM based on these two design guidelines. In a typical implementation of SIM, we use three different lateral phase shifts (0, 2π/3, 4π/3) for the sinusoidal illumination patterns and capture three corresponding sample images. These three images are then used to decouple the Fourier spectrum copies of 0, + 1, and −1 orders [1]. By putting these three spectrum copies to their corresponding positions in the Fourier space, we can double the passband along the direction that is perpendicular to the sinusoidal pattern, as shown in Fig. 1. Figure 1(a) shows a sample image under uniform illumination and its corresponding frequency support in the Fourier space. Figure 1(b1) shows the three illumination patterns with three different lateral phase shifts (0, 2π/3, 4π/3). Figure 1(c1) shows the recovered resolution-doubled image and the corresponding frequency support.
Fig. 1 Comparison between a typical SIM implementation and the proposed approach using a minimum number of complementary illumination patterns. (a) The sample image under uniform illumination and its corresponding frequency support. (b1) Conventional SIM implementation with 3 lateral phase shift. (b2) The proposed approach using two complementary illumination patterns. (c1-c2) The corresponding super-resolution recoveries.
It is obvious that, the summation of the 3 phase-shifted sinusoidal patterns is a constant in the conventional SIM implementation. Therefore, the modulation is uniform in the spatial domain. In addition, the sinusoidal pattern provides maximum modulation strength as it contains two delta peaks at the frequency cutoff in the Fourier domain. However, the three phase shifts in the conventional SIM may not be necessary, as the Fourier spectrum copies of + 1 and −1 orders contains the same information. We argue that, we can simply use 0 and π phase shifts for modulating the sample image, as shown in Fig. 1(b2). In this case, the summation of the illumination patterns is a constant and the patterns also provide maximum modulation strength for the sample, in accord with the design guidelines.
To recover the resolution-doubled image from the two modulated images with 0 and π phase shifts, we propose to use an empirical Fourier ptychographic procedure modified from the previous approaches [5, 17]. We have the following definitions for modeling the image formation and recovery process: the high-resolution sample image , the sinusoidal illumination pattern (n = 1, 2) with two phase shifts, the diffraction-limited measurement (n = 1, 2), and the point spread function (PSF) of the employed optics. The forward imaging model can be described as follows:
where ‘*’ stands for convolution. The goal of the reported procedure is to recover from the 2 measurements (n = 1, 2). The recovery process begins with an initial guess of high-resolution sample image, .This initial guess is then multiplied with one of the illumination sinusoidal patterns to generate a target image in the spatial domain:. The target image is then updated with the measurement with:where ‘deconvwnr’ stands for Wiener deconvolution. The updated target image is then used to update the initial guess as follows:Equation (2) and (3) are repeated for the two measurements iteratively and the entire process is terminated until convergence, which can be measured by the difference between two successive recoveries. In a practical implementation, we can simply terminate it with a predefined loop number, typically 50-400. Different from the previous reported Fourier ptychographic approach [5], the reported procedure is performed at the spatial domain and the use of deconvolution is able to capture the noise characteristics and improve the convergence performance (Fig. 3(c)). Figure 1(c2) shows the recovered image using the above procedures.3. Resolution doubling using 4 frames
In the previous section, we have discussed the use of 2 complementary patterns for resolution doubling in one direction. In a typically implementation, we need to double the resolution isotopically in the Fourier domain, and thus, at least 3 different orientations of the sinusoidal patterns are needed (totally 6 acquisitions are needed). Here, we propose to further reduce the number of image acquisitions to 4 by using the illumination patterns shown in Fig. 2.
Fig. 2 The proposed 4 illumination patterns for resolution doubling in the structured illumination technique. (a1)-(a2) The two complementary patterns (with 0 and π shifts) for uniform sample modulation (also see Fig. 3(d)). (b)-(c) Patterns with two other orientations for expanding the bandwidth along two other directions.
The choice of these 4 illumination patterns is not trivial. The first two commentary patterns (Fig. 2(a1) and (a2)) provide uniform modulation for the sample (no dark region) and the summation of these two images forms a good initial guess for the high-resolution sample image. If we only use one pattern from Fig. 2(a1)-(a2) and the two patterns with other orientations (totally three images), there are ‘dark’ regions on the sample that cannot be modulated, leading to artifacts in the recovery process.
Figure 3 summarizes the simulation performance of the reported approach. Figure 3(b1) shows the results using the conventional three-phase-shifted technique and the 9-frame recovery with direct matrix inversion. Figure 3(b2) shows the results using two-phase-shifted complementary patterns and the 6-frame Fourier ptychographic recovery. Figure 3(b3) shows the results corresponding to the 4 illumination patterns in Fig. 2. In this simulation, we used both the mean-square-error (MSE) and structural similarity (SSIM) index to compare the imaging performances. We can see that, the reported 4-frame, 6-frame approaches have similar performances as the conventional 9-frame SIM. Figure 3(c) shows the convergence performance between the spatial-domain (this paper) and the Fourier domain [5] implementations, where the reported approach enables a higher convergence speed. Figure 3(d) shows the phase difference between the two complementary patterns as a function MSE performance. We can see that, the performance gradually degrades as the phase difference differs from 180 degree.
Fig. 3 (a) Raw images with uniform illumination. Simulation comparison between a typical SIM implementation (b1) and the proposed approach using a reduced number of image acquisitions (b2)-(b3). We used 200-400 loops in the 6-frame and 4-frame recovery processes. (c) Comparison between the spatial-domain implementation (this paper) and the Fourier domain implementation. (d) Phase differences of complementary patterns versus the MSE performance.
In a practical implementation, we may not know about the absolute phases of the illumination patterns. To address this issue, we can use random phases as the initial guesses for the 4 sinusoidal patterns and use the following procedures to recover the unknown phases in the iterative recovery process. We first add one more step to update the illumination pattern following the updating step of Eq. (3):
The updated illumination pattern from Eq. (4) is then fitted to a sinusoidal pattern with an updated phase. Figure 4 shows the recovered phases of the 4 sinusoidal patterns. We can see that, they all converge to the ground truth after 500 loops and the recovered phases are accurate (~0.0014 λ error) even with different levels of simulated noises.4. Experimental validations
To validate the reported imaging procedure, we performed an experiment using a photographic lens for image acquisition and a video projector (with digital mirror device) for projecting different patterns on the sample. Figure 5(a) shows the setup of the experiment. Figure 5(b) shows the diffraction-limited sample image captured under uniform illumination. We then projected different patterns on the sample and used the corresponding acquisitions to recover the resolution-doubled sample image. Figure 5(d) shows one of raw image, and Fig. 5(e-f) show the recovered resolution-doubled images using 4 frames and 6 frames. The corresponding line traces are shown in Fig. 5(g). In Fig. 6, we further demonstrated the performance of the reported approach using a dollar bill and a color object. These two experimental demonstrations validate the effectiveness of the reported imaging procedure.
Fig. 5 (a) The experimental scheme, similar to the epi-illumination mode in microscopy. (b) The diffraction-limited sample image captured under uniform illumination. (c) The deconvolved image of (b). (d) The captured image under the sinusoidal pattern illumination, with ~2% noise. The resolution-doubled images using 4-frame (e) and 6-frame (f) recovery schemes (400 loops). (g) Line traces of the images.
Fig. 6 Experimental validation of the reported recovery approach using a dollar bill and a color object. (a) The diffraction-limited sample image captured under uniform illumination. (b) The deconvolved image of (a). The resolution-doubled images using 4-frame (c) and 6-frame (d) recovery schemes.
5. Discussion and conclusion
We have discussed an imaging procedure for resolution doubling with a reduced number of image acquisitions. The key contributions of this paper are three folds: 1) we propose the use of complementary patterns with maximum modulation strength for the structured illumination technique. In particular, we propose to use 4 sinusoidal patterns for resolution doubling. Two for uniform modulation and forming a good initial guess; other two for resolution improvement along other directions. 2) We report a modified ptychographic procedure that is based on spatial operations and use Wiener deconvolution to better capture the noise characteristics and improve the convergence performance. 3) We demonstrate a simple process for estimating the unknown phases of the illumination patterns. In summary, the reported imaging procedure is easy to implement and may find board applications in high-speed super-resolution imaging.
We note that, the use of deconvolution in the reported Fourier ptychographic procedure require the use of signal to noise (SNR) ratio as an input parameter. A higher SNR input leads to faster convergence but also amplifies the noise. In practice, we can simply choose a low SNR ratio (for example, 0.5 from our experiments) as the input and use more iterations for solution convergence. The choice of the best SNR ratio for the reported procedure is a future direction that requires more research. Finally, the reported approach can also be extended to tackle 3D sample for fluorescence imaging; work along this direction is in progress.
Acknowledgments
The work was supported by NSF CBET 1510077.
References and links
1. M. G. Gustafsson, “Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy,” J. Microsc. 198(2), 82–87 (2000). [CrossRef] [PubMed]
2. M. G. Gustafsson, “Nonlinear structured-illumination microscopy: wide-field fluorescence imaging with theoretically unlimited resolution,” Proc. Natl. Acad. Sci. U.S.A. 102(37), 13081–13086 (2005). [CrossRef] [PubMed]
3. M. G. Gustafsson, L. Shao, P. M. Carlton, C. J. Wang, I. N. Golubovskaya, W. Z. Cande, D. A. Agard, and J. W. Sedat, “Three-dimensional resolution doubling in wide-field fluorescence microscopy by structured illumination,” Biophys. J. 94(12), 4957–4970 (2008). [CrossRef] [PubMed]
4. E. Mudry, K. Belkebir, J. Girard, J. Savatier, E. Le Moal, C. Nicoletti, M. Allain, and A. Sentenac, “Structured illumination microscopy using unknown speckle patterns,” Nat. Photonics 6(5), 312–315 (2012). [CrossRef]
5. S. Dong, P. Nanda, R. Shiradkar, K. Guo, and G. Zheng, “High-resolution fluorescence imaging via pattern-illuminated Fourier ptychography,” Opt. Express 22(17), 20856–20870 (2014). [CrossRef] [PubMed]
6. H. Yilmaz, E. G. van Putten, J. Bertolotti, A. Lagendijk, W. L. Vos, and A. P. Mosk, “Speckle correlation resolution enhancement of wide-field fluorescence imaging,” Optica 2(5), 424–429 (2015). [CrossRef]
7. J. Min, J. Jang, D. Keum, S.-W. Ryu, C. Choi, K.-H. Jeong, and J. C. Ye, “Fluorescent microscopy beyond diffraction limits using speckle illumination and joint support recovery,” Sci. Rep. 3, 2075 (2013). [CrossRef] [PubMed]
8. O. Wagner, A. Schwarz, A. Shemer, C. Ferreira, J. García, and Z. Zalevsky, “Superresolved imaging based on wavelength multiplexing of projected unknown speckle patterns,” Appl. Opt. 54(13), D51–D60 (2015). [CrossRef]
9. S. Dong, P. Nanda, K. Guo, J. Liao, and G. Zheng, “Incoherent Fourier ptychographic photography using structured light,” Photon. Res. 3(1), 19–23 (2015). [CrossRef]
10. C. B. Müller and J. Enderlein, “Image Scanning Microscopy,” Phys. Rev. Lett. 104(19), 198101 (2010). [CrossRef] [PubMed]
11. G. P. J. Laporte, N. Stasio, C. J. R. Sheppard, and D. Psaltis, “Resolution enhancement in nonlinear scanning microscopy through post-detection digital computation,” Optica 1(6), 455–460 (2014). [CrossRef]
12. I. J. Cox, C. J. Sheppard, and T. Wilson, “Super-resolution by confocal fluorescent microscopy,” Optik (Stuttg.) 60, 391–396 (1982).
13. F. Orieux, E. Sepulveda, V. Loriette, B. Dubertret, and J.-C. Olivo-Marin, “Bayesian estimation for optimized structured illumination microscopy,” IEEE Trans. Image Process. 21(2), 601–614 (2012). [CrossRef] [PubMed]
14. B.-C. Chen, W. R. Legant, K. Wang, L. Shao, D. E. Milkie, M. W. Davidson, C. Janetopoulos, X. S. Wu, J. A. Hammer 3rd, Z. Liu, B. P. English, Y. Mimori-Kiyosue, D. P. Romero, A. T. Ritter, J. Lippincott-Schwartz, L. Fritz-Laylin, R. D. Mullins, D. M. Mitchell, J. N. Bembenek, A. C. Reymann, R. Böhme, S. W. Grill, J. T. Wang, G. Seydoux, U. S. Tulu, D. P. Kiehart, and E. Betzig, “Lattice light-sheet microscopy: Imaging molecules to embryos at high spatiotemporal resolution,” Science 346(6208), 1257998 (2014). [CrossRef] [PubMed]
15. G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution Fourier ptychographic microscopy,” Nat. Photonics 7(9), 739–745 (2013). [CrossRef] [PubMed]
16. K. Wicker and R. Heintzmann, “Resolving a misconception about structured illumination,” Nat. Photonics 8(5), 342–344 (2014). [CrossRef]
17. A. M. Maiden and J. M. Rodenburg, “An improved ptychographical phase retrieval algorithm for diffractive imaging,” Ultramicroscopy 109(10), 1256–1262 (2009). [CrossRef] [PubMed]
